Autodecoding Latent 3D Diffusion Models
Authors: Evangelos Ntavelis
Aliaksandr Siarohin
Kyle Olszewski
Chaoyang Wang
Luc Van Gool
Sergey Tulyakov
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Generation
Diffusion Models
TL;DR: We generate 3D assets from diverse 2D multi-view datasets by training a 3D Diffusion model on the intermediate features of a Volumetric AutoDecodER.

Abstract: Diffusion-based methods have shown impressive visual results in the text-to-image domain. They first learn a latent space using an autoencoder, then run a denoising process on the bottleneck to generate new samples. However, learning an autoencoder requires substantial data in the target domain. Such data is scarce for 3D generation, prohibiting the learning of large-scale diffusion models for 3D synthesis. We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation for rendering view-consistent appearance and geometry. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations to learn a 3D diffusion from 2D images or monocular videos of rigid or articulated objects. Our approach is flexible enough to use either existing camera supervision or no camera information at all -- instead efficiently learning it during training. Our evaluations demonstrate that our generation results outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and a large-scale, real video dataset of static objects.

Autodecoding Latent 3D Diffusion Models
Evangelos Ntavelis" Computer Vision Lab ETH Zurich
Aliaksandr Siarohin Creative Vision Snap Inc. Santa Monica, CA, USA Asiarohin@smapchat. com
Kyle Olszewski Creative Vision Snap Inc.
Zürich, Switzerland eutavelis@visionnc www.:/tiiieeeeeee.......m ch
Santa Monica, CA, USA kolszewskk@enap.. com
Chaoyang Wang CI2CV Lab
Luc Van Gool
Sergey Tulyakov Creative Vision Snap Inc. Santa Monica, CA, USA stullyakov@anapchat.com
CVL, ETH Zurich, CH PSI, KU Leuven, BE INSAIT, Un. Sofia, BU wotgool@vision..oo ee. ethz.
Carnegie Mellon University Pittsburgh, PA, USA
chaoyanwεcs. cmu edu
Abstract
We present novel approach the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation and geometry. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and normalization operations to learn a 3D diffusion from 2D images monocular videos of rigid or articulated objects. Our approach is flexible enough to use either existing camera supervision camera information all instead efficiently learning it during training. Our evaluations demonstrate that our generation results outperform state-of-the- alternatives various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and large-scale, real video dataset of static objects.
rendering view-consistent appearance
Code & Visualizations:
1 Introduction
Photorealistic generation is undergoing period that future scholars may well compare to the enlightenment era. The improvements in quality, composition, stylization, resolution, scale, and manipulation capabilities of images were unimaginable just over year ago. The abundance of online images, often enriched with text, labels, tags, and sometimes per-pixel segmentation, has significantly accelerated such progress. The emergence and development of denoising diffusion probabilistic models (DDPMs) [75, (7 propelled these advances in image synthesis [53
78
80 76 17 18
63
36
86
and other domains, e.g. audio (TO 20 951) and video ([44 89
However, the world is 3D, consisting static and dynamic objects. Its geometric and temporal nature poses major challenge for generative methods. First of all, the data have consists mainly of images and monocular videos. For some limited categories of objects, we have 3D meshes with corresponding multi-view images videos, often obtained using tedious capturing process or created manually by artists. Second, unlike CNNs, there is no widely accepted 3D or 4D representation suitable for 3D geometry and appearance generation. As result, with only a few exceptions [74), most of the existing 3D generative methods are restricted to narrow range of object
"Work done during internship at Creative Vision Team Snap Inc
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

categories, suitable to the available data and common geometric representations. Moving, articulated objects, e.g. humans, compound the problem, as the representation must also support deformations.
In this paper, we present a novel approach to designing and training denoising diffusion models for 3D-aware content suitable for efficient usage with datasets of various scales. It is generic enough to handle both rigid and articulated objects. It is versatile enough to learn diverse 3D geometry and appearance from multi-view images and monocular videos of both static and dynamic objects. Recognizing the poses of objects in such data has proven to be crucial to learning useful 3D representations 6 7 73 74). Our approach is thus designed be robust the use of ground-truth or using no input pose information at all, but rather learning effectively during training. It scalable enough train on single- or cattory datasets of large numbers of diverse objects suitable for synthesizing wide range of realistic content.
poses, those estimated using structure- snnecturefrrmm--o
Recent diffusion methods consist of two stages [63] During the first stage, autoencoder learns rich latent space. To generate new samples, a diffusion process is trained during the second stage to explore this latent space. To train an image-to-image autoencoder, many images are needed. Similarly, training 3D autoencoders requires large quantities of 3D data, which is very scarce. Previous works used synthetic datasets such as ShapeNet [8] (DiftRF 49). SDFusion [12] etc.), and were thus restricted to domains where such data is available.
In contrast to these works, we propose to volumetric autodecoder to learn the latent space for diffusion sampling. In contrast to the autoencoder- -based approach, our autodecoder maps 1D vector to each object in the training set, and thus does not require 3D supervision. The autodecoder learns 3D representations from 21) observations, using rendering consistency as supervision. Following UVA 700 this 3D representation supports the articulated parts necessary model non-rigid objects.
There are several key challenges with learning such a rich, latent 3D space with an autodecoder. First, our autodecoders do not have clear "bottleneck." Starting with a ID embedding, they upsample it to latent features at many resolutions, until finally reaching the output radiance and density volumes. Here, each intermediate volumetric representation could potentially serve "bottleneck." " Second, autoencoder- based methods typically regularize the bottleneck by imposing a KL-Divergence constraint (38 631, meaning diffusion must be performed in this regularized space.
To identify the best intermediate representation to perform diffusion, one can perform exhaustive layer-by-layer search. This, however, is very computationally expensive, as it requires running hundreds of computationally expensive experiments. Instead, we propose robust normalization and denormalization operations which can be applied to any layers of a pre-trained and fixed autodecoder. These operations compute robust statistics to perform layer normalization and, thus, allow us to train the diffusion process at any intermediate resolution of the autodecoder. We find that at fairly low resolutions, the space is compact and provides the necessary regularization for geometry, allowing the training data contain only sparse observations of each object. The deeper layers, the other hand, operate more as upsamplers. We provide extensive analysis to find the appropriate resolution for our autodecoder-baste based diffusion techniques.
We demonstrate the versatility and scalability of our approach on various tasks involving rigid and articulated 3D object synthesis. We first train our model using multi-view images and cameras in setting similar to DifiRE 49 to generate shapes of a limited number of object categories. We then scale our model to hundreds of thousands of diverse objects train using the real-world MVImgNet [92] dataset, which is beyond the capacity of prior 3D diffusion methods. Finally, we train our model on a subset of Seleb--Text 190), consisting of ~44K sequences of high-quality videos of human motion.
2
Related Work
2.1 Neural Rendering for 3D Generation
Neural radiance fields, or NeRFs (Mildenhall et al., 2020 (48), enable high-quality novel view synthesis (NVS) of rigid scenes learned from 2D images. Its approach to volumetric neural rendering has been successfully applied to various tasks, including generating objects suitable for 3D-aware NVS. Inspired by the rapid development of generative adversarial models (GANs) 22 for generating
2D images 22 32
RK] 35
and videos [79 72 91], subsequent work extends them to 3D
content generation with neural rendering techniques. Such works (67 51
54
52
88 show promising

---

results for this task, yet suffer from limited multi-view consistency from arbitrary viewpoints, and experiencing difficulty in generalizing multi-category image datasets.
notable work in this area is pi-GAN (Chan et al., 2021 66), which employs neural rendering with
periodic activation functions for generation with fiew-consistent rendering. However, it requires precise estimate of the dataset camera pose distribution, limiting its suitability for free-viewpoint videos. In subsequent works, EG3D (Chan et al., 2022 77) and EpiGRAF (Skorokhodov al. (73) use tri-plane representations of 3D scenes created by snnerator-discrimnnan
framework based on StyleGAN2 (Karras et al., 2020 335). However, these works require pose estimation from keypoints (e.g. facial features) for training, again limiting the viewpoint range.
These works primarily generate content within one object category with limited variation in shape and appearance. A notable exception is 3DGP (74), which generalizes to ImageNet [15]. However, its reliance on monocular depth prediction limits it generating front-facing scenes. These limitations
also prevent these approaches from addressing deformable, articulated objects. In contrast, our method is applicable to both deformable and rigid objects, and covers wider range of viewpoints.
2.2 Denoising Diffusion Modeling
Denoising diffusion probabilistic models (DDPMs) 75 28 represent the generation process as the learned denoising of data progressively corrupted by sequence of diffusion steps. Subsequent works
improving the training objectives, architecture, and sampling process [28] 17 86
36
63
53 76
demonstrated rapid advances in high-quality data generation on various data domains. However, such works have primarily shown results for tasks in which samples from the target domain are fully observable, rather than operating in those with only partial observations of the dataset content.
One of the most important of such domains is 3D data, which is primarily observed in 2D images for most real-world content. Some recent works have shown promising initial results in this area. DiffRF (49 proposes reconstructing per-object NeRF volumes for synthetic datasets, then applying diffusion training on them within U-Net framework. However, it requires the reconstruction of many object volumes, and is limited to low-resolution volumes due to the diffusion training's 's high computational cost. As our framework instead operates in the latent space of the autodecoder, effectively shares the learned knowledge from all training data, thus enabling low-resolution, latent 3D) diffusion. In [12], 3D autoencoder is used for generating 3D shapes, but this method require ground-truth 3D supervision, and only focuses on shape generation, with textures added using an uff-the-shelf method [60]. In contrast, our framework learns to generate the surface appearance and corresponding geometry without such ground-truth 3D supervision.
68 23 9 combine denoising diffusion approach with a tri-plane representation 7 for 3D generation. They perform diffusion on the embedding vector of an autodecoder
Many recent works 2
pre-computed [68] or (9. Nevertheless, these works focus on small datasets or require a readily available for real-object image datasets. The triplane representation requires an MI.P decoder that substantially increases the volumetric rendering time. Our voxel-decoder does not has such requirement it directly outputs color and density, and thus permitting faster training on large-scale real image datasets.
2 the bottleneck of an autoencoder (23], or directly,
a simultaneously learned tri plane dense point clouds and ground truth object meshes, which
Recently, several works [60] 43 [ propose using large-scale, pre-trained text-to-image 2D diffusion models for 3D generation. The key idea behind these methods is to use 2D diffusion models to evaluate the quality of renderings from randomly sampled viewpoints, then use this information to optimize a 3Daaware representation of the content. Compared to our method, however, such approaches require far more expensive optimization process to generate each novel object.
3 Methodology
Our method a two-stage approach. In the first stage, we learn an autodecoder G containing library of embedding vectors corresponding to the objects in the training dataset. These vectors are first processed to create a low-resolution, latent 3D feature volume, which is then progressively upsampled and finally decoded into voxelized representation of the generated object's shape and appearance. This network is trained using volumetric rendering techniques on this volume, with 2D reconstruction supervision from the training images.
3

---

Stage Autuderadinn
Singe
Villenes
Trus
Genennal Ralance Velane
Lakes
Denaviiigg
volann
Figure I: Our proposed two-stage framework. Stage 1 trains an autodecoder with two generative components, Gi and G2. It learns assign each training set object ID embedding that is processed by G1 into latent volumetric space. G2 decodes these volumes into larger radiance volumes suitable for rendering. Note that are using only 2D supervision to train the autodecoder. In Stage 2, the autodecoder parameters are frozen. Latent volumes generated by G1 are then used to train the 3D denoising diffusion process. At inference time, G₁ is not used, as the generated volume is randomly sampled, denoised, and then decoded by G2 for rendering.
During the second stage, we split the autodecoder G into two parts, G G2 G1. We then employ this autodecoder to train a 3D diffusion model operating in the compact, 3D latent space obtained from Gi. Using the structure and appearance properties extracted from the autodecoder training dataset, this 3D diffusion process allows us to use this network to efficiently generate diverse and realistic 3D content. The full pipeline is depicted in Fig.
3.1 We then describe the training procedure and reconstruction losses for the autodecoder (Sec. 3.2) Finally, provide details
Below, we first describe the volumetric autodecoding architecture (Sec.
for our training and sampling strategies for 3D diffusion in the decoder's s latent space (Sec. 3.3
3.1 Autodecoder architecture
Canonical Representation. We use 3D voxel grid to represent the 3D structure and appearance of an object. We assume the objects are in their canonical pose, such that the 3D representation is decoupled from the camera poses. This decoupling is necessary for learning compact representations of objects, and also serves as a necessary constraint to learn meaningful 3D structure from 2D images without direct 3D supervision. Specifically, the canonical voxel representation consists of density grid / DDeminyy e RS* which discrete representation of the density field with resolution S and VRGB E RR5*x3 which represents the RGB radiance field. We employ volumetric rendering, integrating the radiance and opacity values along each view ray similar to NeRFs [48). In contrast to the original NeRF, however, rather than computing these local values using an MLP, we tri- tri-linearly interpolate the density and RGB values from the decoded voxel grids, similar to Plenoxels [64].
The 3D voxel grids for density and radiance, Denssty and /RGB are generated by volumetric autodecoder G that is trained using rendering supervision from 2D images. We choose to directly generate Density and RRGB rather than intermediate representations such as feature volumes or tri- tri-planes, as it is more efficient to render and ensures consistency across multiple views. Note that feature volumes and tri-planes require running an MLP pass for each sampled point, which requires significant computational cost and memory during training and inference.
Voxel Decoder.
The decoder learned in the manner of GLO 4 across various object categories from large-scale multi-view or monocular video datasets. The architecture of our autodecoder is adapted from [70]. However, in our framework we want to support large scale datasets which poses challenge in designing the decoder architecture with the capability to generate high-quality 3D) content across various categories. In order to represent each of the ~30K objects our largest dataset we need very high-capacity decoder. As we found the relatively basic decoder of (70) produced poor reconstruction quality, we introduce the following key extensions:
* To support the diverse shapes and appearances in target datasets, we find it crucial to increase the length of the embedding vectors learned by our decoder from 64 to 1024.
We experimented with diffusion at different feature volume resolutions, ranging from 4 at the earliest stage 16 in the later stages. These results are described in evaluations (Sec. 4.3 Fig

---

We increase the number of residual blocks at each resolution in the autodecoder from to 4. Finally, to harmonize the appearance the reconstructed objects we introduce self-attention in the second and third levels (resolutions 8 and 16%).
layers 81
Scaling the Embedding Codebook for Large Datasets. Each object in the training set is encoded by an embedding vector. However, storing separate vector for each object is burdensome, especially for large datasets. As such, we propose a technique to significantly reduce the parameter footprint of our embeddings, while allowing effective generation from large-scale datasets.
Similar to StyleGenes' approach [35], we combine smaller embedding subvectors to create unique per-object vectors. The decoder's s input a per-object embedding vector Hş IR' with length L. It is concatenation of smaller subvectors h2: where each subvector is selected from ordered codebook with ne entries, with each entry containing collection of ny embedding vectors of length Ly/ nee
H
h4
(1)
where K; {1, n } is the set of indices used to select from the n, possible codebook entries for position i in the final vector. This method allows for exponentially more combinations of embedding vectors, greatly reducing the number of stored parameters compared a single embedding codebook. In contrast to (55), the index j for the vector hig at position is not randomly selected for each
position to access its corresponding codebook entry. We use hashing function [16] to map each
training object index & to its corresponding embedding index.
3.2 Autodecoder Training
We train the decoder from image data through analysis-by-syntteer with the primary objective of minimizing the difference between the decoder's 's rendered images and the training images. We render RGB color image C' using volumetric rendering [48), additionally in order to supervise silhouette of the objects we render 2D occupancy mask O.
Pyramidal Perceptual Loss. As in 669 70, we employ a pyramidal perceptual loss based on on the rendered images as our primary reconstruction loss:
31
f. ΣΣ 1=U 1=
Lreo Cr
VCC,( -VG.D//)CC))VVV......))).
where é, C E [0. |0 Nx are the RGB rendered and training images of resolution H x W, respectively; VGG, is the layer of a pre-trained VGG-19 771 network; and operator D downsamples images to the resolution for pyramid level (.
Foreground Supervision. Since only interested in modeling single objects, in all the datasets considered in this work, we remove the background. However if the color of the object is black (which corresponds the absence of density), the network can make the object semi-transparent. To improve the overall shape of the reconstructed objects, we make use of a foreground supervision loss. Using binary foreground masks (estimated by off-the-shelf matting method (44), Segment Anything (39 or synthetic ground-truth masks, depending on the dataset), we apply an L. loss on the rendered occupancy map match that of the mask corresponding the image.
Leeg (Ō,
HW 10 1
(3)
where Ô,0
= are the inferred and ground-truth occupancy masks, respectively, We provide visual comparison of the inferred geometry for this loss in the supplement.
Multi-Frame Training. Because our new decoder have a large capacity, generating a volume incur much larger overhead compared to rendering an image based on this volume (which mostly consists of tri-linear sampling of the voxel cube). Thus, rather than rendering a single view for the canonical representation of the target object in each batch, we instead render views for each object in the batch. This technique incurs no significant overhead, and effectively increases the batch size four times. As
5

---

an added benefit, we find that this technique improves on the overall quality of the generated results, since it significantly reduce batch variance. We ablate this technique and our key architectural design choices, showing their effect on the sample quality (Sec. 4.3
Tab. 2
Learning Non-Rigid Objects. For articulated, non-rigid objects, e-g. videos of human subjects, must model a subject's shape and local motion from dynamic poses, as well as the corresponding non- rigid deformation of local regions. Following [70), we assume these sequences can be decomposed into a set of N, smaller, rigid components (10 in our experiments) whose poses can be estimated for consistent alignment in the canonical 3D space. The camera poses for each component are estimated and progressively refined during training, using combination of learned 3D keypoints for each component of the depicted subject and the corresponding 2D projections predicted in each image. This estimation is performed via a differentiable Perspective- ---intt (PnP) algorithm [40].
To combine these components with plausible deformations, employ learned volumetric linear blend skinning (LBS) operation. We introduce a voxel grid VLBS E R55*N,, to represent the skinning weights for each deformation components. As we assume no prior knowledge about the content or assignment of object components, the skinning weights for each component are also estimated during training. Please see the supplement for additional details.
3.3 Latent 3D Diffusion
Architecture. Our diffusion model architecture extends prior work on diffusion in 2D space [36 to the latent 3D space. We implement its 2D operations, including convolutions and self atteetttntonn layers, in our 3D decoder space. In the text-conditioning experiments, after the self-attention layer, we use a cross-attention layer similar to that of 63 Please see the supplement for more details.
Feature Processing. One of our key observation is that the features F in the latent space of the 3D autodecoder have a bell-shaped distribution (see the supplement), which eliminates the need to enforce any form of prior on it, e.g. in [63]. Operating in the latent space without a prior enables training a single autodecoder for each of the possible latent diffusion resolutions. However, we observe that the feature distribution F has very long tails. We hypothesise this is because the final density values inferred by the network do not have any natural bounds, and thus can fall within any range. In fact, the network is encouraged make such predictions, as they have the sharpest boundaries between the surface and empty regions. However, to allow for a uniform set of diffusion hyper-parametert for all datasets and all trained autodecoders, the same range. This is equivalent to computing the center and the scale of the distribution. Note that, due to the very long-tailed feature distribution, typical mean and standard deviation statistics will be heavily biased. We thus propose a robust alternative based on the feature distribution quantiles. We take the median m as the center of the distribution and approximate its scale using the Normalized InterQuartile Range (IQR) 85 for a normal distribution: 0. 0.7413 x IQR. Before using the features F for diffusion, we normalize them to F [F-m) During inference, when producing the final
must normalize their features into
TQR
volumes we de-normalize them as F x TQR mn. We call this method robust normalization. Please see the supplement for an evaluation of its impact.
Sampling for Object Generation. During inference we rely on the sampling method from EDM (36, with several slight modifications. We fix EDM's hyperparameter matching the dataset's 's distribution to 0.5 regardless of the experiment, and modify the feature statistics in our feature processing step. We also introduce classifirr free guidance (27) for our text-condi experiments (Sec.4.5 (45). We found that setting the weight equal to 3 yields good results across all datasets.
4 Results and Evaluations
In this section, we evaluate our method on multiple diverse datasets (see Sec. 4.1 for both unconditional 4.2 conditional settings/ We also ablate the design choices in our autodecoder and diffusion Secs. 4.3 and respectively.

---

PhotoShape Chairs [57
ABO Tables
13
Model Variant
PSNR 1
LPIPS
Method
FID
KID
FID 4
KID 4
27.719
6.255
i--AN 6
52.71
13.64
41.67
13.81
Multi-Framee Training
27.176
6.855
RG3D
16.54
8.412
31.18
11.67
Self-Attettonn
27.335
6,738
DifRR 49
15.95
7.935
27.06
10.03
Increased Depth
27.24
6,924
EEmeediiig Length (1024 -> 64)
25.985
8.332
Ours
11.28
4.714
18.44
6.854
Table 2:
Our 3D autodecoder ablation results, 'L"
Table 1: Results on the synthetic PhotoShape Chairs [57]
indicates this component has been removed. As we remove each sequentially. the top row depicts outperforms state-of-the-art GAN-based and diffusion-based results for the unmodified architecture and training
and ABO Tables [13] datasets. Overall. our method
approaches. KID scores are multiplied by 10
procedure. LPIPS results are multiplied by 10a.
Generated Novel Views
Generated Novel Views
20
10"
0"
10"
20'
Depth Normals
200
10"
O"
16"
20'
Depth Normals
Driving
Driving
DES
Figure 2: Qualitative comparisons with Direct Latent Sampling (DLS) 70 on CelebV [90]. We show the two driving vidcos for two random identities; the top identity in each block is generated by our method, the bottom identity in each block is generated by DLS 70).
We also show the rendered depth and normals.
4.1 Datasets and Data Processing
Below we describe the datasets used for our evaluations. We mostly evaluate our method on datasets
of synthetic renderings of 3d objects 11
57 14] However, also provide results on a challenging
video dataset of dynamic human subjects 900 and dataset of static object videos [92].
ABO Tables. Following 449
evaluate approach renderings of objects from the Tables subset of the Amazon Berkeley Objects (ABO) dataset (13), consisting of 1, 676 training sequences with 91 renderings per sequence, for total of 152, 516 renderings.
PhotoShape Chairs. Also as [49 we use images from the Chairs subset of the PhotoShape dataset (57, totaling 3, 115, 200 frames, with 200 renderings for each of 15, 576 chair models.
Objaverse. This dataset [14] contains ~00K publicly available 3D models. As the of the object geometry and appearance varies, we use a manually-filtere subset of ~300K unique objects (see supplement for details). We render 6 images per training object, for total of 11.8 million frames.
MVmgNNet. For this dataset 922,, we use ~-655 million frames from 219, 188 videos of real-world objects from 239 categories, with an average of 30 frames each. We use Grounded Segment Anything [45 39 for background removal, then apply filtering (see supplement) to remove objects with failed segmentation. This process results in 206, 990 usable objects.
V-eet. The CelebV-Text dataset 90) consists of 700 sequences of high-quality videos of celebrities captured in in-the-wild environments, lighting, motion, and poses. They generally depict head, neck, and upper-torso region, but contain more challenging pose and motion variation than prior datasets, e.g. VoxCeleb [50]. We use the robust video matting framework of [44] to obtain our masks for foreground supervision (Sec. (.2c. Some sample filtering (described in the supplement) was needed for sufficient video quality and continuity for training. This produced 444.4K unique videos, with an average of ~ 373 frames each, totaling ~ 16.66 frames.
For training, we use the camera parameters used to render each synthetic object dataset, and the estimated parameters provided for the real video sequences in MVImgNet, adjusted center and scale the content to our rendering volume, (see supplement for details). For the human videos in CelebV-Text. we train an additional pose estimator along with the autodecoder G to predict poses for each articulated region per frame, such that all objects can be aligned in the canonical space (Sec. 3.2) Note that for creating dynamic 3D video, we can use sequences of poses transferred from the real video of another person from the dataset.
7

---

yellow dog"
"This object green skall"
blue with sumber the side"
"This is blue and white vase"
toy as green toddy
"This tey red tarpot"
"This shows. woring black wit and rod lic."
"The singlasees white with orange Iemess"
Figure 3:
We show generated samples of our model trained using rendered images from Objaverse 14 show three views for each object, along with the normals for each view. We also show depth for the right-most view. Text-conditioned results are shown. Grouth-truth captions are generated by MiniGPT-4 [94]
"The coffee mug white with circular handle"
"The soy tuttle is brown with yellow spots oa its shell"
"The hammer a red and silver tool with flat head""
"The kowi is green fruit with brown outsie"
"The iodely
beige ches!"
"The bus mubierr duck las rund body and Fat bolton"
shape"
Figure 4: We show generated samples from our model trained using monocular vidcos from MVImgNet [2]] We show three views for cach object, along with the normals for each view. We also show depth for the right-most view. Text-conditioned results shown. Ground-truth captions generated MiniGPT-4 94
4.2 Unconditional Image Generation
Synthetic Datasets. Following the evaluation protocol of 49 and PhotoShape Chairs datasets. These results on single-category, synthetically rendered datasets that are relatively small compared to the others, demonstrate that our approach also performs well with smaller, more homogeneous data. We render 10 views of 1K samples from each dataset, and report the Frèchet Inception Distance (FID) (26) and Kernel Inception Distance (KID) (3) when compared to 10 randomly selected ground -ondd-truuu -trtt images from each training sequence. We report the results compared to both GAN-based [6 7 and more recent diffusion-basee approaches [49] methods, as seen in Tab. We see that our method significantly outperforms sate-of-the-ar methods using both metrics on the Tables dataset, and achieves better or comparable results on the Chairs dataset.
report results on the ABO Tables
Large-Scale Datasets. We run tests on the large-scale datasets described above: MVImgNet. CelebV-Text and Objaverse. For each dataset, we render images from random poses for each of 10K generated samples. We report the FID and KID for these experiments compared 5 ground-truth images for each of 10K training objects. As no prior work demonstrates the ability to generalize to such large-scale datasets, we compare our model against directly sampling the ID latent space of our base autodecoder architecture (using noise vectors generated from standard normal distribution). This method of 3D generation was shown work reasonably well 77)
also evaluate our approach with different numbers of diffusion steps (16, 32 and 64). The results can be seen in Tab. 3
Visually, we compare with [70] Fig. 2 Our qualitative results show substantially higher fidelity. quality of geometry and texture. We can also see that when identities sampled directly in the ID

---

latent space, the normals and depth are significantly less sharp. indicating that there exist spurious density in the sampled volumes. Tab. further supports this observation: both the FID and KID are significantly lower than those from direct sampling, and generally improve with additional steps.
4.3 Autodecoder Ablation
Cele [90
MVImgNet
[2
Objaverse [
We conduct ablation study the key design choices for our autodecoder archi- tecture and training. Starting with the final version, we subtract the each component described in Sec. 3.11 We then train a model on the PhotoShape Chairs dataset and render 4 images for each of the 155.5K object embeddings.
Method
FID
KID
FID
KTD
FID
KTD
Sumpling 770
Direct Latent
69.21
73.74
97.51
69.22
72.76
53.68
Ours Steps
48.01
49.49
62.21
39.94
47.49
32.44
Ours Steps
49.74
46.2
51.26
28.45
43.68
31.7
Ours 64 Steps
50.27
47.72
43,85
23.91
40.49
29,37
Table 3: Results on large-scale multi-view image (Obja-
verse 14] & MVImggee (92) and monocular video (CelebV- Text (90)) datasets. The KID score is multiplied by 10
Tab. 2 provides the the PSNR 30 and
LPIPS [93 reconstruction metrics. We find that the final version of our process significantly outperforms the base architecture (70) and training process. While the largest improvement comes from our increase in the embedding size, we see that simply removing the multi-frame training causes noticeable drop in quality by each metric. Interestingly, removing the self-attention layers marginally increases the PSNR and lowers the LPIPS. This is likely due to the increased complexity in training caused by these layers, which for a dataset of this size, may be unnecessary. For large-scale datasets, we observed significant improvement with this feature. Both decreasing the depth of the residual convolution blocks and reducing the embedding size cause noticeable drops in the overall quality. particularly the latter. This suggests that the additional capacity provided by these components impactful, even on a smaller dataset.
60
16
4x4x4 Latent Volume
4.4 Diffusion Ablation
55
Latent Volume
16
16x16x16 Latent Volume
We also perform ablation on our diffusion process, evaluating the effect of the choice of the number of diffusion steps (16, 32, and 64), and perform
50
32
64
32
45
the autodecoder resolution at which diffusion (4S, 8 and 16 For these variants, follow the generation quality training and evaluation protocol on the PhotoShape Chairs (Sec. 4.2). except that we disable stochasticity in our sampling during inference for more consistent performance across these tests. Each model was trained using roughly the same amount of time and computation. Fig. sJsoos the results. Interestingly, we can see a clear distinction between the results obtained from diffusion at the earlier or later autodecoder stages, and those from our the results with resolution 83. We hypothesize that at lowest resolution layers overfit to the training dataset, thus when processing novel objects via diffusion, the quality degrades significantly. Training at higher resolution requires substantial resources, limiting the convergence seen in a reasonable amount of time. The number of sampling steps has smaller, more variable impact. Going from 16 to 32 steps improves the results with a reasonable increase in inference time, but at 64 steps, the largest improvement is at the 163 resolution, which requires more than 30 seconds per sample. Our chosen diffusion resolution of 8 achieves the best results, allowing for high sample quality at 64 steps (used in our other experiments) with only ~ seconds of computation, but provides reasonable results with 32 steps in 44 seconds.
40
35
30
25
64
Inference time per Sample (seconds)
Figure 5: Impact of diffusion resolution and number of sampling steps on sample quality and inference time.
4.5 Conditional Image Generation
Finally, we train diffusion models with text-conditioning. For MVImgNet and Objaverse, we generate the text with an off-the-shel captioning system 194). Qualitative results for MVImgNet and Objaverse are in Figs. 4 and 3. We observe that in all cases, our method generates objects with reasonable geometry that generally follow the prompt. However, some details can be missing. We believe our

---

model learns to ignore certain details from text prompts, as MiniGPT-4 often hallucinates details inconsistent with the object's appearance. Better captioning systems should help alleviate this issue.
4.6 Design Choices for Large-Scale 3D Object Synthesis
The goal of our work is to enable 3D object synthesis by training a model on large and diverse multi-view image datasets. To realize this goal there are two main design choices that we need to make: (a) what is the appropriate 3D representation and (b) generative modelling approach.
use tri-planes their 3D representation. However, when the multi-view supervision is scarce and ground truth camera information is not available, such as in video datasets like CelebV, tri-planes tend to degrade to prediction of flat objects [70). Moreover, tri tri-planes require an additional MLP for volumetric rendering, applied for every of the 128 ray point samples and 1282 output pixels and density. Tri-planes are faster to autodecode, but rendering them is much slower. Training for an iteration with our 64 voxel grid takes 0. 0.22ss. Tri-planes of size 642, 128 256² and 5122 require 0.33, 0.33, 0.38, 0.46 seconds respectively. We use 32 channels per plane, a two-layer MLP with 32 hidden channels, and a 2D autodecoder. This can severely affect the training time for large dataset.
Recent works 2
68 23
[
in our setting. In contrast, our voxel grid autodecoder outputs directly color
EG3D 7 and GET3D 21 propose an adversarial approach to 3D synthesis. Both, base their generators on StyleGAN 33 which for 2D datasets requires considerable changes to produce good results in large and diverse datasets [65]. Training both on Objaverse, we find they fail to converge, as seen in Fig. Thus, we believe our diffuusion-oe basedd approach is better suited for our goal.
5 Conclusion
Despite the inherent challenges in performing flexible 3D content generation for arbitrary content domains without 3D supervision, our work demonstrates this is possible with the right approach. By exploiting the inherent power of autodecoders synthesize content in domain without corresponding encoded input, our method learns representations of the structure and appearance of diverse and complex content suitable for generating high-fidelity 3D objects using only 2D supervision. Our latent volumetric representation is conducive to 3D diffusion modeling for both conditional and unconditional generation, while enabling consisesttt rendering of the synthesized objects. As seen in our results, this generalizes well to various types of domains and datasets, from relatively small, single-category, synthetic renderings to large-scale, multi-category real-world datasets. It also supports the challenging task of generating articulated moving objects from videos. No prior work addresses each of these problems in single framework. The progress shown here suggests there is potential to develop and extend our approach to address other open problems.
Limitations. While demonstrate impressive and state-of-the-art art results diverse tasks and content, several challenges and limitations remain. Here we focus on images and videos with foregrounds depicting one key person or object. The generation or composition of more complex, multi-ohject scenes is a challenging task and an interesting direction for future work. As we require multi-view or video sequences of each object in the dataset for training, single-image datasets are not supported. Learning the appearance and geometry of diverse content for controllable 3D generation and animation from such limited data is quite challenging, especially for articulated objects. However, using general knowledge about shape, motion, and appearance extracted from datasets like ours to reduce or remove the multi-image -mage requirement when learning to generate additional object categories may be feasible with further exploration. This would allow the generation of content learned from image datasets of potentially unbounded size and diversity.
Broader Impact. Our work shares similar concerns with other generative modeling efforts, e.g., potential exploitation for misleading content. As with all such learning-based methods, biases in training datasets may be reflected in the generated content. Appropriate caution must be applied when using this method to avoid this when it may be harmful, e.g. human generation. Care must be taken to only use this method on public data, as the privacy of training subjects may be compromised our framework is used to recover their identities. The environmental impact of methods requiring substantial energy for training and inference is also concern. However, our approach makes our tasks more tractable by removing the need for the curation and processing of large-scale 3D datasets, and is thus more amenable to efficient use than methods requiring such input.
10

---

Ackmowledgements We would like to thank Michael Vasilkovsky for preparing the Obja Verae renderings, and Colin Eles for his support with infrastructure. Moreover, we would like to thank Norman Müller, author of DiffR paper, for his invaluable help with setting up the DiffRF baseline, the ABO Tables and PhotoShape Chairs datasets, and the evaluation pipeline as well as answering all related questions. A true marvel of a scientist. Finally, Evan would like to thank Claire and Gio for making the best cappuccinos and fueling up this research.

---

References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning Representations and Generative Models for 3D Point Clouds. In Proceedings of the International Conference on Machine Learning, 2018
[2] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talhott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Josh Susskind. Gaudi: A neural architect for immersive 3d scene generation. Proceedings of the Neural Information Processing Systems Conference, 2022.
[3] Mikoaj Bikowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2018.
[4] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. In arXiv, 2017.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In arXiv, 2018.
[6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[7] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient Geometry-aware 3D Generative Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[8] Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- Rich 3D Model Repository. In arXiv, 2015.
[9] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEEE International Conference on Computer Vision, 2023.
[10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[11] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. In arXiv, 2023.
[12] Yen-Chi Cheng, Hsin- Sing Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In Procediigss of the IEEF Conference on Computer Vision and Patiern Recognition, 2023.
[13] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthiuu Guillaumin, and Jitendra Malik. ABO: Dataset and Benchmarks for Real-World 3D Object Understanding. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, 2022.
[14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Universe of Annotated 3D Objects. In arXiv, 2022.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[16] Srini Devadas and Konstantinos Daskalakis. MIT 6.006, Lecture 5: Hashing I: Chaining, Hash Functions, 2009.
[17] Prafulla Dhariwal and Alexander Nichol, Diffusion Models Beat Gans on Image Synthesis. In Proceedings of the Neural Information Processing Systems Conference, 2021.
[18] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
12

---

[19] William Falcon et al. Py PyTorc Lightning. Gitflub. Note: lightning, 3, 2019.
[20] Seth* Forsgren and Hayk* Martiros. Riffusion Stable diffusion for real-time music generation, 2022. URL httpss//rriffuiiinn. www.:ooccooooooee about
[21] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. In arXiv, 2022.
[22] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the Neural Information Processing Systems Conference, 2014.
[23] Anchit Gupta and Anchit Gupta. 3dgen: Triplane latent diffusion for textured mesh generation. ArXiv, abs/2303.05377 2023, URL /tttts/p:://appp semanticscholarr Satgg/CorpusID... 257 257427345
[24] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood. Flexible Diffusion Modeling of Long Videos. In Proceedings of the Neural Information Processing Systems Conference, 2022.
[25] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. In arXiv. 2023.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In Proceedings of the Neural Information Processing Systems Conference, 2017.
[27] Jomathan Ho and Tim Salimans. Classifier-free difflusion guidance. In arXiv, 2022.
[28] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the Neural Information Processing Systems Conference, 2020.
[29] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
[30] A. Horé and D. Ziou. Image quality metries: Psnr ssim. In Proceedings of the International Conference on Pattern Recognition, 2010.
[31] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proceedings of the European Conference on Computer Vision, 2016.
[32] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lchtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
[33] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[34] Tero Karras, Miika Aittala, Janne Hellsten. Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In arXiv, 2020.
[35] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[36] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion Based Generative Models. In Proceedings of the Neural Information Processing Systems Conference, 2022.
[37] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[38] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In arXiv, 2023.
13

---

[40] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. EPnP: An Accurate O(n) Solution to the PnP Problem. In International Journal Computer Vision, 2009.
[41] J. P. Lewis. Matt Cordner, and Nickson Fong. Pose Space Deformation: A Unified Approach to Shape Interpolation and Aaeleton-Driven Deformation. ACM Transactions on Graphics, 2000.
[42] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-Adjusting Neural Radiance Fields. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
[43] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3D: High-Resolution Text-to-3D Content Creation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[44] S. Lin, L. Yang. L. Saleemi, and S. Sengupta. Robust High-Resolution Video Matting with Temporal Guidance. In Proceedings of the Winter Conference on Applications of Computer Vision, 2022.
[45] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jic Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv, 2023.
[46] William F. Lorensen and Harvey E. Cline. Marching Cubes: High Resolution 3D Surface Construction Algorithm. In ACM Transactions on Graphics, 1987.
[47] Kangfu Mei and Vishal M. Patel. VIDM: Video Implicit Diffusion Models. In Association for the Advancement of Artificial Intelligence Conference, 2023.
[48] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as Neural Radiance Fields for View Synthesis, Proceedings of the European Conference on Computer Vision, 2020.
[49] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias NieSSner. DiftRF: Rendering Senidring-Guided 3D Radiance Field Diffusion. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2023.
[50] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. VoxCeleb: Large-scale speaker verification in the wild. Computer Science and Language. 2019.
[51] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. HoloGAN: Unsupervised Learning of 3D Representations From Natural Images. In Proceedings of the IEEE International Conference Computer Vision, 2019.
[52] Thu Nguyen- Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. In arXiv, 2020.
[53] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[54] Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. In Proceedings the IEEE Conference on Computer Vision and Pottern Recognition, 2021.
[55] Evangelos Ntavelis, Mohamad Shahbazi, Iason Kastanis, Radu Timofte, Martin Danelljan, and Lue Van Gool. StyleGenes: Discrete and Efficient Latent Distributions for GANs. In arXiv, 2023.
[56] Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl. and Elvis Yu-Jing Lin. High-fidelity performance metrics for generative models in PyTorch, 2020. URL httpa:/tt http///ttttttttteee com/ toshae seehas/torch-fideliss Version: 0.3.0, DOI: 10.5281 12280oo20oo..
[S7] Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M. Seitz. PhotoShape: Photorealistic Materials for Large-Scale Shape Collections. In ACM Transactions on Graphics, 2018
[58] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,o Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic Differentiation in PyTorch, 2017.
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style. Hligh-Pefformnne Deep Learning Library. In Proceedings of the Neural Information Processing Systems Conference, 2019.
14

---

[60] Poole, Ben and Jain. Ajay and Barron, Jonathan T. and Mildenhall, Ben. Dreamfusion: Text-to-3d using 2d diffusion. In arXiv, 2022.
[61]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer I Learning with Unified Text-to-Text Transformer. In The Journal of Machine Learning Research, 2020.
[62] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D Deep Learning with py Torch3D. In arXiv, 2020.
[63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. Sigh-Rcsolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2022.
[64] Sara Fridovich-Kei and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2022.
[65] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In Special Interest Group on Computer Graphics and Interactive Techniques. 2022.
[66] Johannes Lutz Schönherger and Jan-Michael Frahm. Stracture-from-Mootia Revisited, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2016.
[67] Katja Schwarz, Yiyi Liao, Michacl Niemeyer, and Andreas Geiger. GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. In Proceedings of the Neural Information Processing Systems Conference, 2020.
[68] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplanc diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[69] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First Order Motion Model for Image Animation. In Proceedings of the Neural Information Processing Systems Conference, 2019.
[70] Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Hsin- Ying Lee, Jian Ren. Menglei Chai, and Sergey Tulyakov. Unsupervised Volumetric Animation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[71] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In arxiv, 2014.
[72] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny, Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[73] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. EpiGRAF: Rethinking Training of 3D GANs. In Proceedings of the Neural Information Processing Systems Conference, 2022.
[74] Ivan Skorokhodov. Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin- Ying Lee, Peter Wonka, and Sergey Tulyakov. 3D Generation ImageNet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[75] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[76] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
177) Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the Neural Information Processing Systems Conference, 2019.
[78] Yang Song, Jascha Score-based generative modeling through stochastic differential equations. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, 2021.
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
[79] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator is what you need for rish-resotution video synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
15

---

[80] Arash Vahdat, Karsten Kreis, and Jan Kautz, Score-based generative modeling in latent space. In Proceedings of the Neural Information Processing Systems Conference, 2021.
[81] Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia. Attention is all you need. In Proceedings the Neural Information Processing Systems Conference, 2017.
[82] Vikram Voleti, Alexia Aalicoeur-Martineall and Christopher Pal. MCVD: Masked Conditional Video Diffusion for Prediction. Generation, and Interpolation. In Proceedings of the Neural Information Processing Systems Conference, 2022.
[83] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--- Neural Radiance Fields Without Known Camera Parameters. In arXiv, 2021.
[84] Chung- Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan Barron, and Ira Hemellmacher-Shlizeeee HumanNeRR: Free-Vies Viewpoint Rendering of Moving People from Monocular Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[85] Dewey Lonzo Whaley III. The Interquartile Range: Theory and Estimation. PhD thesis, East Tennessee State University, 2005.
[86] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs, In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
[87] Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian. Wanli Ouyang. Ping Luo, and Xiaogang Wang. Pose for Everything: Towards Category-Agnoste Pose Estimation. In Proceedings of the European Conference on Computer Vision, 2022.
[88] Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jac Lee. GIRAFFE HD: A Resolutco 3D-aware Generative Model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[89] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni, Zhengyuan Yang. Linjic Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming. Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation. In arXiv, 2023.
[90] Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. CelebV- Text: A Large-Scale Facial Text- Video Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[91] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wer Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awars implicit generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[92] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han, MeImgNet: A Large-scale Dataset Multi-view Images. In arXiv, 2023.
[93] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[94] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In arXiv, 2023.
[95] Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for modal music and image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
16

---

A Additional Experiments and Results
A.1 Geometry Generation Evaluation
Following the point cloud evaluation protocol of [. we measure the Coverage Score (COV) and the Minimum Matching Distance (MMD) for points sampled from generated density volumes. Given a distance metric for two point clouds X and Y, e.g. the Chamfer Distance (CD),
CD(X,Y)
E min | ZEX
min o - v1l2: v112 + Σ BEY
NEY
ZEX
COV measures the diversity of the generated point cloud set Sy. with respect to a reference point clout set S,. by finding the closest neighbor in the reference set to each one in the sample set, and computing the fraction of the reference set covered by these samples:
(S. [aar MargminyeS Es, CD(X, E S.yl
(5)
S
MMD, in contrast, measures the the overall quality of these samples, by measuring the average distance between each sampled point cloud and its closest neighbor in the reference set:
MMD S,)
Σ min CD(X,Y). YES,
(6)
St
XES,
We compute these metrics for the PhotoShape Chairs and ABO Tables datasets, comparing our generated results to points sampled from the the same reference meshes used in the data splits from the evaluations in DifRR [4]] For each generated object, we sample 2048 points from mesh extracted from the decoded density volume DDensityy (see Sec. 3.1) using the Marching Cubes [46] algorithm. We use a volume of resolution 64 and 128 for training the Chairs and Tables models, respectively. However, we note that downsampling these density volumes 323, as used in DiffRE, before applying this point-sampling operation did not noticeably impact the results of these evaluations.
The results can be seen in Tab. alongside the perceptual metrics from the main paper. Interestingly. these results show that, despite the increased flexibility of our approach, and DiftRF's restrictive use of both 2D rendering and 3D supervision on synthetic data when training their diffusion model, we obtain comparable or superior geometry compared to their approach, while substantially increasing the overall perceptual quality for these datasets. We also substantially outperform prior state-of-the -hh----rrttt methods across both perceptual and geometric comparisons with
approaches using G-N-based 6 [ these metrics.
Figs. Ees.7and and [8 using our method and DiffRF for each of these datasets. In each case, it is clear that for similar objects, our method produces more coherent and complete shapes without missing features, e.g. legs, and textures that are more realistic and detailed, leading to better and more consistent image synthesis results.
show qualitative comparisons between the unconditional generation results rendered
A.2 Foreground Supervision
For some datasets with foregrounds with complex and varying appearance which can casily be mixed with the background environment, found it necessary supplement our primary autodecoder reconstruction loss (Sec. 3.2) with an additional foreground supervision loss. This loss measures how well depicted objects are separated from the background during rendering. To evaluate the effect of this foreground supervision, we ran experiments on the CelebV -ext [90] dataset both with and without this loss. We conduct training until the autodecoder has seen total of 9 million frames from the training set, then reconstruct examples from the learned embeddings.
The result can be seen in Fig- 6 As depicted, the reconstructions without foreground supervision not only lack fidelity to the target appearance, but the estimated opacity and surfaces normals clearly show that the overall geometry is insufficiently recovered.
17

---

PhotoShape Chairs [57
ABO Tables [3
Method
FID |
KID
COV f
MMD
FID І
KID
COV f
MMD
T-GAN 66
52.71
13.64
39.92
7.387
41.67
13.81
44.23
10.92
EG3D 7
16.54
8,412
47.55
5.619
31.18
11.67
48.15
9.327
DifIRF
49
15.95
7.935
58.93
4.416
27.06
10.03
61.54
7.610
Ours
11.28
4.714
64.20
4.445
18.44
6.854
60.25
6.684
Table 4: Quantitative comparison of unconditional generation on the PhotoShape Chairs [57] and ABO Tables [33 datasets. Our method achieves better perceptual quality, while maintaining similar geometric quality to the state-of-the-ar diffusion-basee approaches. MMD and KID scores are multiplied by 103.
With Foreground Mask Supervision
Without Foreground Mask Supervision
Original
Mask
Recon
Opacity
Normals
Original
Mask
Recon
Opacity
Normals
Figure 6: In real video datasets, e.g. CelebV-Tex( (90, we have diverse set of foreground shapes and textures with common background color. In these cases, find that supervising the autodecoder with a foreground mask loss is important for the network to properly learn the shape of the object. Both examples shown after training for ~9 million frames.
A.3 Training the autodecoder on a single view per object
We conduct an experiment on training our autodecoder on the Chairs (57] dataset but only using a single view per object. To do so, we pick and train on a single rendering view among the 200 available, which is different for each object. Our autodecoder learns, nevertheless, to encode general geometry information and roughly infer the full 3D shape, of the object by utilizing information learned from other objects. However, stills struggles with the texture details, and thus training with more training views is beneficial. We show the results on Fig. 9
A.4 Training diffusion model the embedding vector of the autodecoder
In the ablation section of our main paper, we experimented on different intermediate representation resolutions train diffusion model In Fig. 10 we show results of training only the single vector we use as input to the autodecoder. Similarly to our x x 4 run, we find the generates samples to be of low quality, failing properly capture the geometry of the objects of the Chair dataset [57]. We hypothesize this compressed vector overfits to the training dataset, and diffusion struggles to fully capture the properties of the dataset distribution.
A.5 Large and Diverge 3D Synthesis with Adversarial Methods
Adversarial 3D generation methods such as EG3D 7 and GET3D (1] have shown generative results of high perceptual quality when trained on single object categories. We experimented with trained them on Objaverse [14] our biggest and most diverse datasets, without utilizing any conditioning. In this setting, these methods fail to produce meaningful samples as we see Fig. 11 In both cases, we see that the results stop improve after a few epochs, with the FID never falling below 100, compared to our methods 40.49. We believe that without additional supervision such as class labels, these methods based methods will not produce good results.
18

---

DiffRF
Ours
Figure 7: Qualitative comparison of unconditional generation using DitRR 49 (left) and our approach
(right) on the ABO Tables dataset [13]. In contrast to DifREF, we train diffusion in the latent features of an autodecoder. Decoupling the expensive and demanding training from the output voxel-grid size lets us increase the resolution of our 3D representation. For this dataset, our output voxel resolution 128 compared the 32 resolution of DifRRE Our method improves the perceptual quality of the results, as as shown the reported FID and KID.
DiffRF
Ours
Figure 8: Qualitative comparison of unconditional generation using DifRR 44 (left) and approach (right) on the PhotoShapes Chairs dataset (37]. For this dataset, our output voxel resolution 643. above, our results are both qualitatively and quantitatively superior.
19

---

Figure Training the autodecoder on a single view per object Chairs dataset [57 offers compressed representation of the dataset; it encapsulates prior knowledge. In contrast to a single-scene NeRF, our method can work with only single view object for single-category datasets, Our method can roughly learn the shape of the objects from multiple instances, but it struggles with textures and geometry details such chair legs, multiple views is still beneficial for precise reconstruction.
autodecoder
Latent Diffusion Model trained the embedding vectors of pre-trained autodecoder. Similarly, with our training on feature grid resolution of x 4. training on the vector embedding struggles to generate the concrete geometry of the objects in the Chairs dataset 57
Figure 10:
EG3D
GET3D
Figure 11: Adversarial-base methods for large-scale and diverse 3D synthesis trained on Objaverse [4]
While gan baneb 3d generation methods EG3D 7 and GET3D [21] produce results for single object
datasets, we observe that they strugle converge large-scale and diverse setting as Objaverse [14]
20

---

A.6 Animated Results
Please visit our project's web page (https://githu) con/snap-researchl 3DVADER) for additional video results, showing consistent novel-view synthesis for rigid objects from multi-categon datasets and animated articulated objects sampled using our approach, and results demonstrating both conditional and unconditional generation.
B
Method Details
B.1
Volumetric Autodecoder
Volumetric Rendering- We use learnable volumetric rendering [48] to generate the final images from the final decoded volume. Given camera intrinsic and extrinsic parameters for target image, and the radiance field volumes generated by the decoder, for each pixel in the image, we cast a ray through the volume, sampling the color and density values to compute the color C(r) by integrating the radiance along the ray r(t) = =o = td, with near and far bounds in and ty:
C(x) = { T()))((()))((((() d)dt,
(7)
tx
where 8, the density and RGB values from the radiance field volumes sampled along these rays, and T(t) = c Sr (r(s))d is the accumulated transmittance between tn and t.
To supervise the silhouette of objects, we also render the 2D occupancy map O using the volumetric equation:
O(r) -(r)=S(" (())5(r(t)d..
We sample 128 points across these rays for radiance field rendering during training and inference.
Articulated Animation. As our approach is flexibly designed to support both rigid and articulated subjects, we employ different approaches to pose supervision to better handle each of these cases.
For articulated subjects, poses are estimated during training, using a set of learnable 3D keypoints K3D and their predicted 2D projections K2D in each image in an extended version of the Perspective- n-Point (PnP) algorithm [40]. To handle articulated animation, however, rather than learn a single pose per image using these points, we assume that the target subjects can be decomposed into Np regions, each containing N₂ points K3D points and their corresponding K2D projections per image. These points are shared across all subjects, and are aligned in the learned canonical space, allowing for realistic generation and motion transfer between these subjects. This allows for learning N, poses per-frame defining the pose of each region p relative to its pose in the learned canonical pose.
To successfully reconstruct the training images for each subject thus requires learning the appropriate canonical locations for each region's 's 3D keypoints, to predict the 2D projections of these keypoints in each frame, and the pose best matching the 3D points and 2D projections for these regions. We can then use this information in our volumetric rendering framework to sample appropriately from the canonical space such that the subject's appearance and pose are consistent and appropriate throughout their video sequence. Using this approach, this information can be learned along with our autodecoder parameters for articulated objects using the reconstruction and foreground supervision losses used for our rigid object datasets.
As noted in Sec. 3.2, to better handle non-rigid shape deformations corresponding this articulated motion, we employ volumetric linear blend skinning (LBS) [41]. This allows us to learn the weight each component p in the canonical space contributes to sampled point point in the deformed space based on the spatial correspondence between these two spaces:
Σ u uf( (Rpxo + tIp), p=1
Zd =
where Tр RRp.ty tpl
-R-1,-R-11 tr] is the estimated pose of part p relative to the camera (where T = R, tr] E R3x1 is the estimated camera pose with respect to canonical volume); zy is the
21

---

3D point deformed to correspond to the current pose; ze is its corresponding point when aligned in the canonical volume; and (zx))) is the learned LBS weight for component p. sampled at position Te in the volume, used to define this correspondence.
Thus, for our non-rigid subjects, in addition to the density and color volumes needed to integrate Eqns. and 8 ahove, our autodecoder learns to produce volume VLBS RS**N,, containing the LBS weights for each of the N2 locally rigid regions constituting the subject.
We assign N₂ 125 3D keypoints to each of the N, = 10 regions. For these tests, we assume fixed camera intrinsics with a field-of-view of 0.175 radians, as in [54). We use the differentiable Perspective-n-Poine (PnP) algorithm [40] implementation from PyTorch3D [22 to accelerate this training process,
As this approach suffices for objects with standard canonical shapes (e.g., human faces) performing non-rigid motion in continuous video sequences, we employ this approach for our tests on the CelebV-Text dataset. While in theory, such an approach could be used for pose estimation for rigid objects (with only component) in each view, for we find that this approach is less reliable for our rigid object datasets, which contain sparse, multi-view images from randomly sampled, non-continuous camera poses, depicting content with drastically varying shapes and appearances (e.g., the multi-category object datasets described below). Thus, for these objects, we use input either known ground-truth or estimated camera poses (using [66]), for synthetic renderings or real object or camera pose estimation without predefined keypoints from sparse images of arbitrary objects or scenes, employing such techniques for such data is beyond the scope of this work.
images, respectively. While some works 83 42, 87 perform category-agnoste
Architecture. Our volumetric autodecoder architecture follows that of [70). with the key extensions described in this work. Given an embedding vector e of size 1024, we use a fully-connected layer followed by reshape operation to transform it into a 43 volume with 512 features per cell. This is followed by series of four 3D residual blocks, each of which upsamples the volume resolution in each dimension and halves the features per cell, to a final resolution of 64 and 32 features. 1 These blocks consist of two 3 x 3 x 3 convolution blocks each followed by batch normalization in the main path, while the residual path consists of four 1 x X 1 convolutions, with ReLU applied after these operations. After the first of these blocks we have the 8 volume with 256 features per cell used for training our diffusion network, as in our final experiments. In this and the subsequent block, we apply self- self-attention layers [8]] as described in See. 3.1. After the final upsampling block, we apply a final batch normalization followed by a x 1 x convolution to produce the final 1 + 3 density V Denssit and RGB color features RRGB used in our volumetric renderer.
Non-Rigid Architecture. For non-rigid subjects, our architecture produces + + 10 output channels, with the latter group with the LBS weights for the n2p 10 locally rigid components each region corresponds to in our canonical space. Our unsupervised 2D keypoint predictor uses the U-Net architecture of [69], which operates on a downsampled 64 x 64 input image to predict the locations of the keypoints corresponding to each of the 3D keypoints used to determine the pose of the camera relative to each region of the subject when it is aligned in the canonical volumetric space.
B.2 Latent 3D Diffusion
Diffusion Architecture and Sampling. For our base diffusion model architecture, we use the Ablated Diffusion Model (ADM) of Dhariwal et al. (2021) 17), a U-Net architecture originally designed for 2D image synthesis. We incorporate the preconditioning enhancements this model described in Karras et al. (2022) (36). As this architecture was originally designed for 2D, we adapt all convolutions and normalizations operations, as well as the attention mechanisms, to 3D.
For the cross-attention mechanism used for our conditioning experiments, we likewise extend the latent-space cross-attention mechanism from Rombach et al. (2022) (63 to our 3D latent space.
Robust Normalization. Autoencoder-bann based latent diffusion models impose a prior to the learned latent vector 663. We find the latent features learned by our 3D autodecoder already form bekl-like curve. However, we also observe extreme values that can severely affect the calculation of the
'In practice, as in 70 we compute an approximate solution using the inverse LBS weights following HumanNeRF [4 to avoid the excessive computation required by the direct solution.
Wee add one block to upsample to 128 for our aforementioned experiments with the ABO Tables dataset.
22

---

Litert Reatures bulion
136,30
Stadetics
37930
11500
Relbur Maa
Gaaa Maas
2303
32500
12200
7530
1500
3203
503
20000
We present the latent feature distribution of 3D AutoDecoder trained on MVImgNet 92]. The features are extracted the 8 resolution, where we apply diffusion. The three subplots show different levels of zzommnng in." We see that the distribution spans a great range due to extreme outliers. Using classic mean and standard deviation computation, as we see in the middle subplot, still provides quite a large range of values. Normalizing the features using classic statistics leads convergence failure for the difflusion model. We propose using robust statistics normalize the distribution [-1, before training the diffusion model. During inference, we de-normalize the diffusion output before feeding them to the upsampling layers the autodecoder.
Figure 12:
4x4x4
8x8x8
16x16x16
H
Figure 13: Qualitative comparison of models trained at different latent resolutions. All visualizations produced with 64 diffusion steps. We find that the model train 8 latent features gives the best trade-off between quality and training speed, rendering it the best option for training large-scale 3D datasets.
mean and standard deviation. As discussed in the main manuscript, we deploy the use of robust normalization adjust the latent features. In particular, we take the median the center of the distribution and approximate its scale using the Normalized InterQuartile Range (IQR) 85] for normal distribution: 0.7413 IQR. We visualize its effect in Fig. This a crucial aspect of our approach, as in our experiments we find that without it, our diffusion training is unable to converge.
Ablating the latent volume resolution used for diffusion. We trained three diffusion models models for the same time, resources, and number of parameters, for diffusion at resolutions in our autodecoder: 43, 83, and 163. We find that the 4 models, even when they train faster, often fail to converge to something meaning full and produce partial results. Most samples produced by the 163 models of reasonable quality. However, many samples also exhibit spurious density values. We hypothesize that this is due to the model being under-trained. The 8 model produces the best results, and its fast training speed makes it suitable for large-scale training. We visualize the results in Fig.
B.3 Hash Embedding
Each object in the training set is encoded by embedding vector. However, employ multi- view datasets of various scales, up to ~300K unique targets from multiple categories, storing a separate embedding vector for each object depicted in the training images burdensome As such,
$ E.g., the codebook alone would require six times the parameters of the largest model in experiments.
23

---

we experimented with a technique enabling the effective use of a significantly reduced number of embeddings (no more than ~32K are required for any of our evaluations), while allowing effective content generation from large-scale datasets.
Similar to the approach in 155 we instead employ concatenations of smaller embedding vectors to create more combinations of unique embedding vectors used during training. For an embedding vector length L, the input embedding vector 1/&- R' used for an object be decoded is a concatenation of smaller embedding vectors fiz where each vector is selected from an ordered codebook with ne entries, with each entry containing collection of n/ embedding vectors of length Le/ne:
H h h12
(10)
where k; E } J is the set of indices used to select from the T& possible codebook entries for position i in the final vector. This method allows for exponentially more combinations of embedding vectors to be provided during training than must be stored in learned embedding vector library.
However, while in [55 the index j for the vector h at position is randomly selected for each position to access its corresponding codebook entry, instead use deterministic mapping from each training object index to its corresponding concatenated embedding vector. This function is implemented using a hashing function employing the multiplication method [16) for fast indexing using efficient bitwise operations. For object index k; the corresponding embedding index is:
m(k) = [[a k) mod 2 > (w r),
(11)
where the table has 2" entries. w and a are hcuristic hashing parameters used to reduce the number of collisions while maintaining an appropriate table size. We use 32 for w. must be an odd integer between 2w-1 and 2d [16]. We give each smaller codebook its own a value:
0; = 2= + +1,
(12)
where is the index of the codebook.
Discussion. In our experiments, we found that employing this approach had negligible impact on the overall speed and quality of our training and synthesis process. During training the memory of the GPU is predominantly occupied by the gradients, which are not affected by this hashing scheme. For Objaverse, our largest dataset using ~30K images, using this technique saves approximately 800MB of GPU memory.
Interestingly, this also suggests that scaling this approach to larger datasets, should they become available, will require special handling. Learning this per-object embedding would soon become intractable. However, simply using this hash embedding approach reduces the model storage requirements by 75% for this dataset.
In our experiments, we use hashing for ABO Tables, CelebV -ext and Objaverse, with codebook sizes ne = of 256, 8192 and 32768, respectively. We set the number of smaller codebooks (n₂, to 256 for each dataset.
C Implementation Details
C.1 Dataset Filtering
CelebV-Text (20). Some heuristic filtering was necessary to obtain sufficient video quality and continuity for our purposes. We omit the first and last 10% of each video to remove fade-in/out effects, and any frames with less than 25% estimated foreground pixels. We also remove videos with less than 4 frames remaining after this, and any videos less than 200 kilobytes due to their relatively low quality. We also omit a small number of videos that were unavailable for download at the time of our experiments (the dataset is provided as a set of URLs for the video sources).
MVImgNet (22.. For these annotated video frames depicting real objects unconstrained settings and environments, applied Grounded Segment Anything (39 for background removal. However,
24

---

as this process sometimes failed to produce acceptable segmentation results, we apply filtering to detect these case. We first remove objects for which Grounding DINO [45] fails detect bounding boxes. We then fit our volumetric autodecoder (Secs. 3.1-2) to only the musks produced by this segmentation (as monochrome images with a white foreground and black background). For objects that are properly segmented in each frame, this produces reasonable approximation of the object's shape that is consistent in each the input frames, while objects with incorrect or inconsistent segmentation will not be fit properly to the input images. Thus, objects for which the fitting loss is unsually high are removed.
Objaverse [14]. While Objaverse contains ~800K 3D models, we found that the overall quality of these varied greatly, making many of them unsuitable for multi-view rendering. We thus filtered models without texture, material maps, or other color and appearance properties suitable, as well as models with an insufficient polygon count for realistic rendering. Interestingly, given the simplicity of the objects when rendered against monochrome background, we found that the foreground segmentation supervision used for the other experiments described in Sec. 3.2 of the main paper was unnecessary. Given the scale of this dataset (~300K unique objects, with 6 frames per object), we thus omit this loss from our training process for this dataset for our final experiments for the sake of improved training efficiency. For datasets with more complex motion and real backgrounds, such as the real image datasets mentioned above, we found this supervision to be essential, as shown in Sec. A.2 and Fig.
C.2 Additional Details
Training Details. Our experiments are implemented in the PyTorch 58
59
using the
PyTorch Lightning [19 framework for fast automatic differentiation and scalable GPU-accelerated parallelization. For calculating the perceptual metrics (FID and KID), we used the Torch Fidelity (56)
library.
We run our experiments on NVIDIA A100 40GB GPUs per node. For some experiments, we use single node, while for larger-scale experiments, we use up to 8 nodes in parallel.
We the Adam optimizer [37] train both the autodecoder and the diffusion Model. For the first network, we use learning rate Ir 5e - and beta parameters / = (0.5, 0.999). For diffusion, we set the learning rate to Ir 4.5e 4. We apply linear decay the learning rate.
Preparing the Text Embeddings for Text-Driven Generation. We train our model for text-conditioned image generation three datasets: CelebV-Text [90], MVImgNet [22] and Objaverse [1]. The two latter datasets provide the object category of each sample, but they do not provide text descriptions. Using MiniGPT4 [94], extract description by providing hint and Describe this <hint> in one sentence. Describe its shape and color. Be concise, use only single sentence." " For MVImgNet, this hint is the "class name", while it is the "asset name" for Objaverse.
<<mnoommmmmmmmmmmmee the first view of each object along with the question: " "<Img> <lmageHere>></mmg
Note this approach is not foolproof. To the contrary, we find that in many cases MiniGPT4 hallucinates descriptive characteristics of the object that do not match its visual input. We can see some examples like this in Fig. 14
With the text-image pairs for these three datasets, we use the 11-billion parameter T5 61 model to extract a sequence of text-embedding vectors. The dimensionality of these vectors is 1024. During training, we fix the length of the embedding sequence 32 elements. We trim longer sentences and pad smaller sentences with zeroes.
25

---

ABO-Table
Chairs
CelebV-Text
MVImgNet
Objaverse
3D AutoDecoder
z-length
1024
1024
1024
1024
1024
MaxChannels
512
512
512
512
512
Depth
2
2
4
AA-Resolutions
8,16
8,16
8,16
8,16
8.16
ForegroundLoss
10
10
#Renders/batch
4
4
4
4
VoxeIGridSize
128 x 4 5e-4
643 x 4
64 x 14
643 x 4
64 x 4
Learning Rate
5e-4
5e-4
5e-4
5e-4
Latent 3D Diffussonn Model
z-shape
x 256
83 x 256
83 x 256
x 256
8 x 256
Sampler
edm
edm
edm
edm
edm
Channels
128
128
192
192
192
Depth
2
3
3
Channel Multiplier
3,4
3,4
3,4
3,4
3,4
SA-resolutions
8,4
8,4
8,4
8,4
8,4
Learning Rate
4.5e-5
4.5e-5
4.5e-5
4.5e-5
4.5e-5
Conditioning
None
None
None/CA 8,4
None/CA 8,4
None/CA
CA-resolutions
8,4
Embedding Dimension
1024
1024
1024
Transformers Depth
Table 5: Architecture details for our models for each dataset. SA and CA stand for Self-Artention and
Cross-Artention respectively. refers our ID embedding vector and our latent 3D volume for the autodecoder and diffusion models, respectively. Note that for Celeb channels per cell: for color values, 1 for density and 10 for part assignment.
the output volume has 14
26

---

The chandeller is white, ornate light fixture hanging from black ceiling.
The object black and red robot with wheels on the bottom
The object is small, white tower with a pointed top
The object waffle with round shape and a golden brown color.
The cylindrical object is made of gray metal and has round base with small handle
The object small, green tree with single leaf top, standing rock..png
The object smiling on a yellow background
object white bird perched a branch with sky in the background
The image shows a cartoon character wearing blue white outfit, holding spear hand standing rocky surface.
The object black and white image sitting white background
Gop
The image is black background with the words das caps" white latters.
The fish yellow and brown striped discus fish.
dE DAS) CAPAS
Figure 14: Automated image captioning with MiniGPT4 (94) on MVImgNet (92). We can observe how the captioning tool hallucinated descriptions that do not match the input image, which lcads to text-to-3D diffusion model lcarning not to adhere to every detail in the input prompt. We believe that better captioning systems should help alleviate this issue the future
27


Reviews:
[{"review_id": "NS1nxDZidE", "reviewer": "Reviewer_Nn3Q", "summary": "In this paper, the authors propose an approach to learning a latent diffusion model for 3D assets from image (or video) supervision. The core is that they use an autodecoder architecture, which learns embeddings in the latent space by decoding them into a volumetric representation for rendering. Then, the authors identify the appropriate intermediate volumetric latent space and introduce a normalization technique to learn a latent diffusion model. In addition, this method can use either existing camera information or no camera information during training. The experiments demonstrate the proposed method is able to generate both static and articulated 3D objects.", "questions": "Please refer to the weakness section. I will adjust my rating according to the rebuttal.", "limitations": "The limitation and broader impact part look adequate to me.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method has a broader application domain and is extendable to large-scale datasets. It can be applied to various datasets, including rigid static and articulated moving objects. \n- The proposed modifications are effective in improving the 3D autodecoder. \n- The authors propose a robust normalization trick to train latent diffusion models on every intermediate volumetric latent space.", "weaknesses": "- The authors miss a proper discussion with an important related work, GAUDI [a]. GAUDI first learns a 3D autodecoder from images and then trains a diffusion model on the latent space as a prior for generation. There are several similar points between this work and GAUDI. However, GAUDI is not even cited.\n- The learned 3D voxel grids are treated as \"Canonical Representation.\" However, how to properly define a canonical view of large-scale 3D datasets? For example, in Objaverse, the object poses are not well-aligned. Even for a single category (e.g., chair), the objects are randomly placed. Will this case influence the performance of the proposed method? Do you have any observations? \n- Considering the autoencoder-based method needs to regularize the bottleneck, the autodecoder can also regularize their bottleneck as done in [c, d]. Have you tried to train a 1D diffusion model (see [e]) in the latent space of the autodecoder to see the generation ability?\n- Missing comparison with some important baselines, such as GET3D [f]. I also suggest the authors train GET3D and DiffRF for Table 3 for a fair comparison. \n- I have some questions or suggestions according to the writing:\n  - In Line 1, the authors assume diffusion models are all latent diffusion models. I suggest the authors polish their writing to avoid this misuse.\n  - In Line 57, the authors mention, \"First, our autodecoders do not have a clear \"bottleneck.\" Is the bottleneck here a latent space? If it is the case, the embedding space is the latent space of the autodecoder.\n  - In Line 147, it is suggested to add some credits to prior work on trilinear interpolation for volumetric rendering, such as Plenoxels [b].\n  - Can you elaborate more on the direct latent sample? \n  - It is suggested to move the \"Hash Embedding\" part to the main paper. Otherwise, the vanilla autodecoder approach can not be scaled to large-scale datasets due to the large size of the embedding.\n  - It is suggested to add [a-e] to the reference.\n- In LDM [46], they use a normalization technique on the latent space. The proposed robust normalization seems to be an extended version of this.\n\n[a] GAUDI: A Neural Architect for Immersive 3D Scene Generation. NeurIPS 2022.\n\n[b] Plenoxels: Radiance Fields without Neural Networks.CVPR 2022.\n\n[c] Demystifying Inter-Class Disentanglement. ICLR 2020.\n\n[d] Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement. ICCV 2021.\n\n[e] Latent Point Diffusion Models for 3D Shape Generation. NeurIPS 2022.\n\n[f] GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank R.Nn3Q for their review. We appreciate their highlight of broader method application, effectiveness of our 3D autodecoder modifications and application of robust normalization. Next we answer weakness and questions:\n\n**W1 (GAUDI Discussion):** Thank you for pointing that out, GAUDI[a] is indeed a relevant work, we will add citation and discussion into the final version of the paper. We would like to highlight a couple of differences between our work and [a]. GAUDI is used for a pretty narrow application - indoor scene fly-through generation. The datasets in this application are relatively small, from 18 to maximum 1.8k scenes. While in our work we try to tackle datasets of 2 orders of magnitude higher. This work also does not model articulated objects. Moreover the diffusion process is trained in the z space of the auto-decoder, which we later show has significantly lower performance compared to the space with spatial arrangement, e.g. $8^3$ voxel, in our setting.\n\n**W2 (Canonical Object Orientation):** The canonical object orientation is indeed important for autodecoder fitting. The better alignment objects have, the easier it is to fit an auto-decoder. To demonstrate that we run the experiment of PhotoChairs dataset, where we randomly flip some of the objects upside down. The fitting error in terms of perceptual loss is 0.76 versus 0.52 (for original dataset). Thus significant effort was made towards pre-processing large scale datasets. For an objaverse dataset we observe that most of the objects are axis aligned, thus we just center at (0,0) and rescale according to the largest axis. In MVImgNet, there is no canonical orientation, thus we instead use partial sparse point clouds provided with a dataset to center and then perform PCA to orient. On the other hand, canonicalization for CelebV is obtained automatically, because of the joint camera estimation and object reconstruction. We believe that automatic canonicalization for a more general dataset, such as MVImgNet could be interesting future work.\n\n**W3 (1D Diffusion model):**. We find that because autodecoder is trying to compress large scale dataset into relatively small latent space, the early layers act more like a dictionary, not as an upsampler, because of this it maybe hard for the diffusion to operate in the early layers. We show this phenomenon for $4^3$ representation in Fig.3 of the main paper. Additionally we run an experiment with 1D representation; the visual samples are provided in Fig. G of the rebuttal pdf. We observe that at this stage the model behavior is very similar to direct latent sampling as the method fails to produce correct object geometry.\n\n**W4 (Get3D comparison):** Since we already compare with two GAN-based baselines, we did not include Get3D. However we scheduled a training run on Objaverse, the preliminary results are 259.18 FID (see Fig. F of supplementary pdf for visual results), which is significantly larger than our model with 40.49. We believe this is because GAN-based methods struggle with fitting diverse categories, without additional supervision such as class labels, Get3D will not produce good results. We still plan to finish the training on this dataset and add this experiment in the supplementary material, but we do not expect it to perform significantly better than this. We also would like to point out that DiffRF requires fitting a NeRF for 300k objects as a preprocessing stage, which is infeasible with our current resources. Running DiffRF on MvImgNet will not work, since in this dataset most of the objects have only partial visibility. For example some clothing items may be only shown from frontal view, while some toys may only shown from the top. This issue will also complicate application of Get3D, since it is not trivial to devise appropriate camera distribution. CelebV, on the other hand, has articulated objects which both Get3D and DiffRF do not support. \n\n**W5 (All Diffusion models are latent):** We fixed this in the new version of the paper.\n\n**W5 (Autodecoder bottleneck):** Sorry for the confusion, we used the “bottleneck” term for “latent representation on which diffusion operates”, since for other latent diffusion models they are the same, but we agree that in our case it makes sense to use a different term. We will change this in the future version of the paper.  \n\n**W5 (Plenoxels):** Thank you for the suggestion, we will add the proposed work.\n\n**W5 (Direct Lattent Sampling):** The direct latent sampling is basic multivariate gaussian fitting on the 1D embeddings for the entire dataset. This method was used in prior work [1]. In case of hash embedding we follow [a] and randomly sample the indices of each hashtable. \n\n**W5 (Hash Embedding):** According to your suggestion, we will shortly describe hash embedding in the main paper, and keep the details for supplement.\n\n**W5 (Citations):** We will add all the proposed methods to the related work.\n\n**W6 (LDM normalization vs robust normalization):** Similarly to LDM we confirm that normalization of the latent space is a crucial part of the diffusion training. However we want to clarify that we did not claim this as our finding. But we disagree that proposed robust normalization is an extension of LDM normalization. LDM normalization is based on KL minimization during training, this is used as an additional loss, which may hurt the reconstruction. Instead our robust normalization can be applied after the training, thus does not affect the reconstruction in any way. Another benefit of the proposed normalization is the ability to select the representation on which to operate after the autodecoder training. \n\n[a] StyleGenes: Discrete and Efficient Latent Distributions for GANs. ArXiV 2023\n"}, {"review_id": "Ebe7T97c8i", "reviewer": "Reviewer_2ZWE", "summary": "The paper proposes a 3D autoencoder to learn a latent volumetric space on the training dataset, which can be decoded into a radiance volumetric representation for novel-view synthesis, and then learn a 3D diffusion model on the latent volumetric representation. The latent volumetric space is acquired by training NeRF on multiple views (or frames).", "questions": "1. In L41-L43, the authors claim that \"our approach is thus designed to be robust to\" different sources of poses. However, I do not find relevant explanations in the method or experiment section. Can the authors elaborate on it?\n2. In L150-L151, the authors claim that \"intermediate representations such as feature volumes or tri-planes, as it is more efficient to render and ensures consistency across multiple views\". However, several prior or concurrent works (e.g., 3D Neural Field Generation using Triplane Diffusion, 3DGen: Triplane Latent Diffusion for Textured Mesh Generation, Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction) show that feature volumes or tri-planes can work well. Can the authors provide any ablation study to support the claim?\n3. In L62, \"To identify the best scale\" is unclear, as the word \"scale\" is not mentioned before.\n4. It is unclear which datasets are used for training and evaluation. In L71-76, it is unclear whether the authors train 3 models on 3 datasets separately or they train a model on 3 datasets progressively. And in Sec 4.1, it is unclear whether all datasets are just for evaluation, or some of them are used for training. The authors should clarify their training and evaluation protocols.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Diverse datasets, including synthetic and real-world, rigid and articulated objects, are used for evaluation.", "weaknesses": "1. Some claims in the paper are not verified by the evidence. See *Questions* for examples.\n2. Only novel-view synthesis is demonstrated. Since the authors use a volumetric representation, the method is restricted by the voxel resolution. The visualized images look small and coarse. It will be better if the authors can show some 3D results (e.g., extracting meshes from radiance fields). Thus, compared to prior and concurrent works (see *Questions*), this method does not seem to be more scalable or extendable. Especially, the voxel resolution is a bottleneck.\n3. Camera poses still seem to be necessary in this work, either automatically being annotated in synthetic data or estimated for in-the-wild data. It is a little tricky to say that this work does not need \"3D supervision\". The authors can do such an experiment to convince the readers that the proposed method is robust to camera poses: comparing a model trained with estimated poses on synthetic data and one trained with GT poses.\n\n\nMinor typo:\n- L212: One of our key observation -> One of our key observations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank R.2ZWE for their thought-provoking review. We are glad they appreciate that our method works on diverse datasets, both real-world and synthetic, as well as rigid and articulated objects. Let us address R.2ZWE points:\n\n**Q4 (Training and Evaluation Protocol):** Regarding evaluation and training. We follow the evaluation protocol of DiffRF, we train on one dataset and then compute FID and KID on the same dataset. In more detail we train 5 different models, for each of the five datasets considered in this paper: PhotoShape Chairs, ABO Tables, CelebV-Text, MVImgNet and Objaverse. Everything in these experiments is separated and they don’t have any interactions of any kind. \n\n**W3 and Q1 (Camera poses):** Regarding the camera poses, we use different sources of the camera poses for different datasets. Predicting camera poses is an extremely challenging task, and we are not claiming to provide a solution for it in this work. This is especially challenging for arbitrary object categories. We claim that our method could work with different sources of camera pose: \nWith ground truth camera poses that are available in synthetic datasets: PhotoShape Chairs, ABO Tables and Objaverse. \nWith datasets of rigid objects where COLMAP can provide reliable estimates of the camera, such as MvIMGNet. \nNon-rigid objects where we train a camera prediction model (which in this case also acts as pose prediction model) together with the autoencoder, without any additional supervision or COLMAP predictions, see CelebV.\nSo hence we are saying that \"our approach is thus designed to be robust to\" different sources of poses, in the meaning “our approach is flexible to work with different sources of poses”. Here we did not claim that it is “robust to pose estimation errors from COLMAP”. We are sorry for the confusion and we try to make it clearer in the final version.\n\n**W2 and Q2: (Concurrent Triplane works)**  We would like to bring to R.2ZWE attention the fact that all the works R.2ZWE mentioned [a, b, c] were not published during the time of the submission, and thus are concurrent works. Because of this we did not include comparison with them in our original manuscript. We provide extracted meshes in the Fig. H (rebuttal pdf). \nAs R.2ZWE correctly mentioned, all these works use triplanes as intermediate representation, and we agree that theoretically Triplanes can be used for datasets where a lot of views for supervision is available, such as objaverse. When the multi-view supervision is scarce and ground truth camera information is not available, such as in video datasets like CelebV, Triplanes tend to degrade to prediction of the flat objects, which was observed in the prior work [d]. Another disadvantage of Triplanes and feature volumes is additional MLP requirement, this MLP significantly increases time for each individual forward pass. We provide a timing comparison below:\n\n| Representation | Time for 1 iteration (s) |\n| :----: | :----:|\n| Voxel Grid $4\\times64^3$     | 0.22|\n| Triplane     $96\\times64^2$   | 0.33|\n| Triplane     $96\\times128^2$ | 0.33|\n| Triplane     $96\\times256^2$ | 0.38|\n| Triplane     $96\\times512^2$ | 0.46|\n\n\nHere we render at 128x128 resolution with 128 points per ray, for Triplane we use 2 layer MLP with 32 hidden neurons and the triplane generator uses exactly the same architecture as our Voxel generator. While the generation of the Triplane is relatively lightweight, the MLP is very heavy. Coming back to the concurrent works mentioned by R.2ZWE, only [b] was shown to be trained on a large scale multi category dataset. This is achieved by training a latent diffusion model with auto**encoder** where the input to it is dense point cloud, autoencoder need sufficiently less iteration to converge thus it is feasible to train with Triplanes as intermediate representation. However the requirement of dense point clouds require ground truth object meshes, which is not available for the datasets like MVImgNet or CelebV. Thus our method covers significantly more possible scenarios than [b]. \n\n**Q3 (best scale):** Thank you for pointing this out, here it should be “To identify best  intermediate representation …”. \n\n**W3 (observations typo):** Thank you, we will fix the typo.\n\n[a] 3D Neural Field Generation using Triplane Diffusion. CVPR - 2023.\n\n[b] 3DGen: Triplane Latent Diffusion for Textured Mesh Generation. Arxiv - 2023. (Probably ICCV - 2023) \n\n[c] Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction.  ICCV - 2023\n\n[d] Unsupervised Volumetric Animation, CVPR - 2023\n"}, {"review_id": "As5DuCnn7I", "reviewer": "Reviewer_xDwW", "summary": "This paper proposes a diffusion model that learns to generate 3D objects, using only multi-view images or videos for training. It first trains a 3D convolutional autodecoder to embed the dataset; this maps latent vectors via a 3D feature space to voxelised scenes, and is trained for reconstruction of volumetrically-rendered images. Then, it trains a diffusion model over the feature space of this autodecoder, to enable a-priori generation; there are experiments on which layer of 3D features are best to use. The method is demonstrated on a variety of datasets, both synthetic and real, including rigid objects such as chairs, and non-rigid objects such as human faces.", "questions": "Why is an autodecoder approach chosen, instead of (for example) VQ-GAN or autoencoding?\n\nSee above under weaknesses regarding performance comparison with baselines on real-world data – this is my largest concern.\n\nHow are the 'driven' face animations in fig. 2 generated? I didn't see this described anywhere in the text.\n", "limitations": "There is adequate discussion of both limitations and broader impact.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed pipeline is novel, its components are clearly described, and most design decisions are well motivated/justified (and appear sound). There are insightful discussions on how the autodecoder architecture affects performance.\n\nThe evaluation is rather comprehensive, covering five datasets of somewhat different character. Some of these are synthetic and use ground-truth camera poses, while others use imperfect poses from SfM.\n\nGeneration results on synthetic images (chairs & tables) are quantitatively better than baselines – both FID and KID are lower than pi-GAN, EG3D, and DiffRF. Qualitative results here also look good.\n\nGeneration results on real images are found to be quantitatively of higher quality (lower FID/KID) than those generated by naively sampling in the latent space of the autodecoder (without any diffusion process).\n\nThere is an ablation study covering various important design decisions in the autodecoder (mainly architectural choices), and an additional set of experiments investigating which feature layer of the autodecoder the diffusion is performed over, and how many diffusion steps are used.\n", "weaknesses": "While it may be accurate that \"no prior work demonstrates the ability to generalize to\nsuch large-scale datasets\" (L282), I feel there should be some attempt at a quantitative comparison here – e.g. retraining the best of the prior works on these datasets to see how well or badly they perform. In particular, Objverse, CelebV-Text and MVImgNet are all extremely recent, so it may simply be that the prior works have not been tested on those datasets (as they were not yet available), but would still work to some degree. This is particularly important to put the (rather high!) FID scores of tab. 3 in context. An alternative that would go some way to mitigating this problem would be to use one of the more-constrained by still photographic datasets used in pi-GAN or EG3D (e.g. human/cat faces or cars), and evaluate how well the proposed method performs on this.\n\nQualitative results on MVImgNet (only given in the supplementary) are not very impressive – it is often impossible for this reviewer to determine the class of the generated objects. Similarly, the results on face generation (with a target) look significantly lower fidelity than other recent methods like EG3D.\n\nThe qualitative text-conditioned generation results in fig. 4 & 5 are not particularly impressive. Moreover, the 'ground-truth' text labels are from a pretrained captioning model, and thus noisy. It would be valuable to include an experiment with high-quality captions, so the impact of caption quality vs model power can be understood.\n\nThe technical contribution is a little small for NeurIPS – both latent diffusion and autodecoding of 3D shapes are pre-existing techniques, and their composition does not seem to be particularly complicated. This is somewhat mitigated by the detailed experiments and extensive discussion of design choices.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank R.xDwW for the informative review. We are delighted that they found our pipeline novel and our evaluation comprehensive. R.xDwW also highlights the comparison with respect to the baselines and ablation study. In the following we address R.2ZWE points:\n\n**W1 and Q2 (Prior methods on large scale dataset):** We agree with the R.xDwW, that comparison of prior gan-based methods on large scale dataset such as objaverse will be useful. To this end we launch a training run of EG3D on Objaverse dataset. Since the dataset is very large, training is only at about 50%, the best FID was 105.15 (that was reached at 20% of the training) while FID for our method is 40.49, the visual results are provided in the Fig. A of the rebuttal pdf. We will provide final results in the next version of the paper, however we observe that FID stops improving at some point and now oscillates at around 120, thus we believe it will not become much better after this point.  We advocate this behavior to the notorious difficulty of training GAN-based with diverse categories, without additional supervision such as class labels. We believe this experiment already demonstrates the scalability issues of the GAN based models. We would like to highlight that running EG3D on MvImgNet will be even more problematic, since the camera distribution in this dataset is unknown and depend on the considered object. For example some clothing items may be only shown from frontal view, while some toys may only shown from the top. CelebV, on the other hand, has articulated objects which EG3D does not support. \n\n**W2 (Qualitative results)**: We agree with the R.xDwW, that it may sometimes be hard to understand the class of the object from generated results in MvImgNet, however we believe that this is partially a dataset issue. This dataset features a lot of fruits and vegetables and their pieces which can be hard to recognize. We provide a sample from this dataset in Fig. B of the supplement rebuttal. Regarding CelebV-Text, while indeed the visual fidelity is lower than EG3D we would like to highlight that the considered setting is very different, we target articulated objects while EG3D targets static ones. Moreover, unlike EG3D we did not use ground truth poses, so a fair comparison would be EG3D w/o cameras where the face geometry is completely flat (see Fig.4 in EG3D supplementary material). \n\n**W3 (High quality captions)**: We fully agree that the method will hugely benefit from better captions, however at the time of the submission no caption for the MvImgNet and Objaverse was available, thus we resort to off-the-shelf captioning system. After the submission, one of the concurrent works [a] proposed a method for annotating the Objaverse dataset; we plan to utilize these captions for a future work.\n\n**Q1 (Why autodecoder)**: The problem of both VQGan and autoencoder, is the encoder part. The encoder assumes that the output of the autoencoder is already known in advance, which does not work for 3D generation, since for most of the real world datasets we only have images. There are several methods [b, c] that work with images as input, but the reconstruction quality of these works is significantly lower compared to optimization approaches. Another disadvantage of these approaches is that they can only be trained with a small number of views, thus it hard to utilize datasets with large number of views such as PhotoShape Chairs and ABO Tables (about 100 views) and CelebV-Text (up to 1000 frames for each video). \n\n**Q3 (Driven animation)**: For driving animation we generate an animatable asset form our diffusion model and then we animate it with poses extracted by our pose predictor from driving video shown at the left. We will explain this in more details in supplementary material, for more technical details on how articulated generation is performed please refer to **Sec. B1** of the supplementary material.\n \n[a] Scalable 3D Captioning with Pretrained Models - Arxiv 2023 \n[b] pixelNeRF: Neural Radiance Fields from One or Few Images - CVPR 2021\n[c] ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers - ECCV 2022\n"}, {"review_id": "jmDrxGb25B", "reviewer": "Reviewer_iNSR", "summary": "This paper presents a 3D generation framework that generalize to large-scale 3D dataset and articulated objects. The method comprises two parts. The first part is a 3D auto-decoder, reconstructing 3D objects from multi-view images or monocular videos. The second part is a latent diffusion model for unconditional or text-conditioned 3D generation. Extensive experiments are performed on 5 datasets, including the largest 3D dataset objaverse.", "questions": "N/A", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I am impressed by the workload of this work. Not only are diverse static 3D objects are supported, but also articulated objects like 3D human heads.\n- Experiments are solid. A total of 5 datasets are used, including Objaverse, which is the largest and the most challenging 3D dataset. Qualitative results on tables and chairs are reported and compared with 3D-aware GAN and 3D diffusion methods.\n- Using the normalized IQR and median to normalize the latent features is an interesting trick. It is crucial for latent diffusion methods to deal with the latent normalization.", "weaknesses": "- The method itself is not novel, which I think is a minor issue since the main contribution of this work I think is the scaling up and generalizing the articulated objects.\n- The quality of generated objects is inferior. I think this is limited by the reconstruction part. The resolution is limited to 64^3 due to the computational complexity of 3D volumes. I am wondering whether a CNN refinement/ super resolution module would help (e.g. EG3D, DiffRF)?\n- The text-conditioned generation can only correctly generate the overall color and object category. Most detail descriptions are ignored. Other than the caption quality, is there any other possible reason for that, like the conditioning method design?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We were delighted to read R.iNSR’s review! They appreciate the workload needed to achieve large and diverse static 3D object generation as well as synthesizing articulated human heads. Moreover, they commend our proposed Robust Normalization and De-Normalization scheme and its importance for latent diffusion modeling. In the following text, we aim to clarify the points they raised:\n\n**W2 (CNN refinement)**: Indeed CNN refiner could help to improve the visual fidelity of the results, although it may be at the expense of 3D consistency. We try autodecoding stage where we use CNN to refinement for ABO Tables dataset, we observe an improvement of the visual fidelity, however 3D geometry become more noisy. We provide visual results in the Fig. F of the rebuttal pdf. We will also add this experiment to the supplementary material. \n\n**W3 (Text Conditioning)**: We believe that the main problem is quality of the captions, mainly all the captions include some spurious details not related to the depicted content or do not describe detail at all, we provide some examples in the Fig. C of the rebuttal pdf, thus the model learns to completely ignore these details. Since we use exactly the same conditioning mechanism as Stable Diffusion, we think that this should not be a problem. \n"}, {"review_id": "8gWiwH1W5T", "reviewer": "Reviewer_nmDk", "summary": "In this manuscript, a novel method for unconditional and text-conditional generative models of 3D shape and texture representations is proposed. More specifically, the authors propose to train a 3D diffusion model of radiance and RGB fields that can be trained from 2D image and object mask supervision. The method comprises two stages, where in the first stage, an auto-decoder is trained, while in the second, a 3D UNet modeling a denoising diffusion process is trained on the latent features. At test time, random latent representations can be sampled and denoised to generate 3D radiance volumes that can be rendered from arbitrary viewpoints.", "questions": "- Can the authors explain why the FID metric is so bad for the pi-GAN method?\n- Can the authors explain what are the main reasons for the limited quality for the texture predictions in Figure 5?\n- How important is it that the datasets contain multiple views (as opposed to only one) of the same object? ", "limitations": "- The authors have discussed limitations as well as potential negative societal impact. As stated above, I believe the manuscript could benefit from a comparison/discussion against test-time-optimization-based methods due to the limited quality of the shown results.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a valid method for training 3D diffusion models from posed 2D images. The key idea to first train an auto-decoder, and then in a second stage a 3D UNet describing a 3D denoising diffusion process is interesting and, with the success of recent latent 2D diffusion models in mind, an important field of study.\n- The authors perform an extensive evaluation on 5 different datasets. To be able to run experiments on these different datasets, multiple automated masking and/or filtering steps needed to be performed (compare e.g. L. 252). \n- The manuscript is well written, the technical explanations are sound, and the use of mathematical terms and symbols is correct.\n- The manuscript contains helpful and well-organised figures such as the overview figure 1", "weaknesses": "- Experimental Evaluation in Table 1: Why does pi-GAN achieve such a high FID value of 52.71? Could the authors share some intuition on why the score is so bad? This seems not to be coherent with comparable prior works, for example GRAF [47] or GIRAFFE [43] report an FID of 34/20 on this dataset. Note that not improving over GAN-based approaches in FID on these single-object datasets is OK I believe, but it is important that the numbers are accurately reported. \n- Qualitative Comparison: I believe the manuscript would benefit from a qualitative baseline comparison, e.g. for the methods reported in Table 1. It would further be interesting to discuss to a greater extent the differences of the results from the different generative model types, i.e. compare GAN-based results to the proposed diffusion model.\n- The results are still quite limited, both qualitatively and quantitatively. I do appreciate that the authors tackle this hard task \"feed-forward\" of unconditional/text-conditional 3D generation, but e.g. the texture from the samples in Fig. 5 are very limited. I believe at least a text-based comparison and discussion to test-time optimization-based methods such as Dreamfusion is relevant here, as they lead to significantly better results. Potentially the proposed method could also be extended with an additional test-time-optimization step in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank R.nmDk for their detailed response. We appreciate their highlighting of our work’s strengths. Namely, R.nmDk (a) finds our proposed 3D Diffusion model an interesting idea in an important field of study; (b) commends our extensive evaluation on different datasets, while appreciating the intricacy of their preparation; (c) appreciates our manuscripts writing, technical analysis and formulas, and well organized figures. Below, we try to address R.nmDk’s concerns and questions:\n\n**W1 and Q1 (pi-GAN FID):** We agree with R.nmDk on the importance of accurately reporting the numbers! For Table 1, we borrow all the metrics from DiffRF [a] (we will add this information on the table) , as we closely follow their dataset preparation and evaluation pipeline, which is different from GRAF’s. In particular, DiffRF renders “...15,576 chairs using Blender Cycles [13] from 200 views on an Archimedean spiral.” [a]. In contrast, GRAF renders “...150k Chairs from Photoshapes [49] following the rendering protocol of [46]” [b]. The dataset differences can vastly affect the output metric. \n\n**W2 (Qualitative comparison with baselines).** As R.nmDk points out, we do not include a visualization or discussion of diffusion-models and GAN-based models. We found that these qualitative results and analyses have been covered by DiffRF[a] and we focused on the differences of how we tackle the 3D Diffusion modeling problem compared to [a]. Similarly to DiffRF we observe that diffusion-based methods can achieve overall better quality and more meaningful structure and do not have view-dependent artifacts. \n\nFor our qualitative comparison with DiffRF we would like to refer R.nmDk to Figures 7 and 8 in our supplement; for the discussion please see L51-55, L108-113, in the main text and L11-30 of the supplementary text. \nWe appreciate R. nmDk’s suggestion and believe that including these comparisons will make our work more complete and comprehensive for the reader. We will add them in the final version of the manuscript.\nSummarizing these points, DiffRF is learning a single voxel grid for each object and is training a diffusion model on a dataset of learned voxel grids. This approach has several limitations: you need to save a Radiance Field for each item in the dataset: a time- and space-consuming process, especially for larger voxel grids. Additionally, 3D UNet training is very costly and thus prohibitive for larger voxel grids limiting the practical grid size. \n\nOur approach alleviates both of these problems. Our AutoDecoder represents the whole dataset as a collection of per-object embeddings plus the weights of a single decoder network. We run diffusion on a 8x8x8 grid which is significantly faster, and independent of the final output resolution. These changes permit our method to tackle large-scale datasets.\n\n\n**W3 (Comparison with Dreamfusion):** We discuss optimization-based methods in our related work section (L116-121). While we agree that Dreamfusion shows high-quality results, we disagree it is an apt comparison for our approach. They aim for generating a single object of high quality based on a text prompt; we aspire to enable large-scale 3D diffusion models. The main benefit is that Dreamfusion requires 1.5 hours on 4 TPU chips per object, while our method needs less than a few seconds to generate an object.\n\nIn summary, we find Dreamfusion is tackling a different problem to ours. Nevertheless, we find that R.nmDk’s suggestion to combine the two approaches an excellent idea and would definitely explore it for future work. We believe score-distillation from a pretrained 2D model could not only help improve the texture quality, but also help distill additional knowledge to the one available in our 3D datasets. \n\n**Q2 (Texture prediction):** The Fig.5 contains objects generated from the model trained on images from the Objaverse dataset. We render these objects with uniform lighting, to exclude potential lighting inconsistencies. Because many objects in this dataset just have uniform texture, the model learned to use uniform color as texture for many objects (see sample from this dataset if Fig. C of the rebuttal pdf). Please, also refer to the supplementary material we show a more representative sample of the generated objects, many of which contain more complicated textures.   \n\n**Q3 (Single view training):** As we mentioned above, the AutoDecoder offers a compressed representation of the dataset; it encapsulates prior knowledge. In contrast to NeRF, our method can work with Single-View for single-category datasets. Our method can roughly learn the shape of the objects from multiple instances, but it struggles with details such as chair legs, so multiple views is still beneficial for precise reconstruction. We show reconstruction results in Figure D of the rebuttal PDF. We are now running the diffusion stage and will add the results to the next version of the paper. \n \n[a] Diffrf: Rendering-guided 3d radiance field diffusion. CVPR 2023. \n[b] Graf: Generative radiance fields for 3d-aware image synthesis. NeurIPS 2020.\n[c] HoloDiffusion: Training a {3D} Diffusion Model using {2D} Images. CVPR 2023\n"}]

Author rebuttal:
We would like to thank all the reviewers for their thoughtful and detailed reviews. We are delighted to see that they found our method broadly applicable (R.Nn3Q), our modification effective (R.Nn3Q) and evaluation extensive (R.2ZWE, R.xDwW, R.iNSR, R.nmDk). It is also nice to see that reviewers appreciate the importance of our proposed robust normalization technique (R.Nn3Q, R.iNSR). We also would like to thank R.nmDk for appreciating the quality of writing and figures, and both R.nmDk and R.iNSR for highlighting the workload needed to handle experiments on such a diverse set of large scale datasets. We provide the response to each individual reviewer in the corresponding section.

Meta-review: The paper proposed a audodecoding framework with a diffusion model on voxel latent space for the 3D generative model, and showed broad applicabilities of the method, including generating rigid and articulated objects, using real and synthetic datasets, with synthetic or estimated cameras. The paper also proposed some effective components, including robust normalization, and diffusion models design on the voxel latent. The rebuttal and discussions addressed the concerns of the additional comparisons with other baselines. Considering the weaknesses and strengths, all the reviewers are becoming positive about this paper. I follow the reviewers' consensus and would recommend accepting this paper. 

Please add the additional comparisons and discussions to other baselines to the main paper, the motivation of why voxel-based representation is better than tri-plane representation, as well as the writing clarifications mentioned by the reviewers.
Decision: Accept (poster)