SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feature Processing. One of our key observation is that the features  $F$  in the latent space of the 3D autodecoder have a bell-shaped distribution (see the supplement), which eliminates the need to enforce any form of prior on it, e.g. as in [63]. Operating in the latent space without a prior enables training a single autodecoder for each of the possible latent diffusion resolutions. However, we observe that the feature distribution  $F$  has very long tails. We hypothesise this is because the final density values inferred by the network do not have any natural bounds, and thus can fall within any range. In fact, the network is encouraged to make such predictions, as they have the sharpest boundaries between the surface and empty regions. However, to allow for a uniform set of diffusion hyper-parameters for all datasets and all trained autodecoders, we must normalize their features into the same range. This is equivalent to computing the center and the scale of the distribution. Note that, due to the very long-tailed feature distribution, typical mean and standard deviation statistics will be heavily biased. We thus propose a robust alternative based on the feature distribution quantiles. We take the median  $m$  as the center of the distribution and approximate its scale using the Normalized InterQuartile Range (IQR) [85] for a normal distribution:  $0.7413 \times IQR$ . Before using the features  $F$  for diffusion, we normalize them to  $\hat{F} = \frac{(F - m)}{IQR}$ . During inference, when producing the final volumes we de-normalize them as  $\hat{F} \times IQR + m$ . We call this method robust normalization. Please see the supplement for an evaluation of its impact.
Sampling for Object Generation. During inference we rely on the sampling method from EDM [36], with several slight modifications. We fix EDM's hyperparameter matching the dataset's distribution to 0.5 regardless of the experiment, and modify the feature statistics in our feature processing step. We also introduce classifier free guidance [27] for our text-conditioning experiments (Sec. 4.5). We found that setting the weight equal to 3 yields good results across all datasets.
4 Results and Evaluations
In this section, we evaluate our method on multiple diverse datasets (see Sec. 4.1) for both unconditional 4.2 and conditional settings 4.5. We also ablate the design choices in our autodecoder and diffusion in Secs. 4.3 and 4.4, respectively.
Method	PhotoShape FID ↓	Chairs [57] KID ↓	ABO Tables [13] FID ↓	KID ↓
π-GAN [6]	52.71	13.64	41.67	13.81
EG3D [7]	16.54	8.412	31.18	11.67
DiffRF [49]	15.95	7.935	27.06	10.03
Ours	11.28	4.714	18.44	6.854
Table 1: Results on the synthetic PhotoShape Chairs [57] and ABO Tables [13] datasets. Overall, our method outperforms state-of-the-art GAN-based and diffusion-based approaches. KID scores are multiplied by  $10^{3}$ .
Model Variant	PSNR ↑	LPIPS ↓
Ours	27.719	6.255
- Multi-Frame Training	27.176	6.855
- Self-Attention	27.335	6.738
- Increased Depth	27.24	6.924
- Embedding Length (1024 → 64)	25.985	8.332
Table 2: Our 3D autodecoder ablation results. "indicates this component has been removed. As we remove each sequentially, the top row depicts results for the unmodified architecture and training procedure. LPIPS results are multiplied by  $10^{2}$
Figure 2: Qualitative comparisons with Direct Latent Sampling (DLS) [70] on CelebV [90]. We show the two driving videos for two random identities: the top identity in each block is generated by our method, the bottom identity in each block is generated by DLS [70]. We also show the rendered depth and normals.
4.1 Datasets and Data Processing
Below we describe the datasets used for our evaluations. We mostly evaluate our method on datasets of synthetic renderings of 3d objects [13, 57, 14]. However, we also provide results on a challenging video dataset of dynamic human subjects [90] and dataset of static object videos [92].
ABO Tables. Following [49], we evaluate our approach on renderings of objects from the Tables subset of the Amazon Berkeley Objects (ABO) dataset [13], consisting of 1,676 training sequences with 91 renderings per sequence, for a total of 152,516 renderings.
PhotoShape Chairs. Also as in [49], we use images from the Chairs subset of the PhotoShape dataset [57], totaling 3, 115, 200 frames, with 200 renderings for each of 15, 576 chair models.
Objaverse. This dataset [14] contains  $\sim 800\mathrm{K}$  publicly available 3D models. As the of the object geometry and appearance varies, we use a manually-filtered subset of  $\sim 300\mathrm{K}$  unique objects (see supplement for details). We render 6 images per training object, for a total of  $\sim 1.8$  million frames.
MVImgNet. For this dataset [92], we use  $\sim 6.5$  million frames from 219, 188 videos of real-world objects from 239 categories, with an average of 30 frames each. We use Grounded Segment Anything [45, 39] for background removal, then apply filtering (see supplement) to remove objects with failed segmentation. This process results in 206, 990 usable objects.
CelebV-Text. The CelebV-Text dataset [90] consists of  $\sim 70\mathrm{K}$  sequences of high-quality videos of celebrities captured in in-the-wild environments, lighting, motion, and poses. They generally depict the head, neck, and upper-torso region, but contain more challenging pose and motion variation than prior datasets, e.g. VoxCeleb [50]. We use the robust video matting framework of [44] to obtain our masks for foreground supervision (Sec. 3.2). Some sample filtering (described in the supplement) was needed for sufficient video quality and continuity for training. This produced  $\sim 44.4\mathrm{K}$  unique videos, with an average of  $\sim 373$  frames each, totaling  $\sim 16.6\mathrm{M}$  frames.
For training, we use the camera parameters used to render each synthetic object dataset, and the estimated parameters provided for the real video sequences in MVImgNet, adjusted to center and scale the content to our rendering volume, (see supplement for details). For the human videos in CelebV-Text, we train an additional pose estimator along with the autodecoder  $G$  to predict poses for each articulated region per frame, such that all objects can be aligned in the canonical space (Sec. 3.2). Note that for creating dynamic 3D video, we can use sequences of poses transferred from the real video of another person from the dataset.
Figure 3: We show generated samples of our model trained using rendered images from Objaverse [14]. We show three views for each object, along with the normals for each view. We also show depth for the right-most view. Text-conditioned results are shown. Growth-truth captions are generated by MiniGPT-4 [94].
Figure 4: We show generated samples from our model trained using monocular videos from MVImgNet [92]. We show three views for each object, along with the normals for each view. We also show depth for the right-most view. Text-conditioned results are shown. Ground-truth captions are generated by MiniGPT-4 [94].
4.2 Unconditional Image Generation
Synthetic Datasets. Following the evaluation protocol of [49], we report results on the ABO Tables and PhotoShape Chairs datasets. These results on single-category, synthetically rendered datasets that are relatively small compared to the others, demonstrate that our approach also performs well with smaller, more homogeneous data. We render 10 views of 1K samples from each dataset, and report the Fréchet Inception Distance (FID) [26] and Kernel Inception Distance (KID) [3] when compared to 10 randomly selected ground-truth images from each training sequence. We report the results compared to both GAN-based [6, 7] and more recent diffusion-based approaches [49] methods, as seen in Tab. 1. We see that our method significantly outperforms state-of-the-art methods using both metrics on the Tables dataset, and achieves better or comparable results on the Chairs dataset.
Large-Scale Datasets. We run tests on the large-scale datasets described above: MVImgNet, CelebV-Text and Objaverse. For each dataset, we render 5 images from random poses for each of 10K generated samples. We report the FID and KID for these experiments compared to 5 ground-truth images for each of 10K training objects. As no prior work demonstrates the ability to generalize to such large-scale datasets, we compare our model against directly sampling the 1D latent space of our base autodecoder architecture (using noise vectors generated from a standard normal distribution). This method of 3D generation was shown to work reasonably well [70]. We also evaluate our approach with different numbers of diffusion steps (16, 32 and 64). The results can be seen in Tab. 3. Visually, we compare with [70] in Fig. 2. Our qualitative results show substantially higher fidelity, quality of geometry and texture. We can also see that when identities are sampled directly in the 1D
latent space, the normals and depth are significantly less sharp, indicating that there exist spurious density in the sampled volumes. Tab. 3 further supports this observation: both the FID and KID are significantly lower than those from direct sampling, and generally improve with additional steps.
4.3 Autodecoder Ablation
We conduct an ablation study on the key design choices for our autodecoder architecture and training. Starting with the final version, we subtract the each component described in Sec. 3.1. We then train a model on the PhotoShape Chairs dataset and render 4 images for each of the  $\sim 15.5\mathrm{K}$  object embeddings.
CelebV-Text [90]	MVImgNet [92]	Objverse [14]
Method	FID ↓	KID ↓	FID ↓	KID ↓	FID ↓	KID ↓
Direct Latent Sampling [70]	69.21	73.74	97.51	69.22	72.76	53.68
Ours - 16 Steps	48.01	49.49	62.21	39.94	47.49	32.44
Ours - 32 Steps	49.74	46.2	51.26	28.45	43.68	31.7
Ours - 64 Steps	50.27	47.72	43.85	23.91	40.49	29.37
Table 3: Results on large-scale multi-view image (Objaverse [14] & MVImgNet [92]) and monocular video (CelebV-Text [90]) datasets. The KID score is multiplied by  $10^{3}$ .
Tab. 2 provides the PSNR [30] and LPIPS [93] reconstruction metrics. We find that the final version of our process significantly outperforms the base architecture [70] and training process. While the largest improvement comes from our increase in the embedding size, we see that simply removing the multi-frame training causes a noticeable drop in quality by each metric. Interestingly, removing the self-attention layers marginally increases the PSNR and lowers the LPIPS. This is likely due to the increased complexity in training caused by these layers, which for a dataset of this size, may be unnecessary. For large-scale datasets, we observed significant improvement with this feature. Both decreasing the depth of the residual convolution blocks and reducing the embedding size cause noticeable drops in the overall quality, particularly the latter. This suggests that the additional capacity provided by these components is impactful, even on a smaller dataset.
4.4 Diffusion Ablation
We also perform ablation on our diffusion process, evaluating the effect of the choice of the number of diffusion steps (16, 32, and 64), and the autodecoder resolution at which we perform diffusion  $(4^{3},8^{3}$  ,and  $16^{3})$  .For these variants, we follow the generation quality training and evaluation protocol on the PhotoShape Chairs (Sec. 4.2), except that we disable stochasticity in our sampling during inference for more consistent performance across these tests. Each model was trained using roughly the same amount of time and computation. Fig. 5 shows the results. Interestingly, we can see a clear distinction between the results obtained from
diffusion at the earlier or later autodecoder stages, and those from our the results with resolution  $8^{3}$ . We hypothesize that at lowest resolution layers overfit to the training dataset, thus when processing novel objects via diffusion, the quality degrades significantly. Training at a higher resolution requires substantial resources, limiting the convergence seen in a reasonable amount of time. The number of sampling steps has a smaller, more variable impact. Going from 16 to 32 steps improves the results with a reasonable increase in inference time, but at 64 steps, the largest improvement is at the  $16^{3}$  resolution, which requires more than 30 seconds per sample. Our chosen diffusion resolution of  $8^{3}$  achieves the best results, allowing for high sample quality at 64 steps (used in our other experiments) with only  $\sim 8$  seconds of computation, but provides reasonable results with 32 steps in  $\sim 4$  seconds.
Figure 5: Impact of diffusion resolution and number of sampling steps on sample quality and inference time.
4.5 Conditional Image Generation
Finally, we train diffusion models with text-conditioning. For MVImgNet and Objaverse, we generate the text with an off-the-shelf captioning system [94]. Qualitative results for MVImgNet and Objaverse are in Figs. 4 and 3. We observe that in all cases, our method generates objects with reasonable geometry that generally follow the prompt. However, some details can be missing. We believe our
model learns to ignore certain details from text prompts, as MiniGPT-4 often hallucinates details inconsistent with the object's appearance. Better captioning systems should help alleviate this issue.
4.6 Design Choices for Large-Scale 3D Object Synthesis
The goal of our work is to enable 3D object synthesis by training a model on large and diverse multi-view image datasets. To realize this goal there are two main design choices that we need to make: (a) what is the appropriate 3D representation and (b) generative modelling approach.
Recent works [2, 68, 23, 9] use tri-planes as their 3D representation. However, when the multi-view supervision is scarce and ground truth camera information is not available, such as in video datasets like CelebV, tri-planes tend to degrade to prediction of flat objects [70]. Moreover, tri-planes require an additional MLP for volumetric rendering, applied for every of the 128 ray point samples and  $128^{2}$  output pixels we use in our setting. In contrast, our voxel grid autodecoder outputs directly color and density. Tri-planes are faster to autodecode, but rendering them is much slower. Training for an iteration with our  $64^{3}$  voxel grid takes  $0.22s$ . Tri-planes of size  $64^{2}$ ,  $128^{2}$ ,  $256^{2}$  and  $512^{2}$  require  $0.33$ ,  $0.33$ ,  $0.38$ ,  $0.46$  seconds respectively. We use 32 channels per plane, a two-layer MLP with 32 hidden channels, and a 2D autodecoder. This can severely affect the training time for a large dataset.
EG3D [7] and GET3D [21] propose an adversarial approach to 3D synthesis. Both, base their generators on StyleGAN [33], which for 2D datasets requires considerable changes to produce good results in large and diverse datasets [65]. Training both on Objaverse, we find they fail to converge, as seen in Fig. 11. Thus, we believe our diffusion-based approach is better suited for our goal.
Despite the inherent challenges in performing flexible 3D content generation for arbitrary content domains without 3D supervision, our work demonstrates this is possible with the right approach. By exploiting the inherent power of autodecoders to synthesize content in a domain without corresponding encoded input, our method learns representations of the structure and appearance of diverse and complex content suitable for generating high-fidelity 3D objects using only 2D supervision. Our latent volumetric representation is conducive to 3D diffusion modeling for both conditional and unconditional generation, while enabling view-consistent rendering of the synthesized objects. As seen in our results, this generalizes well to various types of domains and datasets, from relatively small, single-category, synthetic renderings to large-scale, multi-category real-world datasets. It also supports the challenging task of generating articulated moving objects from videos. No prior work addresses each of these problems in a single framework. The progress shown here suggests there is potential to develop and extend our approach to address other open problems.
Limitations. While we demonstrate impressive and state-of-the-art results on diverse tasks and content, several challenges and limitations remain. Here we focus on images and videos with foregrounds depicting one key person or object. The generation or composition of more complex, multi-object scenes is a challenging task and an interesting direction for future work. As we require multi-view or video sequences of each object in the dataset for training, single-image datasets are not supported. Learning the appearance and geometry of diverse content for controllable 3D generation and animation from such limited data is quite challenging, especially for articulated objects. However, using general knowledge about shape, motion, and appearance extracted from datasets like ours to reduce or remove the multi-image requirement when learning to generate additional object categories may be feasible with further exploration. This would allow the generation of content learned from image datasets of potentially unbounded size and diversity.
Broader Impact. Our work shares similar concerns with other generative modeling efforts, e.g., potential exploitation for misleading content. As with all such learning-based methods, biases in training datasets may be reflected in the generated content. Appropriate caution must be applied when using this method to avoid this when it may be harmful, e.g. human generation. Care must be taken to only use this method on public data, as the privacy of training subjects may be compromised if our framework is used to recover their identities. The environmental impact of methods requiring substantial energy for training and inference is also a concern. However, our approach makes our tasks more tractable by removing the need for the curation and processing of large-scale 3D datasets, and is thus more amenable to efficient use than methods requiring such input.
Acknowledgements We would like to thank Michael Vasilkovsky for preparing the ObjVerse renderings, and Colin Eles for his support with infrastructure. Moreover, we would like to thank Norman Müller, author of DiffRF paper, for his invaluable help with setting up the DiffRF baseline, the ABO Tables and PhotoShape Chairs datasets, and the evaluation pipeline as well as answering all related questions. A true marvel of a scientist. Finally, Evan would like to thank Claire and Gio for making the best cappuccinos and fueling up this research.
A Additional Experiments and Results
A.1 Geometry Generation Evaluation
Following the point cloud evaluation protocol of [1], we measure the Coverage Score (COV) and the Minimum Matching Distance (MMD) for points sampled from our generated density volumes. Given a distance metric for two point clouds  $X$  and  $Y$ , e.g. the Chamfer Distance (CD),
$$
\mathrm {C D} (X, Y) = \sum_ {x \in X} \min  _ {y \in Y} \| x - y \| _ {2} ^ {2} + \sum_ {y \in Y} \min  _ {x \in X} \| x - y \| _ {2} ^ {2}, \tag {4}
$$
COV measures the diversity of the generated point cloud set  $S_{g}$ , with respect to a reference point clout set  $S_{r}$ , by finding the closest neighbor in the reference set to each one in the sample set, and computing the fraction of the reference set covered by these samples:
$$
\operatorname {C O V} \left(S _ {g}, S _ {r}\right) = \frac {\left| \left\{\arg \min  _ {Y \in S _ {r}} C D (X , Y) \mid X \in S _ {g} \right\} \right|}{\left| S _ {r} \right|}. \tag {5}
$$
MMD, in contrast, measures the overall quality of these samples, by measuring the average distance between each sampled point cloud and its closest neighbor in the reference set:
$$
\operatorname {M M D} \left(S _ {g}, S _ {r}\right) = \frac {1}{\left| S _ {r} \right|} \sum_ {Y \in S _ {r}} \min  _ {X \in S _ {g}} C D (X, Y). \tag {6}
$$
We compute these metrics for the PhotoShape Chairs and ABO Tables datasets, comparing our generated results to points sampled from the the same reference meshes used in the data splits from the evaluations in DiffRF [49]. For each generated object, we sample 2048 points from a mesh extracted from the decoded density volume  $V^{\text{Density}}$  (see Sec. 3.1) using the Marching Cubes [46] algorithm. We use a volume of resolution  $64^3$  and  $128^3$  for training the Chairs and Tables models, respectively. However, we note that downsampling these density volumes to  $32^3$ , as is used in DiffRF, before applying this point-sampling operation did not noticeably impact the results of these evaluations.
The results can be seen in Tab. 4, alongside the perceptual metrics from the main paper. Interestingly, these results show that, despite the increased flexibility of our approach, and DiffRF's restrictive use of both 2D rendering and 3D supervision on synthetic data when training their diffusion model, we obtain comparable or superior geometry compared to their approach, while substantially increasing the overall perceptual quality for these datasets. We also substantially outperform prior state-of-the-art approaches using GAN-based [6, 7] methods across both perceptual and geometric comparisons with these metrics.
Figs. 7 and 8 show qualitative comparisons between the unconditional generation results rendered using our method and DiffRF for each of these datasets. In each case, it is clear that for similar objects, our method produces more coherent and complete shapes without missing features, e.g. legs, and textures that are more realistic and detailed, leading to better and more consistent image synthesis results.
A.2 Foreground Supervision
For some datasets with foregrounds with complex and varying appearance which can easily be mixed with the background environment, we found it necessary to supplement our primary autodecoder reconstruction loss (Sec. 3.2) with an additional foreground supervision loss. This loss measures how well depicted objects are separated from the background during rendering. To evaluate the effect of this foreground supervision, we ran experiments on the CelebV-Text [90] dataset both with and without this loss. We conduct our training until the autodecoder has seen a total of 9 million frames from the training set, then reconstruct examples from the learned embeddings.
The result can be seen in Fig. 6. As depicted, the reconstructions without foreground supervision not only lack fidelity to the target appearance, but the estimated opacity and surfaces normals clearly show that the overall geometry is insufficiently recovered.
Method	PhotoShape Chairs [57]	ABO Tables [13]
FID ↓	KID ↓	COV ↑	MMD ↓	FID ↓	KID ↓	COV ↑	MMD ↓
π-GAN [6]	52.71	13.64	39.92	7.387	41.67	13.81	44.23	10.92
EG3D [7]	16.54	8.412	47.55	5.619	31.18	11.67	48.15	9.327
DiffRF [49]	15.95	7.935	58.93	4.416	27.06	10.03	61.54	7.610
Ours	11.28	4.714	64.20	4.445	18.44	6.854	60.25	6.684
Table 4: Quantitative comparison of unconditional generation on the PhotoShape Chairs [57] and ABO Tables [13] datasets. Our method achieves a better perceptual quality, while maintaining similar geometric quality to the state-of-the-art diffusion-based approaches. MMD and KID scores are multiplied by  $10^{3}$ .
With Foreground Mask Supervision
Without Foreground Mask Supervision
A.3 Training the autodecoder on a single view per object
We conduct an experiment on training our autodecoder on the Chairs [57] dataset but only using a single view per object. To do so, we pick and train on a single rendering view among the 200 available, which is different for each object. Our autodecoder learns, nevertheless, to encode general geometry information and roughly infer the full 3D shape, of the object by utilizing information learned from other objects. However, it stills struggles with the texture details, and thus training with more training views is beneficial. We show the results on Fig. 9.
A.4 Training a diffusion model on the embedding vector of the autodecoder