SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evangelos Ntavelis*
Computer Vision Lab ETH Zurich
entavelis@vision.ee.ethz.ch
Aliaksandr Siarohin
Snap Inc.
Santa Monica, CA, USA
asiarohin@snapchat.com
Kyle Olszewski
kolszewski@snap.com
Chaoyang Wang
CI2CV Lab
Pittsburgh, PA, USA
chaoyanw@cs.cmu.edu
Luc Van Gool
CVL, ETH Zurich, CH
PSI, KU Leuven, BE
INSAIT, Un. Sofia, BU
vangool@vision.ee.ethz.ch
Sergey Tulyakov
stulyakov@snapchat.com
We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation for rendering view-consistent appearance and geometry. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations to learn a 3D diffusion from 2D images or monocular videos of rigid or articulated objects. Our approach is flexible enough to use either existing camera supervision or no camera information at all – instead efficiently learning it during training. Our evaluations demonstrate that our generation results outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and a large-scale, real video dataset of static objects.
Code & Visualizations: https://github.com/snap-research/3DVADER
Photorealistic generation is undergoing a period that future scholars may well compare to the enlightenment era. The improvements in quality, composition, stylization, resolution, scale, and manipulation capabilities of images were unimaginable just over a year ago. The abundance of online images, often enriched with text, labels, tags, and sometimes per-pixel segmentation, has significantly accelerated such progress. The emergence and development of denoising diffusion probabilistic models (DDPMs) [75, 77, 28] propelled these advances in image synthesis [53, 78, 80, 76, 17, 18, 63, 36, 86] and other domains, e.g. audio ([10, 20, 95]) and video ([24, 89, 82, 29, 25, 47]).
However, the world is 3D, consisting of static and dynamic objects. Its geometric and temporal nature poses a major challenge for generative methods. First of all, the data we have consists mainly of images and monocular videos. For some limited categories of objects, we have 3D meshes with corresponding multi-view images or videos, often obtained using a tedious capturing process or created manually by artists. Second, unlike CNNs, there is no widely accepted 3D or 4D representation suitable for 3D geometry and appearance generation. As a result, with only a few exceptions [74], most of the existing 3D generative methods are restricted to a narrow range of object
*Work done during internship at Creative Vision Team - Snap Inc
categories, suitable to the available data and common geometric representations. Moving, articulated objects, e.g. humans, compound the problem, as the representation must also support deformations.
In this paper, we present a novel approach to designing and training denoising diffusion models for 3D-aware content suitable for efficient usage with datasets of various scales. It is generic enough to handle both rigid and articulated objects. It is versatile enough to learn diverse 3D geometry and appearance from multi-view images and monocular videos of both static and dynamic objects. Recognizing the poses of objects in such data has proven to be crucial to learning useful 3D representations [6, 7, 73, 74]. Our approach is thus designed to be robust to the use of ground-truth poses, those estimated using structure-from-motion, or using no input pose information at all, but rather learning it effectively during training. It is scalable enough to train on single- or multi-category datasets of large numbers of diverse objects suitable for synthesizing a wide range of realistic content.
Recent diffusion methods consist of two stages [63]. During the first stage, an autoencoder learns a rich latent space. To generate new samples, a diffusion process is trained during the second stage to explore this latent space. To train an image-to-image autoencoder, many images are needed. Similarly, training 3D autoencoders requires large quantities of 3D data, which is very scarce. Previous works used synthetic datasets such as ShapeNet [8] (DiffRF[49], SDFusion [12], etc.), and were thus restricted to domains where such data is available.
In contrast to these works, we propose to use a volumetric autodecoder to learn the latent space for diffusion sampling. In contrast to the autoencoder-based approach, our autodecoder maps a 1D vector to each object in the training set, and thus does not require 3D supervision. The autodecoder learns 3D representations from 2D observations, using rendering consistency as supervision. Following UVA [70] this 3D representation supports the articulated parts necessary to model non-rigid objects.
There are several key challenges with learning such a rich, latent 3D space with an autodecoder. First, our autodecoders do not have a clear "bottleneck." Starting with a 1D embedding, they upsample it to latent features at many resolutions, until finally reaching the output radiance and density volumes. Here, each intermediate volumetric representation could potentially serve as a "bottleneck." Second, autoencoder-based methods typically regularize the bottleneck by imposing a KL-Divergence constraint [38, 63], meaning diffusion must be performed in this regularized space.
To identify the best intermediate representation to perform diffusion, one can perform exhaustive layer-by-layer search. This, however, is very computationally expensive, as it requires running hundreds of computationally expensive experiments. Instead, we propose robust normalization and denormalization operations which can be applied to any layers of a pre-trained and fixed autodecoder. These operations compute robust statistics to perform layer normalization and, thus, allow us to train the diffusion process at any intermediate resolution of the autodecoder. We find that at fairly low resolutions, the space is compact and provides the necessary regularization for geometry, allowing the training data to contain only sparse observations of each object. The deeper layers, on the other hand, operate more as upsamplers. We provide extensive analysis to find the appropriate resolution for our autodecoder-based diffusion techniques.
We demonstrate the versatility and scalability of our approach on various tasks involving rigid and articulated 3D object synthesis. We first train our model using multi-view images and cameras in a setting similar to DiffRF [49] to generate shapes of a limited number of object categories. We then scale our model to hundreds of thousands of diverse objects train using the real-world MVImgNet [92] dataset, which is beyond the capacity of prior 3D diffusion methods. Finally, we train our model on a subset of CelebV-Text [90], consisting of  $\sim 44\mathrm{K}$  sequences of high-quality videos of human motion.
2.1 Neural Rendering for 3D Generation
Neural radiance fields, or NeRFs (Mildenhall et al., 2020 [48]), enable high-quality novel view synthesis (NVS) of rigid scenes learned from 2D images. Its approach to volumetric neural rendering has been successfully applied to various tasks, including generating objects suitable for 3D-aware NVS. Inspired by the rapid development of generative adversarial models (GANs) [22] for generating 2D images [22, 5, 32, 33, 35, 34] and videos [79, 72, 91], subsequent work extends them to 3D content generation with neural rendering techniques. Such works [67, 51, 54, 52, 88] show promising
results for this task, yet suffer from limited multi-view consistency from arbitrary viewpoints, and experiencing difficulty in generalizing to multi-category image datasets.
A notable work in this area is pi-GAN (Chan et al., 2021 [6]), which employs neural rendering with periodic activation functions for generation with view-consistent rendering. However, it requires a precise estimate of the dataset camera pose distribution, limiting its suitability for free-viewpoint videos. In subsequent works, EG3D (Chan et al., 2022 [7]) and EpiGRAF (Skorokhodov et al. [73]) use tri-plane representations of 3D scenes created by a generator-discriminator framework based on StyleGAN2 (Karras et al., 2020 [35]). However, these works require pose estimation from keypoints (e.g. facial features) for training, again limiting the viewpoint range.
These works primarily generate content within one object category with limited variation in shape and appearance. A notable exception is 3DGP [74], which generalizes to ImageNet [15]. However, its reliance on monocular depth prediction limits it to generating front-facing scenes. These limitations also prevent these approaches from addressing deformable, articulated objects. In contrast, our method is applicable to both deformable and rigid objects, and covers a wider range of viewpoints.
2.2 Denoising Diffusion Modeling
Denoising diffusion probabilistic models (DDPMs) [75, 28] represent the generation process as the learned denoising of data progressively corrupted by a sequence of diffusion steps. Subsequent works improving the training objectives, architecture, and sampling process [28, 17, 86, 36, 63, 53, 76] demonstrated rapid advances in high-quality data generation on various data domains. However, such works have primarily shown results for tasks in which samples from the target domain are fully observable, rather than operating in those with only partial observations of the dataset content.
One of the most important of such domains is 3D data, which is primarily observed in 2D images for most real-world content. Some recent works have shown promising initial results in this area. DiffRF [49] proposes reconstructing per-object NeRF volumes for synthetic datasets, then applying diffusion training on them within a U-Net framework. However, it requires the reconstruction of many object volumes, and is limited to low-resolution volumes due to the diffusion training's high computational cost. As our framework instead operates in the latent space of the autodecoder, it effectively shares the learned knowledge from all training data, thus enabling low-resolution, latent 3D diffusion. In [12], a 3D autoencoder is used for generating 3D shapes, but this method require ground-truth 3D supervision, and only focuses on shape generation, with textures added using an off-the-shelf method [60]. In contrast, our framework learns to generate the surface appearance and corresponding geometry without such ground-truth 3D supervision.
Many recent works [2, 68, 23, 9] combine a denoising diffusion approach with a tri-plane representation [7] for 3D generation. They perform diffusion on the embedding vector of an autodecoder [2], the bottleneck of an autoencoder [23], or directly, on a pre-computed [68] or a simultaneously learned tri-plane [9]. Nevertheless, these works focus on small datasets or require a dense point clouds and ground truth object meshes, which are not readily available for real-object image datasets. The triplane representation requires an MLP decoder that substantially increases the volumetric rendering time. Our voxel-decoder does not has such a requirement as it directly outputs color and density, and thus permitting faster training on large-scale real image datasets.
Recently, several works [60, 43, 11] propose using large-scale, pre-trained text-to-image 2D diffusion models for 3D generation. The key idea behind these methods is to use 2D diffusion models to evaluate the quality of renderings from randomly sampled viewpoints, then use this information to optimize a 3D-aware representation of the content. Compared to our method, however, such approaches require a far more expensive optimization process to generate each novel object.
Our method is a two-stage approach. In the first stage, we learn an autodecoder  $G$  containing a library of embedding vectors corresponding to the objects in the training dataset. These vectors are first processed to create a low-resolution, latent 3D feature volume, which is then progressively upsampled and finally decoded into a voxelized representation of the generated object's shape and appearance. This network is trained using volumetric rendering techniques on this volume, with 2D reconstruction supervision from the training images.
Figure 1: Our proposed two-stage framework. Stage 1 trains an autodecoder with two generative components,  $\mathrm{G}_1$  and  $\mathrm{G}_2$ . It learns to assign each training set object a 1D embedding that is processed by  $\mathrm{G}_1$  into a latent volumetric space.  $\mathrm{G}_2$  decodes these volumes into larger radiance volumes suitable for rendering. Note that we are using only 2D supervision to train the autodecoder. In Stage 2, the autodecoder parameters are frozen. Latent volumes generated by  $\mathrm{G}_1$  are then used to train the 3D denoising diffusion process. At inference time,  $\mathrm{G}_1$  is not used, as the generated volume is randomly sampled, denoised, and then decoded by  $\mathrm{G}_2$  for rendering.
During the second stage, we split the autodecoder  $G$  into two parts,  $G = G_{2} \circ G_{1}$ . We then employ this autodecoder to train a 3D diffusion model operating in the compact, 3D latent space obtained from  $G_{1}$ . Using the structure and appearance properties extracted from the autodecoder training dataset, this 3D diffusion process allows us to use this network to efficiently generate diverse and realistic 3D content. The full pipeline is depicted in Fig. 1.
Below, we first describe the volumetric autodecoding architecture (Sec. 3.1). We then describe the training procedure and reconstruction losses for the autodecoder (Sec. 3.2). Finally, we provide details for our training and sampling strategies for 3D diffusion in the decoder's latent space (Sec. 3.3).
3.1 Autodecoder architecture
Canonical Representation. We use a 3D voxel grid to represent the 3D structure and appearance of an object. We assume the objects are in their canonical pose, such that the 3D representation is decoupled from the camera poses. This decoupling is necessary for learning compact representations of objects, and also serves as a necessary constraint to learn meaningful 3D structure from 2D images without direct 3D supervision. Specifically, the canonical voxel representation consists of a density grid  $V^{\mathrm{Density}} \in \mathbb{R}^{S^3}$  which is a discrete representation of the density field with resolution  $S^3$ , and  $V^{RGB} \in \mathbb{R}^{S^3 \times 3}$  which represents the RGB radiance field. We employ volumetric rendering, integrating the radiance and opacity values along each view ray similar to NeRFs [48]. In contrast to the original NeRF, however, rather than computing these local values using an MLP, we tri-linearly interpolate the density and RGB values from the decoded voxel grids, similar to Plenoxels [64].
Voxel Decoder. The 3D voxel grids for density and radiance,  $V^{\mathrm{Density}}$  and  $V^{\mathrm{RGB}}$ , are generated by a volumetric autodecoder  $G$  that is trained using rendering supervision from 2D images. We choose to directly generate  $V^{\mathrm{Density}}$  and  $V^{\mathrm{RGB}}$ , rather than intermediate representations such as feature volumes or tri-planes, as it is more efficient to render and ensures consistency across multiple views. Note that feature volumes and tri-planes require running an MLP pass for each sampled point, which requires significant computational cost and memory during training and inference.
The decoder is learned in the manner of GLO [4] across various object categories from large-scale multi-view or monocular video datasets. The architecture of our autodecoder is adapted from [70]. However, in our framework we want to support large scale datasets which poses a challenge in designing the decoder architecture with the capability to generate high-quality 3D content across various categories. In order to represent each of the  $\sim 300\mathrm{K}$  objects in our largest dataset we need very high-capacity decoder. As we found the relatively basic decoder of [70] produced poor reconstruction quality, we introduce the following key extensions:
- To support the diverse shapes and appearances in our target datasets, we find it crucial to increase the length of the embedding vectors learned by our decoder from 64 to 1024.
2We experimented with diffusion at different feature volume resolutions, ranging from  $4^{3}$  at the earliest stage to  $16^{3}$  in the later stages. These results are described in our evaluations (Sec. 4.3, Fig. 5).
- We increase the number of residual blocks at each resolution in the autodecoder from 1 to 4.
- Finally, to harmonize the appearance of the reconstructed objects we introduce self-attention layers [81] in the second and third levels (resolutions  $8^{3}$  and  $16^{3}$ ).
Scaling the Embedding Codebook for Large Datasets. Each object in the training set is encoded by an embedding vector. However, storing a separate vector for each object is burdensome, especially for large datasets. As such, we propose a technique to significantly reduce the parameter footprint of our embeddings, while allowing effective generation from large-scale datasets.
Similar to StyleGenes' approach [55], we combine smaller embedding subvectors to create unique per-object vectors. The decoder's input is a per-object embedding vector  $H_{k} \in \mathbb{R}^{l}$  with length  $l_{v}$ . It is a concatenation of smaller subvectors  $h_{i}^{j}$ , where each subvector is selected from an ordered codebook with  $n_{c}$  entries, with each entry containing collection of  $n_{h}$  embedding vectors of length  $l_{v} / n_{c}$ :
$$
H _ {k} = \left[ h _ {1} ^ {k _ {1}}, h _ {2} ^ {k _ {2}}, \dots , h _ {n _ {c}} ^ {k _ {n _ {c}}} \right], \tag {1}
$$
where  $k_{i} \in \{1,2,\dots,n_{h}\}$  is the set of indices used to select from the  $n_{h}$  possible codebook entries for position  $i$  in the final vector. This method allows for exponentially more combinations of embedding vectors, greatly reducing the number of stored parameters compared to a single embedding codebook. In contrast to [55], the index  $j$  for the vector  $h_{i}^{j}$  at position  $i$  is not randomly selected for each position to access its corresponding codebook entry. We use a hashing function [16] to map each training object index  $k$  to its corresponding embedding index.
3.2 Autodecoder Training
We train the decoder from image data through analysis-by-synthesis, with the primary objective of minimizing the difference between the decoder's rendered images and the training images. We render RGB color image  $C$  using volumetric rendering [48], additionally in order to supervise silhouette of the objects we render 2D occupancy mask  $O$ .
Pyramidal Perceptual Loss. As in [69, 70], we employ a pyramidal perceptual loss based on [31] on the rendered images as our primary reconstruction loss:
$$
\mathcal {L} _ {\text {r e c}} (\hat {C}, C) = \sum_ {l = 0} ^ {L} \sum_ {i = 0} ^ {I} \left| \mathrm {V G G} _ {i} \left(\mathrm {D} _ {l} (\hat {C})\right) - \mathrm {V G G} _ {i} \left(\mathrm {D} _ {l} (C)\right) \right|, \tag {2}
$$
where  $\hat{C}, C \in [0,1]^{H \times W \times 3}$  are the RGB rendered and training images of resolution  $H \times W$ , respectively;  $\mathrm{VGG}_i$  is the  $i^{\text{th}}$ -layer of a pre-trained VGG-19 [71] network; and operator  $D_l$  downsamples images to the resolution for pyramid level  $l$ .
Foreground Supervision. Since we only interested in modeling single objects, in all the datasets considered in this work, we remove the background. However if the color of the object is black (which corresponds to the absence of density), the network can make the object semi-transparent. To improve the overall shape of the reconstructed objects, we make use of a foreground supervision loss. Using binary foreground masks (estimated by an off-the-shelf matting method [44], Segment Anything [39] or synthetic ground-truth masks, depending on the dataset), we apply an L1 loss on the rendered occupancy map to match that of the mask corresponding to the image.
$$
\mathcal {L} _ {\mathrm {s e g}} (\hat {O}, O) = \frac {1}{H W} \| O - \hat {O} \| _ {1}, \tag {3}
$$
where  $\hat{O}, O \in [0,1]^{H \times W}$  are the inferred and ground-truth occupancy masks, respectively. We provide visual comparison of the inferred geometry for this loss in the supplement.
Multi-Frame Training. Because our new decoder have a large capacity, generating a volume incur much larger overhead compared to rendering an image based on this volume (which mostly consists of tri-linear sampling of the voxel cube). Thus, rather than rendering a single view for the canonical representation of the target object in each batch, we instead render 4 views for each object in the batch. This technique incurs no significant overhead, and effectively increases the batch size four times. As
an added benefit, we find that this technique improves on the overall quality of the generated results, since it significantly reduce batch variance. We ablate this technique and our key architectural design choices, showing their effect on the sample quality (Sec. 4.3, Tab. 2).
Learning Non-Rigid Objects. For articulated, non-rigid objects, e.g. videos of human subjects, we must model a subject's shape and local motion from dynamic poses, as well as the corresponding non-rigid deformation of local regions. Following [70], we assume these sequences can be decomposed into a set of  $N_{p}$  smaller, rigid components (10 in our experiments) whose poses can be estimated for consistent alignment in the canonical 3D space. The camera poses for each component are estimated and progressively refined during training, using a combination of learned 3D keypoints for each component of the depicted subject and the corresponding 2D projections predicted in each image. This estimation is performed via a differentiable Perspective-n-Point (PnP) algorithm [40].
To combine these components with plausible deformations, we employ a learned volumetric linear blend skinning (LBS) operation. We introduce a voxel grid  $V^{LBS} \in \mathbb{R}^{S^3 \times N_p}$  to represent the skinning weights for each deformation components. As we assume no prior knowledge about the content or assignment of object components, the skinning weights for each component are also estimated during training. Please see the supplement for additional details.
3.3 Latent 3D Diffusion
Architecture. Our diffusion model architecture extends prior work on diffusion in a 2D space [36] to the latent 3D space. We implement its 2D operations, including convolutions and self-attention layers, in our 3D decoder space. In the text-conditioning experiments, after the self-attention layer, we use a cross-attention layer similar to that of [63]. Please see the supplement for more details.