SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the ablation section of our main paper, we experimented on different intermediate representation resolutions to train our diffusion model on. In Fig. 10 we show results of training only on the single vector we use as input to the autodecoder. Similarly to our  $4 \times 4 \times 4$  run, we find the generates samples to be of low quality, failing to properly capture the geometry of the objects of the Chair dataset [57]. We hypothesize this compressed vector overfits to the training dataset, and diffusion struggles to fully capture the properties of the dataset distribution.
A.5 Large and Diverge 3D Synthesis with Adversarial Methods
Adversarial 3D generation methods such as EG3D [7] and GET3D [21] have shown generative results of high perceptual quality when trained on single object categories. We experimented with trained them on Objaverse [14] our biggest and most diverse datasets, without utilizing any conditioning. In this setting, these methods fail to produce meaningful samples as we see in Fig. 11. In both cases, we see that the results stop to improve after a few epochs, with the FID never falling below 100, compared to our methods 40.49. We believe that without additional supervision such as class labels, these methods based methods will not produce good results.
Figure 7: Qualitative comparison of unconditional generation using DiffRF [49] (left) and our approach (right) on the ABO Tables dataset [13]. In contrast to DiffRF, we train diffusion in the latent features of an autodecoder. Decoupling the expensive and demanding training from the output voxel-grid size lets us increase the resolution of our 3D representation. For this dataset, our output voxel resolution is  $128^3$ , compared to the  $32^3$  resolution of DiffRF. Our method improves the perceptual quality of the results, as it as shown in the reported FID and KID.
Figure 8: Qualitative comparison of unconditional generation using DiffRF [49] (left) and our approach (right) on the PhotoShapes Chairs dataset [57]. For this dataset, our output voxel resolution is  $64^3$ . As above, our results are both qualitatively and quantitatively superior.
Figure 9: Training the autodecoder on a single view per object on the Chairs dataset [57]. the autodecoder offers a compressed representation of the dataset; it encapsulates prior knowledge. In contrast to a single-scene NeRF, our method can work with only a single view per object for single-category datasets. Our method can roughly learn the shape of the objects from multiple instances, but it struggles with textures and geometry details such as chair legs, so multiple views is still beneficial for precise reconstruction.
Figure 10: Latent Diffusion Model trained on the embedding vectors of a pre-trained autodecoder. Similarly, with our training on feature grid resolution of  $4 \times 4 \times 4$ , training on the vector embedding struggles to generate the concrete geometry of the objects in the Chairs dataset [57]
Figure 11: Adversarial-based methods for large-scale and diverse 3D synthesis trained on Objaverse [14]. While gan-based 3d generation methods EG3D [7] and GET3D [21] produce high-quality results for single object datasets, we observe that they struggle to converge in a large-scale and diverse setting such as Objaverse [14]
A.6 Animated Results
Please visit our project's web page (https://github.com/snap-research/3DVADER) for additional video results, showing consistent novel-view synthesis for rigid objects from multi-category datasets and animated articulated objects sampled using our approach, and results demonstrating both conditional and unconditional generation.
B Method Details
B.1 Volumetric Autodecoder
Volumetric Rendering. We use learnable volumetric rendering [48] to generate the final images from the final decoded volume. Given a camera intrinsic and extrinsic parameters for a target image, and the radiance field volumes generated by the decoder, for each pixel in the image, we cast a ray through the volume, sampling the color and density values to compute the color  $C(\mathbf{r})$  by integrating the radiance along the ray  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ , with near and far bounds  $t_n$  and  $t_f$ :
$$
C (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \delta (\mathbf {r} (t)) \mathbf {c} (\mathbf {r} (t), \mathbf {d}) d t, \tag {7}
$$
where  $\delta$ ,  $\mathbf{c}$  are the density and RGB values from the radiance field volumes sampled along these rays, and  $T(t) = \exp \left\{-\int_{t_n}^t\sigma (\mathbf{r}(s))ds\right\}$  is the accumulated transmittance between  $t_n$  and  $t$ .
To supervise the silhouette of objects, we also render the 2D occupancy map  $O$  using the volumetric equation:
$$
O (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \delta (\mathbf {r} (t)) d t. \tag {8}
$$
We sample 128 points across these rays for radiance field rendering during training and inference.
Articulated Animation. As our approach is flexibly designed to support both rigid and articulated subjects, we employ different approaches to pose supervision to better handle each of these cases.
For articulated subjects, poses are estimated during training, using a set of learnable 3D keypoints  $K^{3D}$  and their predicted 2D projections  $K^{2D}$  in each image in an extended version of the Perspective-n-Point (PnP) algorithm [40]. To handle articulated animation, however, rather than learn a single pose per image using these points, we assume that the target subjects can be decomposed into  $N_{p}$  regions, each containing  $N_{k}$  points  $K_{p}^{3D}$  points and their corresponding  $K_{p}^{2D}$  projections per image. These points are shared across all subjects, and are aligned in the learned canonical space, allowing for realistic generation and motion transfer between these subjects. This allows for learning  $N_{p}$  poses per-frame defining the pose of each region  $p$  relative to its pose in the learned canonical pose.
To successfully reconstruct the training images for each subject thus requires learning the appropriate canonical locations for each region's 3D keypoints, to predict the 2D projections of these keypoints in each frame, and the pose best matching the 3D points and 2D projections for these regions. We can then use this information in our volumetric rendering framework to sample appropriately from the canonical space such that the subject's appearance and pose are consistent and appropriate throughout their video sequence. Using this approach, this information can be learned along with our autodecoder parameters for articulated objects using the reconstruction and foreground supervision losses used for our rigid object datasets.
As noted in Sec. 3.2, to better handle non-rigid shape deformations corresponding to this articulated motion, we employ volumetric linear blend skinning (LBS) [41]. This allows us to learn the weight each component  $p$  in the canonical space contributes to a sampled point point in the deformed space based on the spatial correspondence between these two spaces:
$$
x _ {d} = \sum_ {p = 1} ^ {N _ {p}} w _ {p} ^ {c} \left(x _ {c}\right) \left(R _ {p} x _ {c} + \operatorname {t r} _ {p}\right), \tag {9}
$$
where  $T_{p} = [R_{p}, t_{p}] = [R^{-1}, -R^{-1}\mathrm{tr}]$  is the estimated pose of part  $p$  relative to the camera (where  $T = [R, \mathrm{tr}] \in \mathbb{R}^{3 \times 4}$  is the estimated camera pose with respect to our canonical volume);  $x_{d}$  is the
3D point deformed to correspond to the current pose;  $x_{c}$  is its corresponding point when aligned in the canonical volume; and  $w_{p}^{c}(x_{c})$  is the learned LBS weight for component  $p$ , sampled at position  $x_{c}$  in the volume, used to define this correspondence.
Thus, for our non-rigid subjects, in addition to the density and color volumes needed to integrate Eqns. 7 and 8 above, our autodecoder learns to produce a volume  $V^{LBS} \in \mathbb{R}^{S^3 \times N_p}$  containing the LBS weights for each of the  $N_p$  locally rigid regions constituting the subject.
We assign  $N_{k} = 125$  3D keypoints to each of the  $N_{p} = 10$  regions. For these tests, we assume fixed camera intrinsics with a field-of-view of 0.175 radians, as in [54]. We use the differentiable Perspective-n-Point (PnP) algorithm [40] implementation from PyTorch3D [62] to accelerate this training process.
As this approach suffices for objects with standard canonical shapes (e.g., human faces) performing non-rigid motion in continuous video sequences, we employ this approach for our tests on the CelebV-Text dataset. While in theory, such an approach could be used for pose estimation for rigid objects (with only 1 component) in each view, for we find that this approach is less reliable for our rigid object datasets, which contain sparse, multi-view images from randomly sampled, non-continuous camera poses, depicting content with drastically varying shapes and appearances (e.g., the multi-category object datasets described below). Thus, for these objects, we use as input either known ground-truth or estimated camera poses (using [66]), for synthetic renderings or real images, respectively. While some works [83, 42, 87] perform category-agnostic object or camera pose estimation without predefined keypoints from sparse images of arbitrary objects or scenes, employing such techniques for such data is beyond the scope of this work.
Architecture. Our volumetric autodecoder architecture follows that of [70], with the key extensions described in this work. Given an embedding vector  $e$  of size 1024, we use a fully-connected layer followed by a reshape operation to transform it into a  $4^3$  volume with 512 features per cell. This is followed by a series of four 3D residual blocks, each of which upsamples the volume resolution in each dimension and halves the features per cell, to a final resolution of  $64^3$  and 32 features. These blocks consist of two  $3 \times 3 \times 3$  convolution blocks each followed by batch normalization in the main path, while the residual path consists of four  $1 \times 1 \times 1$  convolutions, with ReLU applied after these operations. After the first of these blocks we have the  $8^3$  volume with 256 features per cell used for training our diffusion network, as in our final experiments. In this and the subsequent block, we apply self-attention layers [81] as described in Sec. 3.1. After the final upsampling block, we apply a final batch normalization followed by a  $1 \times 1 \times 1$  convolution to produce the final  $1 + 3$  density  $V^{\mathrm{Density}}$  and RGB color features  $V^{\mathrm{RGB}}$  used in our volumetric renderer.
Non-Rigid Architecture. For non-rigid subjects, our architecture produces  $1 + 3 + 10$  output channels, with the latter group with the LBS weights for the  $n_p = 10$  locally rigid components each region corresponds to in our canonical space. Our unsupervised 2D keypoint predictor uses the U-Net architecture of [69], which operates on a downsampled  $64 \times 64$  input image to predict the locations of the keypoints corresponding to each of the 3D keypoints used to determine the pose of the camera relative to each region of the subject when it is aligned in the canonical volumetric space.
B.2 Latent 3D Diffusion
Diffusion Architecture and Sampling. For our base diffusion model architecture, we use the Ablated Diffusion Model (ADM) of Dhariwal et al. (2021) [17], a U-Net architecture originally designed for 2D image synthesis. We incorporate the preconditioning enhancements to this model described in Karras et al. (2022) [36]. As this architecture was originally designed for 2D, we adapt all convolutions and normalizations operations, as well as the attention mechanisms, to 3D.
For the cross-attention mechanism used for our conditioning experiments, we likewise extend the latent-space cross-attention mechanism from Rombach et al. (2022) [63] to our 3D latent space.
Robust Normalization. Autoencoder-based latent diffusion models impose a prior to the learned latent vector [63]. We find the latent features learned by our 3D autodecoder already form a bell-like curve. However, we also observe extreme values that can severely affect the calculation of the
<sup>3</sup>In practice, as in [70], we compute an approximate solution using the inverse LBS weights following HumanNeRF [84] to avoid the excessive computation required by the direct solution.
<sup>4</sup>We add one block to upsample to  $128^{3}$  for our aforementioned experiments with the ABO Tables dataset.
Figure 12: We present the latent feature distribution of a 3D AutoDecoder trained on MVImgNet[92]. The features are extracted at the  $8^{3}$  resolution, where we apply diffusion. The three subplots show different levels of "zooming in." We see that the distribution spans a great range due to extreme outliers. Using classic mean and standard deviation computation, as we see in the middle subplot, still provides quite a large range of values. Normalizing the features using classic statistics leads to convergence failure for the diffusion model. We propose using robust statistics to normalize the distribution to  $[-1,1]$ , before training the diffusion model. During inference, we de-normalize the diffusion output before feeding them to the upsampling layers of the autodecoder.
Figure 13: Qualitative comparison of models trained at different latent resolutions. All visualizations produced with 64 diffusion steps. We find that the model train on  $8^{3}$  latent features gives the best trade-off between quality and training speed, rendering it the best option for training on large-scale 3D datasets.
mean and standard deviation. As discussed in the main manuscript, we deploy the use of robust normalization to adjust the latent features. In particular, we take the median  $m$  as the center of the distribution and approximate its scale using the Normalized InterQuartile Range (IQR) [85] for a normal distribution:  $0.7413 \times IQR$ . We visualize its effect in Fig. 12. This is a crucial aspect of our approach, as in our experiments we find that without it, our diffusion training is unable to converge.
Ablating the latent volume resolution used for diffusion. We trained three diffusion models models for the same time, resources, and number of parameters, for diffusion at 3 resolutions in our autodecoder:  $4^{3}$ ,  $8^{3}$ , and  $16^{3}$ . We find that the  $4^{3}$  models, even when they train faster, often fail to converge to something meaning full and produce partial results. Most samples produced by the  $16^{3}$  models are of reasonable quality. However, many samples also exhibit spurious density values. We hypothesize that this is due to the model being under-trained. The  $8^{3}$  model produces the best results, and its fast training speed makes it suitable for large-scale training. We visualize the results in Fig. 13
B.3 Hash Embedding
Each object in the training set is encoded by an embedding vector. However, as we employ multiview datasets of various scales, up to  $\sim 300\mathrm{K}$  unique targets from multiple categories, storing a separate embedding vector for each object depicted in the training images is burdensome  $^{5}$ . As such,
${}^{5}E$  .g.,the codebook alone would require six times the parameters of the largest model in our experiments.
we experimented with a technique enabling the effective use of a significantly reduced number of embeddings (no more than  $\sim 32\mathrm{K}$  are required for any of our evaluations), while allowing effective content generation from large-scale datasets.
Similar to the approach in [55], we instead employ concatenations of smaller embedding vectors to create more combinations of unique embedding vectors used during training. For an embedding vector length  $l_{v}$ , the input embedding vector  $H_{k} \in \mathbb{R}^{l}$  used for an object to be decoded is a concatenation of smaller embedding vectors  $h_{i}^{j}$ , where each vector is selected from an ordered codebook with  $n_c$  entries, with each entry containing collection of  $n_h$  embedding vectors of length  $l_{v} / n_{c}$ :
$$
H _ {k} = \left[ h _ {1} ^ {k _ {1}}, h _ {2} ^ {k _ {2}}, \dots , h _ {n _ {c}} ^ {k _ {n _ {c}}} \right], \tag {10}
$$
where  $k_{i} \in \{1,2,\dots,n_{h}\}$  is the set of indices used to select from the  $n_h$  possible codebook entries for position  $i$  in the final vector. This method allows for exponentially more combinations of embedding vectors to be provided during training than must be stored in learned embedding vector library.
However, while in [55], the index  $j$  for the vector  $h_i^j$  at position  $i$  is randomly selected for each position to access its corresponding codebook entry, we instead use a deterministic mapping from each training object index to its corresponding concatenated embedding vector. This function is implemented using a hashing function employing the multiplication method [16] for fast indexing using efficient bitwise operations. For object index  $k$ , the corresponding embedding index is:
$$
m (k) = \left[ \left(a \cdot k\right) \bmod 2 ^ {w} \right] \gg (w - r), \tag {11}
$$
where the table has  $2^r$  entries.  $w$  and  $a$  are heuristic hashing parameters used to reduce the number of collisions while maintaining an appropriate table size. We use 32 for  $w$ .  $a$  must be an odd integer between  $2^{w - 1}$  and  $2^w$  [16]. We give each smaller codebook its own  $a$  value:
$$
a _ {i} = 2 ^ {w - 1} + 2 * i ^ {2} + 1, \tag {12}
$$
where  $i$  is the index of the codebook.
Discussion. In our experiments, we found that employing this approach had negligible impact on the overall speed and quality of our training and synthesis process. During training the memory of the GPU is predominantly occupied by the gradients, which are not affected by this hashing scheme. For Objaverse, our largest dataset using  $\sim 300\mathrm{K}$  images, using this technique saves approximately 800MB of GPU memory.
Interestingly, this also suggests that scaling this approach to larger datasets, should they become available, will require special handling. Learning this per-object embedding would soon become intractable. However, simply using this hash embedding approach reduces the model storage requirements by  $\sim 75\%$  for this dataset.
In our experiments, we use hashing for ABO Tables, CelebV-Text and Objaverse, with codebook sizes  $n_c =$  of 256, 8192 and 32768, respectively. We set the number of smaller codebooks  $(n_h)$  to 256 for each dataset.
C.1 Dataset Filtering
CelebV-Text [90]. Some heuristic filtering was necessary to obtain sufficient video quality and continuity for our purposes. We omit the first and last  $10\%$  of each video to remove fade-in/out effects, and any frames with less than  $25\%$  estimated foreground pixels. We also remove videos with less than 4 frames remaining after this, and any videos less than 200 kilobytes due to their relatively low quality. We also omit a small number of videos that were unavailable for download at the time of our experiments (the dataset is provided as a set of URLs for the video sources).
MVImgNet [92]. For these annotated video frames depicting real objects in unconstrained settings and environments, we applied Grounded Segment Anything [39] for background removal. However,
as this process sometimes failed to produce acceptable segmentation results, we apply filtering to detect these case. We first remove objects for which Grounding DINO [45] fails to detect bounding boxes. We then fit our volumetric autodecoder (Secs. 3.1-2) to only the masks produced by this segmentation (as monochrome images with a white foreground and a black background). For objects that are properly segmented in each frame, this produces a reasonable approximation of the object's shape that is consistent in each of the input frames, while objects with incorrect or inconsistent segmentation will not be fit properly to the input images. Thus, objects for which the fitting loss is unusually high are removed.
Objaverse [14]. While Objaverse contains  $\sim 800\mathrm{K}$  3D models, we found that the overall quality of these varied greatly, making many of them unsuitable for multi-view rendering. We thus filtered models without texture, material maps, or other color and appearance properties suitable, as well as models with an insufficient polygon count for realistic rendering. Interestingly, given the simplicity of the objects when rendered against a monochrome background, we found that the foreground segmentation supervision used for the other experiments described in Sec. 3.2 of the main paper was unnecessary. Given the scale of this dataset ( $\sim 300\mathrm{K}$  unique objects, with 6 frames per object), we thus omit this loss from our training process for this dataset for our final experiments for the sake of improved training efficiency. For datasets with more complex motion and real backgrounds, such as the real image datasets mentioned above, we found this supervision to be essential, as shown in Sec. A.2 and Fig. 6.
C.2 Additional Details
Training Details. Our experiments are implemented in the PyTorch [58, 59], using the PyTorch Lightning [19] framework for fast automatic differentiation and scalable GPU-accelerated parallelization. For calculating the perceptual metrics (FID and KID), we used the Torch Fidelity [56] library.
We run our experiments on 8 NVIDIA A100 40GB GPUs per node. For some experiments, we use a single node, while for larger-scale experiments, we use up to 8 nodes in parallel.
We use the Adam optimizer [37] to train both the autodecoder and the diffusion Model. For the first network, we use a learning rate  $lr = 5e - 4$  and beta parameters  $\beta = (0.5, 0.999)$ . For diffusion, we set the learning rate to  $lr = 4.5e - 4$ . We apply linear decay to the learning rate.
Preparing the Text Embeddings for Text-Driven Generation. We train our model for text-conditioned image generation on three datasets: CelebV-Text [90], MVImgNet [92] and Objaverse [14]. The two latter datasets provide the object category of each sample, but they do not provide text descriptions. Using MiniGPT4 [94], we extract a description by providing a hint and the first view of each object along with the question: “<Img><ImageHere></Img> Describe this <hint> in one sentence. Describe its shape and color. Be concise, use only a single sentence.” For MVImgNet, this hint is the “class name”, while it is the “asset name” for Objaverse.
Note this approach is not foolproof. To the contrary, we find that in many cases MiniGPT4 hallucinates descriptive characteristics of the object that do not match its visual input. We can see some examples like this in Fig. 14
With the text-image pairs for these three datasets, we use the 11-billion parameter T5 [61] model to extract a sequence of text-embedding vectors. The dimensionality of these vectors is 1024. During training, we fix the length of the embedding sequence to 32 elements. We trim longer sentences and pad smaller sentences with zeroes.
ABO-Tables	Chairs	CelebV-Text	MVImgNet	Objaverse
3D AutoDecoder
z-length	1024	1024	1024	1024	1024
MaxChannels	512	512	512	512	512
Depth	2	4	2	4	4
SA-Resolutions	8,16	8,16	8,16	8,16	8,16
ForegroundLoss λ	10	10	10	10	0
#Renderers/batch	4	4	4	4	4
VoxelGridSize	1283 × 4	643 × 4	643 × 14	643 × 4	643 × 4
Learning Rate	5e-4	5e-4	5e-4	5e-4	5e-4
Latent 3D Diffusion Model
z-shape	83 × 256	83 × 256	83 × 256	83 × 256	83 × 256
Sampler	edm	edm	edm	edm	edm
Channels	128	128	192	192	192
Depth	2	2	3	3	3
Channel Multiplier	3,4	3,4	3,4	3,4	3,4
SA-resolutions	8,4	8,4	8,4	8,4	8,4
Learning Rate	4.5e-5	4.5e-5	4.5e-5	4.5e-5	4.5e-5
Conditioning	None	None	None/CA	None/CA	None/CA
CA-resolutions	-	-	8,4	8,4	8,4
Embedding Dimension	-	-	1024	1024	1024
Transformers Depth	-	-	1	1	2
Table 5: Architecture details for our models for each dataset. SA and CA stand for Self-Attention and Cross-Attention respectively.  $z$  refers to our 1D embedding vector and our latent 3D volume for the autodecoder and diffusion models, respectively. Note that for CelebV-Text, the output volume has 14 channels per cell: 3 for color values, 1 for density and 10 for part assignment.