SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\tilde{\beta}_{t} = (1 - \bar{\alpha}_{t - 1}) / (1 - \bar{\alpha}_{t})\beta_{t}$ . Notably,  $\mathcal{V}_0$  is the generation target that is unknown. Thus, we need to learn a denoiser  $g_{\Phi}$  parameterized by  $\Phi$  to approximate these conditional probabilities  $p_{\Phi}(\mathcal{V}_{t - 1}|\mathcal{V}_t)\approx q(\mathcal{V}_{t - 1}|\mathcal{V}_t,\mathcal{V}_0)$ , i.e., estimate the mean  $\tilde{\mu} (\cdot)$ . In practice, we train the denoiser  $g_{\Phi}$  to predict  $\mathcal{V}_0$  such that:
$$
\mu_ {\Phi} \left(\mathcal {V} _ {t}, t\right) = \frac {1}{\sqrt {\alpha} _ {t}} \left(\mathcal {V} _ {t} - \frac {1 - \alpha_ {t}}{1 - \bar {\alpha} _ {t}} \left(\mathcal {V} _ {t} - \sqrt {\bar {\alpha} _ {t}} g _ {\Phi} \left(\mathcal {V} _ {t}, t\right)\right)\right), \tag {7}
$$
where the posterior probabilities can be approximated as  $p_{\Phi}(\mathcal{V}_{t - 1}|\mathcal{V}_t) = \mathcal{N}(\mathcal{V}_{t - 1};\mu_\Phi (\mathcal{V}_t,t),\tilde{\beta}_t\mathbf{I})$  During inference, this posterior is sampled at each time step  $t$  to gradually denoise the noisy sample  $\mathcal{V}_t$  . The denoiser  $g_{\Phi}$  is trained to minimize the difference from  $\tilde{\mu}$  which can be simplified [17] as:
$$
\mathcal {L} _ {t} ^ {\text {s i m p l e}} = \mathbb {E} _ {t \sim [ 1, T ], \mathcal {V} _ {0}, \mathcal {V} _ {t}} [ | | \mathcal {V} _ {0} - g _ {\Phi} (\mathcal {V} _ {t}, t) | | _ {2} ^ {2} ]. \tag {8}
$$
Dataset. We obtain 796 high-quality 3D humans from RenderPeople [55] with diverse identities and clothes. For each person, we repose the mesh with 20 different human poses [31] to ensure pose diversity. For each pose instance of a person, we render 36 multi-view images with known camera poses. All methods are trained from scratch (except mentioned) on this dataset for fair comparisons.
Figure 5: Qualitative comparisons of unconditional 3D human generation between PrimDiffusion and baseline methods. RGB renderings and corresponding depths are placed side-by-side.
Table 1: Quantitative results on unconditional 3D human generation. The top three techniques are highlighted in red, orange, and yellow, respectively. In this table, "ft" indicates a finetuned model that is pre-trained on the DeepFashion [26] dataset.
Methods	FIDCLIP ↓	FID ↓	KID×102↓	PCK ↑	Depth×102↓	FPS ↑
StyleSDF [37]	18.55	51.27	4.08 ± 0.13	-	49.37 ± 22.18	2.53
EG3D [5]	19.54	24.32	1.96 ± 0.10	-	16.59 ± 21.03	22.97
EVA3D [18]	15.03	44.37	2.68 ± 0.13	91.84	3.24 ± 9.93	6.14
EVA3D ft [18]	14.58	40.40	2.99 ± 0.14	91.30	2.60 ± 7.95	6.14
Ours	12.11	17.95	1.63 ± 0.09	97.62	1.42 ± 1.78	88.24
Implementation Details. We use  $K = W^{2} = 1024$  primitives to represent each 3D human. The denoiser  $g_{\Phi}$  is implemented as a 2D U-Net [45] with intermediate attention layers. We first train the generalizable encoder using all available images. Then, the denoiser  $g_{\Phi}$  is trained according to the primitives produced by the frozen encoder. Please check the supplementary material for more details.
Evaluation Metrics. We adopt Fréchet Inception Distance (FID) [16] and Kernel Inception Distance (KID) [4] to evaluate the quality of rendered images. Note that, we use different backbones to evaluate FID in different latent spaces in order to get a thorough evaluation, i.e.,  $\mathrm{FID}_{\mathrm{CLIP}}$  employs the CLIP image encoder [40] while FID employs the Inception-V3 model [54]. To evaluate the 3D geometry, we use an off-the-shelf tool [42] to estimate depth maps from renderings and compute L2 distance against rendered depths. We further adopt a human-centric metric, Percentage of Correct Keypoints (PCK) [2], to evaluate the pose controllability of 3D human generative models.
Comparison Methods. As the first diffusion model for 3D human generation, we compare with three GAN-based methods. EVA3D [18] learns to generate 3D human from 2D image collections. EG3D [5] and StyleSDF [37] are approaches for 3D-aware generation from 2D images, succeeding in generating human faces and objects. All of these methods enable view control by implicitly conditioning on the camera pose, which leads to extra forward passes upon viewpoint changes.
We show the RGB renderings and corresponding depth maps generated by PrimDiffusion and baselines in Fig. 5. StyleSDF and EG3D achieve coarse results in terms of appearance and geometry. However, due to the lack of human prior, most of the capacity is wasted on modeling the empty space that is not occupied by the human body, which leads to blurry facial details. Furthermore, the inefficient 3D representations make them hard to scale up to high resolution without super-resolution modules, which also limits their geometry to low resolution ( $64^2$  or  $128^2$ ). EVA3D succeeds in high-resolution rendering with much better geometry by incorporating human prior. However, it fails to generate enough facial details. We attribute this to its part-wise human representation. For example, EVA3D models the human head as a single volume, assigning a difficult learning problem of distinguishing facial details within one uniform representation. In contrast, PrimDiffusion divides humans into many primitives tailed with powerful diffusion modeling, which is not only capable of
Prnreee (Ours)
Figure 6: Qualitative ablation study of different human representations for 3D diffusion. Alternatively, we replace primitives with voxel grids as human representation to validate the effectiveness. RGB renderings, depths, and representation occupancy are placed sequentially for each sample.
Table 2: Quantitative ablation study of different human representations for 3D diffusion. Note that, the "Resolution" denotes the 3D spatial resolution of the representation.
Methods	Resolution	FIDCLIP ↓	FID ↓	KID×102 ↓	Depth×102 ↓
Voxel	64 × 64 × 64	19.53	81.78	7.72 ± 0.17	8.22 ± 3.78
Latents [38]	-	12.13	37.70	3.44 ±0.11	1.76 ± 1.91
Ours	256 × 256 × 8	12.11	17.95	1.63 ± 0.09	1.42 ± 1.78
modeling enough details (e.g., faces and textures) but also supporting  $512 \times 512$  resolution for both RGB and depth without any super-resolution decoder.
The results of numerical comparisons are presented in Tab. 1. As general 3D generation approaches, StyleSDF and EG3D get fair scores in terms of generation quality, but the geometries are much worse than others. EVA3D generates good geometries, indicated by its comparable depth error with us. However, the image quality is still far behind us. Note that, we observe the vanilla FID might be inaccurate, which leads to discrepancies between visualizations and numerical results. For example, the images rendered by EG3D are of poor quality but the FID is much lower than other baselines. Therefore, we report  $\mathrm{FID}_{\mathrm{CLIP}}$  metric which we found more consistent with qualitative results. Nevertheless, the proposed method still outperforms baselines by a large margin.
Moreover, we also compare our model with a finetuned EVA3D that is first pre-trained on DeepFashion [26] dataset to establish 3D representations of humans and then finetuned on our RenderPeople dataset. The motivation for this experiment is to fully leverage the ability to learn 3D representations on both multiview images and collections of images for methods like EVA3D instead of training from scratch with multiview images only. However, as shown in Tab. 1, the finetuned EVA3D still has a performance gap compared with our method.
Different Representations for 3D Diffusion. Apparently, the volumetric primitive is not the only representation for 3D human diffusion models. We also explore the effectiveness of other representations, like voxel grids [53] and latent codes [38]. The "voxel" baseline differentiates itself
Table 3: Ablation study of design choices for volumetric primitives fitting. The metrics PSNR, SSIM [58], and LPIPS [63] are averaged across all training identities, views, and poses.
Methods	PSNR ↑	SSIM ↑	LPIPS ↓
w/o varying δs	18.35	0.803	0.182
w/o R, T	31.70	0.982	0.058
w/o attention	28.39	0.962	0.081
Ours	32.15	0.984	0.048
Figure 7: Visualizations on effects of different design choices for volumetric primitives fitting. "GT" denotes the ground truth image.
from our method in two ways: 1) it utilizes voxel grids with a resolution of  $64 \times 64 \times 64$  for fitting 3D representation from multi-view images, and 2) it leverages 3D convolution layers instead of 2D for instantiation of  $g_{\Phi}(\cdot)$ . Note that, as 3D convolution layers occupy much more memories than 2D, the feasible resolution of voxel grids is rather limited. Moreover, the "latents" baseline inherits the idea of anchoring latent codes [38] to the vertices of the SMPL model for 3D human representation learning. However, the vanilla NeuralBody [38] takes 14 hours per subject to converge which makes it unscalable to large-scale datasets like RenderPeople which contains lots of identities. Thus, we implement the "latents" baseline in the following way. We replace the payload of volumetric primitives with latent codes and leverage sparse convolution layers as the decoder to output the color and density of radiance fields. This implementation of a NeuralBody version of our method keeps the ability to learn cross-identity 3D representation through our generalizable primitive learning framework while getting rid of per-subject fitting.
Quantitative results are presented in Tab. 2. The voxel-based representation fails to achieve a reasonable result both in rendering quality and geometry. The latent codes baseline achieves plausible generated results but worse quality compared with ours. We further visualize the qualitative comparisons in Fig. 6. The voxel-based approach wastes many parameters to densely model the scene, leading to the low spatial resolution of the representation. This is the root cause of its blurry renderings, which validates the importance of our primitive-based representation for 3D human diffusion model.
Ablation on Design Choices for Primitives Fitting. The quality of primitives fitting significantly affects the generation quality, as it provides 3D ground truth for the diffusion model. Therefore, we ablate the effectiveness of different designs for generalizable primitives fitting, presented in Fig 7 and Tab. 3. Note that, we use reconstruction metrics for evaluation. We denote "w/o varying  $\delta s$  as the baseline that replaces the spatially-varied delta scale factor  $\delta s$  with a uniformly distributed scale factor, i.e., all primitives share the same delta scale factor. It fails to reconstruct reasonable 3D humans. The "w/o  $R$ ,  $T$  denotes the method that uses predefined rotation and translation calculated from a template human mesh instead of an identity-specific one, which leads to a slight drop in quality. And "w/o attention" denotes the method trained without our proposed cross-modal attention module, which is important for detailed textures and motion-dependent effects.
Real-time Inference. Thanks to our efficient representation and decoder-free rendering procedure, PrimDiffusion can render 3D consistent humans with varying viewpoints and poses in a resolution of  $512 \times 512$  at 88.24 FPS once the denoising process is done. As shown in Tab. 1, due to the use of the decoder which decodes the 3D representation to images, all baselines fail to high-resolution real-time rendering at a comparable quality with us. Furthermore, these methods implicitly model 3D-aware conditions, i.e., viewpoints and poses, which causes extra forward pass calls upon conditions changes.
Generalizability to Novel Poses. It is worth mentioning that a decoder (e.g., MLP for NeRF decoding or CNN for super-resolution) may lead to overfitting to the pose distribution of the training dataset, which prohibits pose generalization to out-of-distribution. Another merit of our decoder-free rendering is that we can generalize to novel poses without test-time optimization, post-processing, and rigging (Fig. 1 and Fig. 4). We refer readers to the supplementary material for video results.
Figure 8: Application of PrimDiffusion. a) Thanks to the dense correspondence offered by volumetric primitives, we can transfer the texture in a 3D consistent way. b) By masking the corresponding primitives with Gaussian noise and performing the denoising process with mask guidance, our method can achieve the conditional generative task, i.e., 3D inpainting, without retraining the model.
Texture Transfer. Based on the dense correspondence offered by volumetric primitives, we can transfer the texture from one human body to another, as shown in Fig. 8 a). Since the entire process is operated in 3D space, we can render the human with transferred texture with free viewpoints.
3D Inpainting. Thanks to the flexibility of primitive-based representation and diffusion model, PrimDiffusion can inpaint the masked region on the human body with new textures as shown in Fig. 8 b). By perturbing the masked region with noise, we take the volumetric primitives as the mask guidance to the denoiser  $g_{\Phi}(\cdot)$ . Note that, in contrast to GAN-based methods, this conditional generation is done without retraining the model.
In this paper, we propose the first diffusion model for 3D human generation. To offer a compact and expressive parameter space with human prior for diffusion models, we propose to directly operate the diffusion and denoising process on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This flexible yet efficient representation enables high-performance rendering for novel views and poses, with a resolution of  $512 \times 512$  in real-time. Furthermore, we also propose a cross-modal attention module tailored with an encoder-based pipeline for fitting volumetric primitives from multi-view images without per-subject optimization. We believe our method will pave the way for further exploration in 3D human generation.
Limitation. While our method shows promising results for unconditional 3D human generation, several limitations remain. First, though we can model off-body topology like loose garments and long hair, how to animate them remains a challenge. Second, our renderings contain artifacts for complex textures. Leveraging 2D pre-trained generative models would be a possible solution. Last, due to the high degree of freedom of primitives, our method is a two-staged pipeline to prevent training instability. Further explorations on single-stage 3D diffusion models would be fruitful. In addition, compared with GAN-based approaches, our method requires a sufficient number of posed views to get good 3D representations for training diffusion models. In this context, it would be interesting to explore few-shot inverse rendering techniques to reduce the reliance on multiview images for the first stage of our method.
Broader Impacts. The generated 3D humans of PrimDiffusion might be misused to create misleading content or fake media. It can also edit personal imagery, leading to privacy concerns.
This work is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-019), NTU NAP, MOE AcRF Tier 2 (T2EP20221-0012), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
Vision and Pattern Recognition, pages 4563-4573, 2023. 1, 3, 5
[58] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. 9
[59] Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds. Technical Report arXiv:2206.07255, arXiv, June 2022. 2
[60] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3D shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[61] Jianfeng Zhang, Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: A 3D generative model for animatable human avatars. In Arxiv, 2022. 2, 3
[62] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.3
[63] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018. 9
[64] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence, 44(6):3170-3184, 2021. 3
[65] Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5826–5835, October 2021. 3