PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation
Authors: Zhaoxi Chen
Fangzhou Hong
Haiyi Mei
Guangcong Wang
Lei Yang
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: neural rendering
3D generative model
diffusion model
volumetric primitives
3D human generation
TL;DR: We propose PrimDiffusion, the first diffusion model for 3D human generation.

Abstract: We present PrimDiffusion, the first diffusion-based framework for 3D human generation. Devising diffusion models for 3D human generation is difficult due to the intensive computational cost of 3D representations and the articulated topology of 3D humans. To tackle these challenges, our key insight is operating the denoising diffusion process directly on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This volumetric primitives representation marries the capacity of volumetric representations with the efficiency of primitive-based rendering. Our PrimDiffusion framework has three appealing properties: **1)** compact and expressive parameter space for the diffusion model, **2)** flexible representation that incorporates human prior, and **3)** decoder-free rendering for efficient novel-view and novel-pose synthesis. Extensive experiments validate that PrimDiffusion outperforms state-of-the-art methods in 3D human generation. Notably, compared to GAN-based methods, our PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of $512\times512$ once the denoising process is done. We also demonstrate the flexibility of our framework on training-free conditional generation such as texture transfer and 3D inpainting.

PrinDiffusion: Volumetric Primitives Diffusion for 3D Human Generation
Zhaoxi Chen'
Fangzhou Hong'
Haiyi Mei
Guangeong Wang'
Lei Yang* Ziwei Liu
LS-Lab, Nanyang Technological University Sensetime Research
(zhaoxi001, fangzhou001, guangcong. wang, ziwei. lntl (meihaiyi, yanglei) yanglei]@sensettin @sessetimm...m com
Abstract
We present PrinDiffusion, the first diffusion-based framework for 3D human generation. Devising diffusion models for 3D human generation is difficult due to the intensive computational cost of 3D representations and the articulated topology 3D humans. To tackle these challenges, our key insight is operating the denois- ing diffusion process directly on set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic informa- tion. This volumetric primitives representation marries the capacity of volumetric representations with the efficiency of primitive-base rendering. Our PrimDiffusion framework has three appealing properties: compact and expressive parameter space for the diffusion model, 2) flexible 31) representation that incorporates human prior, and 3) decoder-free rendering for efficient novel-view and novel-pose synthe- sis. Extensive experiments validate that PrimDiffusion outperforms state-of-the- methods in 3D human generation. Notably, compared GAN-based methods, our PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of 512 x 512 once the denoising process is done. We also demonstrate the flexibility of our framework on training-free frree conditional generation such as texture transfer and 3D inpainting.
Introduction
Generating 3D humans with high-resolution and 3D consistency is a central research focus in computer vision and graphics. It has wide-ranging applications in virtual reality, telepresence, and the film industry. These applications are becoming increasingly important as technology advances and the demand for realistic virtual experiences grows. Consequently, a flexible and high-quality 3D human generation system possesses not only immense scientific merit but also practical relevance.
For high-qualige and versatile 3D human generative model, argue that there two key factors: 1) high-capacity generative modeling, and 2) efficient 3D human representation. Recent advances in 3D-aware generative models [5, 18, 35, 37] have made significant progress. By learning from 2D images using generative adversarial networks (GAN) [14], these methods can synthesize images across different viewpoints in a 3D-aware manner. However, their results still leave significant gaps w.r.t. rendering quality, resolution, and inference speed. Besides, these GAN-based methods often require task-specific adaptations be conditioned on particular task, limiting their scope of application. Meanwhile, diffusion- models [17,51] have recently surpassed GANs on generative tasks with unprecedented quality, suggesting their great potential in 3D generation. Several recent works [1, 33, 57] have made initial attempts to generate 3D rigid objects based on the diffusion model. Nevertheless, how to integrate the denoising and diffusion processes into generation framework of articulated 3D humans remains an open problem.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

t Primitives Diffusion Process
Pose View
Shape
Well-defined Depth
Primitives Denoising Process
Pose Generalization
Texture Transfer
Figure I: PrimDiffusion is the first diffusion model for 3D human generation. (Left) We perform the diffusion and denoising process on a set of primitives which compactly represent 3D humans. (Right) This generative modeling enables explicit pose, view, and shape control, with the capability of modeling off- aff-body topology in well-defined depth. Moreover, our method can generalize to novel poses without post-processing and enable downstream human-centir tasks like 3D texture transfer.
Specifically, applying diffusion models to 3D human generation is non-trivial task due to the articulated topology, diverse poses, and various identities, among which an efficient human represen- tation is the most important. Unlike human faces [23] and rigid objects [6, 22] where most existing 3D-aware generative models [1, 5, 15, 33] succeed, the human body exists within the vast expanse of 3D space, occupies only sparse portion of it, characterized by an intricate, articulated topology. An ideal representation should compactly model the human body without wasting parameters on modeling empty space while also enabling explicit view and pose control. The most widely used representation for 3D-aware generative models is based on neural radiance field (NeRF) [32], which can be parameterized by coordinate-basee MLP [15, 37] and hybrid representation like tri-plane [5]. voxel grids [50] or compositional subnetworks [18]. However, these methods model humans as coarse volumes or planes, which limits their generation quality. In addition, they also suffer from slow inference speeds due to the need for dense MLP decoder queries during rendering.
In this paper, propose PrimDiffusion the first diffusion model for 3D human generation. Our key insight is that the denoising and diffusion process can be directly operated on set volumetric primitives, which models the 3D human body kinematic information. Each volume parameterizes the spatially-varied color and density with six degrees of freedom. This representation fuses the capacity of volumetric representations with the efficiency of grimitive-basee rendering, which enables: 1) compact and expressive parameter space for the diffusion model, 2) flexible representation that inheres human prior, and efficient and straightforward decoder-fiie rendering. Furthermore, these primitives naturally capture dense correspondence, facilitating downstream human-centric tasks such texture transfer and inpainting with 3D consistency. Notably, previous GAN- methods [18, 61] implicitly encode control signals (e.g., view directions, poses) as inputs, indicating that they need extra forward passes upon condition changes (e.g., novel view/pose synthesis). In contrast, PrinDiffusion supports real-time rendering of high-quality 3D humans at a resolution of 512 x 512 once the denoising process is done.
number of tiny volumes with radiance and
Extensive experiments are conducted both qualitatively and quantitatively, demonstrating the superi- ority of PrimDiffusion contributions as follows: 1) To the best of our knowledge, we introduce the first diffusion model for 3D human generation. 2) We propose represent 3D humans volumetric primitives a generative context, which enables efficient training and high-performance rendering. 3) We design an encoder tailored with cross- Pross-modal attention, which accounts for learning volumetric primitives from images across identities without per-subject optimization. 4) We demonstrate applications of PrimDiffusion, including texture transfer and 3D inpainting, which can be naturally done without retraining.
tate-of-the-ane methods for 3D human generation. We summarize
Related Work
3D-aware Generation. With recent advances in neural radiance field (NeRF) [32], many studies [5, 9, 11, 15, 18, 35, 37, 49, 50, 59] have incorporated NeRF the key inductive bias to make GANs be

---

(To Ro)
K Primitives
TM, ne
st
(T se)
ga
ct,ak
U-Net
p = (T R Rg.
{cp. oz) V
(Co
Vo
Figure Overview of PrimDiffusion. We represent the 3D human as K primitives learned from multi-view images. Each primitive V has independent kinematic parameters (T R* (translation, rotation, and per-axis scales, respectively) and radiance parameters {e", o²) (color and density). For each time step we diffuse the primitives Vo with noise sampled according to fixed noise schedule. The resulting is fed to g+(() which learns to predict the denoised Vo-
3D-aware, which enables GANs to learn 3D-aware representations from 2D images. Researchers have proposed to use efficient representations like voxel grids [50] and triplanes [5] to improve the spatial resolution of the scene representation, which crucial factor for the quality of geometry and images. However, these techniques necessitate an implicit decoder (e.g.. a Multilayer Perceptron (MLP) or Convolutional Neural Network (CNN)) following the 3D representation to obtain the rendered images, which results in substantial computational overheads associated with neural volume rendering. Consequently, neither the image quality nor the rendering speed are satisfactory.
Diffusion Model. Despite the remarkable success of GAN-based models, diffusion brasee mod- els [17, 51, 52] have recently exhibited impressive performance various generative tasks, especially for 2D tasks like text-to-image synthesis [34, 41, 44, 46]. However, their potential for 3D generation remains largely unexplored. A few attempts have been made on shape [25, 30, 60). point cloud [65], and text-to-31) [39] generation. Recently, some works have succeeded in learning diffusion models from 2D images for unconditional 3D generation [1, 33, 57] of human faces and objects. Still, given the articulated topology of the human body, efficiently training 3D diffusion models is challenging.
Human Generation. Extensive efforts have been made to generate human-centric assets [8, 19, 47, 62, 64], especially for 2D human images [12, 13, 21, 24, 48]. For 3D human generation, initial attempts have been made to generate human shapes [7, 56] using scanned data. Several works learn 3D-aware GANs from 2D image collections driven by differentiable rendering. However, they are either in low-resolution [36] rely super-resolution module [3, 61] which does not guarantee 3D consistency. EVA3D [18] succeeds high-resolution 3D human generation within clean framework. As concurrent work, HumanGen [20] leverages explicit image priors from strong pretrained 2D human image generators and 3D geometry priors from PIFu [47]. However, these 3D-aware GAN-based methods implicitly condition the viewpoints and poses, which causes additional forward calls when conditions change, preventing high-resolution real-time performance.
3 Methodology
The goal of PrimDiffusion is learning to generate 3D) humans from multi-view images built upon denoising and diffusion probabilistic models. It learns revert the diffusion process that gradually corrupts 3D human representations by injecting noise at different scales. In this paper, 3D humans are represented volumetric primitives, facilitating feasible training of diffusion models and efficient rendering. We first introduce this 3D human representation and corresponding rendering algorithm (Sec. 3.1). Then, generalizable encoder employed to robustly fit primitive representations from multi-view images across identities without per-subject optimization (Sec. 3.2). Once the volumetric primitives are learned from images, the problem of 3D human generation is reduced to learning distribution of 3D representations. We thereby present our denoising and diffusion modeling operated on the parameter space of volumetric primitives to generate high-quality 3D humans (Sec.3.3). overview of PrimDiffusion is presented in Fig. 2.
3

---

3.1 Primitive-based 3D Human Representation
To learn denoising and diffusion model with explicit 3D supervision, we need an expressive and efficient 3D human representation that accounts for 1) compact parameter space for diffusion models, 2) articulated topology and motion of 3D humans, and 3) fast rendering with explicit 3D controls.
Volumetric Primitives. Inspired by previous work [28, 43] in reconstructing dynamic human heads and actors, we propose to represent 3D humans as a set of volumetric primitives which acts as the target space for diffusion models. Note that, whereas previous research has centered on the per-subject reconstruction, we enable learning volumetric primitives across multiple identities. This paves the way for shared latent space conducive to generative modeling. In specific. this representation consists of a set of K primitives that jointly parameterize the spatially varied color and density;
V = [V* where V = (T*, R c ,0,).
(1)
Each primitive V is tiny volume, parameterized by translation vector T E R an orientation R E (O))), a three-dimensional scale factor s* E R a spatially-varied color c* E R and density o² R5*8*8, where S is the per-axis base resolution. practice, we set the base resolution of one primitive to S 8, which we find the best to strike balance between reconstruction quality and memory consumption.
To model humans with articulated topology and motion, we attach primitives to the vertices of SMPL [29]. SMPL is parametric human model, defined as ( 0) E R6890x3, where E 1. controls body shapes and 0 E 1272 controls body poses. In this paper, we leverage this parametric human model for the purposes of 1) serving as a geometry proxy to weakly constrain volumetric primitive representations, 2) offering geometry cues for 3D representation fitting, and 3) generalizing to novel poses through linear blending skinning (LBS) algorithm tailored with decoder- free rendering.
Specifically, the kinematic parameters of primitives are modeled relative to the base transformation from SMPL.. Namely, generate W W 2D grid in the UV space of SMPL. The 3D positions of primitives are initialized by uniformly sampling UV space and mapping each primitive to the closest surface point SMPL that corresponds to the UV coordinates of the grid points. The orientation of each primitive is set to the local tangent frame of the 3D surface point. The per-axis scale is calculated based on the gradient of the UV coordinates the corresponding grid point. All primitives are given an initial scale that is proportional to the distances between them and their neighbors. Furthermore, to model off-body topologies like loose clothes and fluffy hair, the scale of primitives is allowed to deviate from the base scale $μ. The scale factor is defined as S = S2. ös,, where $ą is the aforementioned initialized base scale and 9sg is the delta scaling prediction.
Therefore, the learnable parameters of each person modeled by its volumetric primitives V are color E TR. xW x3xSxSXS density a E XRWXWxxxXXXx
and delta scale factor ds E RWxW33
Efficient Decoder-free Rendering. Once the radiance and kinematic information of primitives are determined, we can leverage differentiable ray marching [27] to render the corresponding camera view. For a given camera ray (t) Op + td, with origin Op R and ray direction d, E R the corresponding pixel value Ip is calculated as an integral:
dT(t).
dt, where T(t) = ff" (Ix((t)) dt,
Ip =
(t)
(2)
dt
where bmin and mmax are the near and far bound for points sampling. The c(-) and o()) are global color and density fields derived from trilinear interpolation of color and density attributes among primitives hit by the camera ray. Notably, in contrast to NeRF-basee -ased methods [32] which often require evaluating MLP decoders per camera rays, this rendering process is decoder-free free which can be directly rendered into pixels. Thus, it is far more efficient for both training and rendering.
To animate the volumetric primitives given a pose sequence { Marame we explicitly employ the LBS algorithm on top of SMPL. [29] to produce per-frame human mesh M(3, 0,), which serve as driving signal to determine the kinematic information of the moving volumetric primitives.

---

Motin Branch
T.R.A R.8
Cross Cross-mda A Saatttonnaa agyer
P
W
Scale factor
Fe
Anartionn
SMPL Model (8,0))
Posz foxtumre
Moboc festure
Figh
UV Space
A ppeerance Branch
Color
Unwmp
W
Fi
F
Density
Mult-view Images (t).
Foature
Apperrance Teatume
Decoder- free Rendeting
Figure 3: Encoder architecture for generalizable primitives fitting, To get rid of per-subject optimization, propose encoder that is capable of learning primitives from multi-view images across identities. The encoder consists of a motion branch and an appearance branch, which are fused by the proposed cross-modal attention layer to get kinematic and radiance information of primitives.
3.2 Generalizable Primitive Learning from Multi- View Images
In order to leverage explicit 3D supervision from multi-view images, it is essential to reconstruct 3D representations for each individual to provide ground truth for the diffusion model. While earlier research [57] on 3D diffusion models adhered to per-identity fitting fashion, which could be time- consuming on a large-scale dataset. To overcome this limitation, we propose generalizable encoder to reconstruct 3D human representation across diverse identities without per-subject optimization.
Our setting is learning 3D representations from multi-view images {I, (I where Neam denotes the number of camera views and i is the corresponding index. We first estimate the SMPL [29] parameters ((3, 0) using off-the-shelf tools [10]. Then, we employ an encoder with dual branches parameterize the coarse human mesh well the multi-view images, which outputs the radiance and kinematic information of volumetric primitives. The architecture is illustrated in Figure 3.
Specifically, the encoder consists of two branches, one is for appearance while the other is for motion. For the appearance branch, we back back-propy project the pixels of input images corresponding to all visible vertices of the SMPL model to the UV space. The unwrapped image textures will be averaged across multiple views and fed to the image encoder (I; 4 parameterized by p,, providing texel-aligned appearance features. For the motion branch, take input the local pose parameters of SMPL model, i.e., 9 excluding the first three elements that denote global orientation. The local pose parameters are padded to W W 2D map and fed the motion encoder (0; &a) parameterized by 49. providing motion features. Instead of fusing the features from two branches via naive concatenation, we propose cross-modal attention module to effectively capture the dependencies between appearance and motion. which significantly improves the reconstruction quality of 3D humans (Tab. 3 and Fig. 7). Intuitively, this module captures the interdependence color and motion, e.g., clothes wrinkles and shadows caused by different human poses. Then, the radiance and kinematic information (color, density, and delta scale) of volumetric primitives are separately predicted by three mapping networks (Frgb, F., Fs) conditioned on the fused feature.
The encoder comprises (F,, Figb. Fo, F3} is trained in end-to-end manner across identities instead of per-subject optimization. The learning objective is formulated as a reconstruction task between ground truth images and rendered images:
(3) where Ligh is the image reconstruction loss in L2-norm: Catl is the silhouette loss, Lvol is the volume regularization term [28], and A, are loss weights. The volume regularization is defined as Lvol Σ Prod(s.), where Prod(-) denotes the product of scale factor along three axes. It aims prevent large primitives from occupying empty space that leads loss of resolution.
Lree Argh AalCan + Avol
3.3 Primitive Diffusson Model
Once the volumetric primitives are reconstructed, the problem of 3D human generation is reduced to learning the distribution p(V) with diffusion models. In specific, generate 3D human with its
5

---

Figure 4: Generated 3D humans of PrimDiffusion. Our method can synthesize 3D humans with explicit controls of view, pose and shape, together with well-defined depths. Additionally, can also handle off-body topologies such as bloomy hair and loose garments using primitives. We visualize the RGB, depth, and primitives side-by-side. Please check the supplementary video for more results.
representation as Vo, the diffusion model learns to denoise VT ~ (0, I) progressively closer the data distribution through denoising steps (Vr=1.... ... Vol. Note that, the targets of denoising and and delta scale factor ds E IR W x x3x To form a parameter space with regular shape, we first pad the delta scale ós the same size of color C. Then, we concatenate and reshape them into 3D tensor. With slight abuse of notation, we denote this tensor Vo E R W (]x(W-))XXX))) in the following subsections.
diffusion process are color E R xW x3x5x5x5 density E XW>x>xxxxxx
Forward Process. Given a data point, i.e., volumetric primitives of a human, sampled from a real data distribution Vo q(V), we define the forward diffusion process in which add Gaussian noise to the sample in T steps, producing sequence of noisy samples (Vi,..,, Vr):
q(V.[V-1) = N(v; /1 3,V,-1, 3,1,),
where the step sizes are controlled by a variance schedule {.3, E (0, We use a linear scheduler [17] in our experiments. Denoting on, - 33, and ã; = II' ag can directly sample V, at any arbitrary time step / in a closed form through reparameterization:
II -01/)11)) NoVi I.)..
(5)
Reverse Process. The reverse process, namely the denoising process, aims generate true sample Vo from a Gaussian noise Vy ~ (O I), which requires an estimation of the posterior distribution:
Vo) = NNV--: Vo), 3.D), i(V., Vo) Var -ar(1-ar-1)) ā +
- Vo. (6)
1 Gr
where 3, = (1- - ā, ā /1 8,) Notably, Vo is the generation target that is unknown. Thus, need learn denoiser go parameterized by + approximate these conditional probabilities po (V:-1(V) R ((v-i/V,, Vo), i.e., estimate the mean ji()). In practice we train the denoiser go to predict Vo such that:
1 1 (V: Voz 1 at
1 - 0: (V: Varga (V,, t)))
p4p (V,
where the posterior probabilities can be approximated po (V:-1 .V(V (V,, B.I.. During inference, this posterior sampled at each time step f to gradually denoise the noisy sample V. The denoiser go is trained to minimize the difference from f which can be simplified [17] as
C simple RR..1I,T,..... (11Vo go (V,, 1)
(8)
4 Experiments
Dataset. We obtain 796 high-quality 3D humans from RenderPeople [55] with diverse identities and clothes. For each person, repose the mesh with 20 different human poses [31] to ensure pose diversity. For each pose instance of a person, we render 36 multi-view images with known camera poses. All methods are trained from scratch (except mentioned) this dataset for fair comparisons.

---

a) StyleSDF
b) EG3D
c) EVA3D
d) Ours
Figure 5: Qualitative comparisons of unconditional 3D human generation between PrimDiffu- sion and baseline methods. RGB renderings and corresponding depths are placed side-by-side.
Table 1: Quantitative results on unconditional 3D human generation. The top three techniques are highlighted red, orange, and yellow, respectively. In this table, indicates finetuned model that is pre-trained on the DeepFashion [26] dataset.
Methods
FIDCLIP
FID 4
KID x 10 +
PCK
Depthx 102
FPS f
StyleSDF [37]
18.55
51.27
4.08 + 0.13
49.37 + 22.18
2.53
EG3D [5]
19.54
24.32
1.96 + 0.10
16.59 + 21.03
22.97
EVA3D [18]
15.03
44.37
2.68 + 0.13
91.84
3,24 9.93
6.14
EVA3D ft [18]
14.58
40.40
2.99 + 0.14
91.30
2.60 7.95
6.14
Ours
12.11
17.95
1.63 + 0.09
97.62
1.42 = 1.78
88.24
Implementation Details. We use K W = 1024 primitives to represent each 3D human. The denoiser 94 implemented as a 2D U-Net [45] with intermediate attention layers. We first train the generalizable encoder using all available images. Then, the denoiser O4 is trained according to the primitives produced by the frozen encoder. Please check the supplementary material for more details.
Evaluation Metrics. We adopt Fréchet Inception Distance (FID) [16] and Kernel Inception Distance (KID) [4] to evaluate the quality of rendered images. Note that, we use different backbones to evaluate FID in different latent spaces in order to get thorough evaluation, i.e., FIDCI.IP employs the CLIP image encoder [40] while FID employs the Inception-Vs. model [54]. To evaluate the 3D geometry, off-the-shelf tool [42] estimate depth maps from renderings and compute 1.2 distance against rendered depths. We further adopt a human-centric metric, Percentage of Correct Keypoints (PCK) [2], to evaluate the pose controllability of 3D human generative models.
Comparison Methods. As the first diffusion model for 3D human generation, we compare with three GAN-based methods. EVA3D [18] learns to generate human from 2D image collections. EG3D [5] and StyleSDF [37] are approaches for 3D-aware generation from 2D images. succeeding in generating human faces and objects. All of these methods enable view control by implicitly conditioning on the camera pose, which leads to extra forward passes upon viewpoint changes.
4.1 Qualitative Results
We show the RGB renderings and corresponding depth maps generated by PrimDiffusion and baselines in Fig. 5. StyleSDF and EG3D achieve coarse results terms of appearance and geometry. However, due to the lack of human prior, most of the capacity is wasted on modeling the empty space that is not occupied by the human body, which leads to blurry facial details. Furthermore, the inefficient 3D representations make them hard to scale up to high resolution without super-resolution modules, which also limits their geometry to low resolution (642 1282). EVA3D succeeds in high-resolution rendering with much better geometry by incorporating human prior. However, it fails to generate enough facial details. We attribute this to its part-wise human representation. For example, EVA3D models the human head as single volume, assigning a difficult learning problem of distinguishing facial details within one uniform representation. contrast, PrimDiffusion divides humans into many primitives tailed with powerful diffusion modeling, which is not only capable of
7

---

Voxel
Latent Codes
N/A
N/A
N/A
N/A
Primitive (Ours)
Figure 6: Qualitative ablation study of different human representations for 3D diffusion. Alter- natively, we replace primitives with voxel grids human representation validate the effectiveness. RGB renderings, depths, and representation occupancy are placed sequentially for each sample.
Table Quantitative ablation study different human representations for 3D diffusion. Note that, the "Resolution" denotes the 3D spatial resolution of the representation.
Methods
Resolution
FIDeLiP
FID +
KIDx 10 +
Depth 10
Voxel
64 64 64
19.53
81.78
7.72 + 0.17
8.22
3.78
Latents [38]
12.13
37.70
3.44 +0.11
1.76 1.91
Ours
256 x 256
12.11
17.95
1.63 + 0.09
1.42 1.78
modeling enough details (e.g., faces and textures) but also supporting 512 x 512 resolution for both RGB and depth without any super-resolution decoder.
4.2 Quantitative Results
The results of numerical comparisons are presented in Tab. 1. As general 3D generation approaches, StyleSDF and EG3D get fair scores in terms of generation quality, but the geometries are much worse than others. EVA3D generates good geometries, indicated by its comparable depth error with us. However, the image quality is still far behind us. Note that, observe the vanilla FID might be inaccurate, which leads to discrepancies between visualizations and numerical results. For example, the images rendered by EG3D are of poor quality but the FID is much lower than other baselines. Therefore, we report FIDCLIP metric which we found more consistent with qualitative results. Nevertheless, the proposed method still outperforms baselines by a large margin.
Moreover, we also compare our model with finetuned EVA3D that is first pre-trained on DeepFash- ion [26] dataset establish 3D representations of humans and then finetuned our RenderPeople dataset. The motivation for this experiment is to fully leverage the ability to learn 3D representations on both multiview images and collections of images methods like EVA3D instead of training from scratch with multiview images only. However, as shown in Tab. 1, the finetuned EVA3D still has performance gap compared with our method.
4.3 Ablation Studies
Different Representations for 3D Diffusion. Apparently, the volumetric primitive is not the only representation for 3D human diffusion models. We also explore the effectiveness of other representations, like voxel grids [53] and latent codes [38]. The "voxel" bascline differentiates itself

---

Table 3: Ablation study of design choices for
volumetric primitives fitting, The metrics PSNR,
SSIM [58], and LPIPS [63] are averaged across all training identities, views, and poses.
Methods
PSNR f
SSIM f
LPIPS +
w/
W/O
w/o varying òs
18.35
0.803
0.182
varying Os w/o R, attention
Ours
w/o T
31.70
0.982
0.058
effects of differ- ent design choices for volumetric primitives fitting. "GT" denotes the ground truth image.
Figure 7: Visualizations
w/o attention
28.39
0.962
0.081
Ours
32.15
0.984
0.048
from our method in two ways: 1) it utilizes voxel grids with a resolution of 64 x 64 x 64 for fitting 3D representation from multi-view images, and 2) leverages 3D convolution layers instead of 2D for instantiation of (+(-). Note that, as 3D convolution layers occupy much more memories than 2D, the feasible resolution of voxel grids rather limited. Moreover, the "latents" baseline inherits the idea of anchoring latent codes [38] to the vertices of the SMPL. model for 3D human representation learning. However, the vanilla NeuralBody [38] takes 14 hours per subject to converge which makes it unscalable to large-scale datasets like RenderPeople which contains lots of identities. Thus, we implement the "latents" baseline in the following way. We replace the payload of volumetric primitives with latent codes and leverage sparse convolution layers the decoder to output the color and density of radiance fields. This implementation of a NeuralBody version of our method keeps the ability to learn cross tross-identity 3D representation through our generalizable primitive learning framework while getting rid of per-subject fitting.
Quantitative results are presented in Tab. 2. The voxel-based representation fails to achieve rea- sonable result both in rendering quality and geometry. The latent codes baseline achieves plausible generated results but worse quality compared with ours. We further visualize the qualitative com- parisons Fig. The voxel-based approach wastes many parameters to densely model the scene, leading to the low spatial resolution of the representation. This is the root cause of its blurry render- ings, which validates the importance of our primitive-basse representation for 3D human diffusion model.
Ablation on Design Choices for Primitives Fitting. The quality primitives fitting significantly affects the generation quality, it provides 3D ground truth for the diffusion model. Therefore, we ablate the effectiveness of different designs for generalizable primitives fitting, presented Fig and Tab. 3. Note that, as the baseline that replaces the spatially- vvaried delta scale factor ós with a uniformly distributed scale factor, i.e., all primitives share the same delta scale factor. It fails to reconstruct reasonable 3D humans. The "w/o R, 7 denotes the method that uses predefined rotation and translation calculated from template human mesh instead of an -dentity-specific one, which leads to slight drop in quality. And "w/o attention" " denotes the method trained without our proposed cross- modal attention module, which important for detailed textures and motion-dependent effects.
reconstruction metrics for evaluation. We denote "w/o varying ds"
4.4 Further Analysis
Real-time Inference. Thanks to our efficient representation and decoder-free rendering procedure, PrimDiffusion can render 3D) consistent humans with varying viewpoints and poses in a resolution of 512 x 512 at 88.24 FPS once the denoising process is done. As shown in Tab. 1, due to the use of the decoder which decodes the 3D representation to images, all baselines fail to high-resolution real-time rendering at comparable quality with us. Furthermore, these methods implicitly model 3D-aware conditions, i.e., viewpoints and poses, which causes extra forward pass calls upon conditions changes.
Generalizability to Novel Poses. It is worth mentioning that a decoder (e.g., MLP for NeRF decoding or CNN for super-resolution) may lead to overfitting to the pose distribution of the training dataset, which prohibits pose generalization to out-of-disttibuttin. Another merit of our decoder- free rendering is that we can generalize to novel poses without test-time optimization, proceproc,,,ing and rigging (Fig. 1 and Fig. 4). We refer readers to the supplementary material for video results.

---

Conesporderce
Transfer
Add Noise
Mask Guidars
Texture Source
Targe
a) Texture Transfer
3D Impainting
Figure 8: Application of PrimDiffusion. a) Thanks the dense correspondence offered by volumet- ric primitives, we can transfer the texture a 3D consistent way. b) By masking the corresponding primitives with Gaussian noise and performing the denoising process with mask guidance, method can achieve the conditional generative task, i.e., 3D inpainting, without retraining the model.
4.5 Applications
Based the dense correspondence offered by volumetric primitíves, we can transfer the texture from one human body to another, as shown Fig. 8 a). Since the entire process operated in 3D space, we can render the human with transferred texture with free viewpoints.
Texture Transfer.
3D Inpainting. Thanks to the flexibility of primitive- erimitive-based representation and diffusion model, PrimDiffusion can inpaint the masked region on the human body with new textures as shown in Fig. 8 b). By perturbing the masked region with noise, we take the volumetric primitives as the mask guidance the denoiser (+).). Note that, contrast to GAN-basee methods, this conditional generation is done without retraining the model.
Conclusion
In this paper, we propose the first diffusion model for 31) human generation. To offer compact and expressive parameter space with human prior for diffusion models, we propose to directly operate of volumetric primitives, which models the human body
the diffusion and denoising process as a number of small volumes with radiance and kinematic information. This flexible yet efficient representation enables high-performance rendering for novel views and poses, with resolution of 512 x 512 in real-time. Furthermore, we also propose a cross-modal attention module tailored with an encoder-based pipeline for fitting volumetric primitives from multi-view images without per-subject optimization. We believe our method will pave the way for further exploration in 3D human generation.
Limitation. While method shows promising results for unconditional 3D human generation, several limitations remain. First, though we can model off-body topology like loose garments and long hair, how to animate them remains challenge. Second, renderings contain artifacts for complex textures. Leveraging 2D pre-trained generative models would be a possible solution. Last, due the high degree of freedom of primitives, our method two-staged pipeline prevent training instability. Further explorations on single-stage 3D diffusion models would be fruitful. In addition, compared with GAN-based approaches, our method requires sufficient number of posed views to get good 3D representations for training diffusion models. In this context, it would be interesting explore few-shot inverse rendering techniques reduce the reliance on multiview images for the first stage of our method.
Broader Impacts. The generated 3D humans PrimDiffusion might be misused to create mislead- ing content or fake media. It can also edit personal imagery, leading to privacy concerns.
Acknowledgments and Disclosure of Funding
This work is supported by the National Research Foundation. Singapore under its AI Singapore Pro- gramme (AISG Award No: AISG2-PhD-20211 2000-221-----9,, NTU NAP, MOE AcRF Tier 2 (T2EP20221- 0012), and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well as cash and in-kind contribution from the industry partner(s).
10

---

References
[1] Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero. RenderDiffusion: Image diffusion for 3D reconstruction, inpainting and generation. arXiv, 2022. 2,
[2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference Computer Vision and Pattern Recognition, pages 3686-3693, 2014.
[3] Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. Advances in Neural Information
Processing Systems, 35: 15:19900-19116 2022. 3
[4] Mikotaj Binkkoski,, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying
MMD GANs. In International Conference on Learning Representations. 2018. 7
[5] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022. 1, 2,
3.
[6] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hamrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rich 2015. [7] Xu Chen, Tianjian Jiang. Jie Song, Jinlong Yang. Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avatars. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2022. 3
3D) model repository. arXiv preprint arXiv:
[8] Zhaoxi Chen and Ziwei Liu. Relighting4D:: Neural relightable human from videos. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, pages 606-623. Springer, 2022.
[9] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Scenedreamer: Unbounded 3D scene generation from 2D image collections. In arXiv, 2023. 2
[10] MMHuman3D Contributors. Openmmlab 3D human parametric model toolbox and benchmark.
https://gt thub. com/ open- - mm ab/ mmhuman3d, 2021. 5
[11] Yu Deng, Jiaolong Yang. Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds
for 3D-aware image generation. In IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2022. 2
[12] Anna Frühstück, Krishna Kumar Singh, Eli Shechtman, Niloy J. Mitra, Peter Wonka, and Jingwan Lu. Insetgan for full-body image generation. In Proceedings of the IEEE/CVF International Conference
Computer Vision and Pattern Recognition (CVPR), pages 7723-
7732, June 2022. 3
[13] Janglin Fu, Shikai Li, Yuming Jiang, Kwan- Lin, Chen Qian, Chen-Change Loy, Wayne Wu, and Ziwei Liu. Stylegan-human data-centric odyssey of human generation. arXiv preprint, arXiv:2204.1 11823,, 2022. 3
[14] Ian Goodfellow, Jean Poudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farlee Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinherger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[15] Jiatao Gu, Lingje Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based based 3D) aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022. 2
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
7
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
3.
[18] Fangzhou Hong, Zhaoxi Chen, Yushi LAN, Liang Pan, and Ziwei Liu. EVA3D: Compositional 3D human generation from 2D image collections. In International Conference on Learning
Representations, 2023. 2, 3.7
[19] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: zero-shot text-driven generation and animation of 3D avatars. ACM Transactions
11

---

Graphics (TOG), -1(44)1-19, 2022. 3
[20] Suyi Jiang, Haoran Jiang. Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12543-12555 June
2023. 3
[21] Yuming Jiang, Shuai Yang, Haonan Qiu. Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics
(TOG), 11(4):1-11, 2022. 3
[22] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017. 2
[23] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019. 2
[24] Kathleen M Lewis, Srivatsan Varadharajan, and Ira Sesttmaacher-Shizzemmaa Tryongan: Body-aware try-on via layered interpolation. ACM Transactions on Graphics (Proceedings of
ACM SIGGRAPH 2021), 40(4), 2021. 3
[25] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3D mesh modeling. In International Conference on Learning Representations. 2023.
[26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June June 2016. 7,
[27] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. 3
July 2019. 4
[28] Stephen Lombardi. Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans.
Graph., 40(4), jul 2021. 5,5
[29] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia),
34(6):248:112444 14, Oct. 2015. 5,5
[30] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3D point cloud generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR),
June 2021. 3
[31] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision. pages 5442-5451, Oct. 2019. 6
[32] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis.
Communications of the ACM, 65(1):99-106, 2021. 2,
[33] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffrf: gonddrring-guider 3D radiance field diffusion. In arxiv, 2022. 1. 2, 3 [34] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning, pages 16784-16804, 2022. 3
[35] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 1,2
[36] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-awar neural articulated representations. arXiv:2204.0S0: 30039, 2022. 3
[37] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High-Resolution 3DD-Consistnt Image and Geometry Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13503-13513, June 2022. 1,
[38] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9054-9063, 2021. 8,9
12

---

[39] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using
2D diffusion. arXiv preprint arXiv:2209, 249,:2209,448, 2022. 3
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR. 2021. 7
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint axXiv2204.00/122 1(2):3 2022.
[42] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179-
12188, 2021. 7
[43] Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu, Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao, Fabian Prada, Jason Saragih, et al. Drivable volumetric avatars using texel-aligned features. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1 1-9, 2022.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, pages 10684-10695, 2022. 3
[45] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation.
Medical Image Computing and Computer-Assistee Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015. 7
[46] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 3 [47] Shunsuke Saito. , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. arXiv preprint 1905 0557 2019. 3
[48] Kripasindhu Sarkar, Lingje Liu, Vladislav Golyanik, and Christian Theobalt. Humangan: A generative model of human images. In 2021 International Conference on 3D Vision (3DV), pages 258-267. IEEE, 2021. 3
[49] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2
[50] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3D-aware image synthesis with sparse voxel grids. In Advances in Neural Information
Processing Systems (NeurIPS). 2022. 2, 3
[51] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2266 Lille, France, 07-09 Jul 2015.
PMLR. 1.3
[52] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pages 1 11895-11907, 2019.
[53] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Saper-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459-5469, 2022. 8
[54] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pages Renderpeople. 2018. 6
2016. 7
[55] https://rendes: //rrenderpooplee com com/3d-wo people/.
[56] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning animatable clothed human models from few depth images. In Advances in Neural Information Processing Systems, 2021.
[57] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3D digital avatars using diffusion. In Proceedings of the IEEE/CVF Conference on Computer
13

---

Vision and Pattern Recognition, pages
2023. 1,3,5
[58] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004.
[59]
Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. GRAM-HD: 3D-Consistent Im- age Generation at High Resolution with Generative Radiance Manifolds. Technical Report
arXiv:2206.0725:
arXiv, June 2022. 2
[60] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeie, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3D shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[61]
Jianfeng Zhang. Zihang Jiang. Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhong- cong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: 3D generative model for animatable
human avatars. In Arxiv, 2022. 2, 3
[62] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv
preprint arXív:2208. 15001, 2022. 3
[63] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features
perceptual metric. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pages 586-595, 2018. 9
[64] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based based human reconstruction. JEEE transactions pattern
analysis and machine intelligence, 44(6):3170 100-3177--8444 2021. 3
[65] Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxe diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5826-588 5835, October 2021. 3
14


Reviews:
[{"review_id": "py4TB7KlsF", "reviewer": "Reviewer_oDDi", "summary": "The paper introduces a novel framework called PrimDiffusion, which utilizes diffusion models to generate 3D human models. The framework leverages a Mixture of Volumetric Primitives (MVP) to represent 3D humans, enabling efficient rendering and incorporating human priors for retargeting. This representation offers a compact and expressive 2D parameter space, making it well-suited for the diffusion operation. Experimental results demonstrate that the proposed model outperforms previous approaches in terms of both quality and efficiency. Furthermore, the proposed method exhibits flexibility in training-free conditional generation tasks, including texture transfer and 3D inpainting.\n", "questions": "The paper conducted a comparison between volumetric primitives and voxel representations in diffusion-based 3D human generation. However, I am also interested in understanding how volumetric primitives compare to triplane representations in diffusion. It would greatly enhance the experimental analysis if this comparison were included.\n\nRegarding the ablation study, the paper compared predefined rotation and translation values derived from a template human mesh with identity-specific ones. However, the paper does not provide a detailed explanation of the identity-specific rotation and translation. It would be beneficial to include a thorough explanation of this approach to improve the understanding of the study.\n", "limitations": "The proposed method is innovative as it combines volumetric primitives with diffusion model. However, there is room for improvement in terms of the writing. There is still a noticeable disparity between the ground truth and the fitting results. It seems that there is significant room for improvement in stage 1 of the method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper clearly proposes to represent 3D humans using volumetric primitives, effectively transforming the 3D representation into a 2D parameter space that is well-suited for 2D diffusion. This method outperforms previous approaches for 3D human face/body generation, which mostly rely on triplane representation. Further, volumetric primitives technique also provides easier animation and conditional generation capabilities.\n\nThrough experiments, the proposed method demonstrates superior qualitative and quantitative performance compared to previous works. These experiments highlight the advantages of using volumetric primitives representation for 3D human generation, specifically, the paper compared it with voxel-based representations.\n", "weaknesses": "In lines 139-140, the proposed method only considers color, density, and delta scale factor as optimized parameters, while delta translation and delta rotation are not included. It would be beneficial to understand the reasons behind this omission, especially considering that the original MVP paper optimized these parameters.\n\nFigure 2 illustrates the diffusion process of rotation (R), translation (T), and scale (s) during training, which appears to contradict the claim that the learnable parameters are solely color, density, and delta scale factor. And it seems that there is a discrepancy, as the expected parameters should be delta rotation, delta translation, and delta scale. Clarification on this point would be appreciated.\n\nThe motion encoder in the proposed method solely considers joint rotations, which may not be sufficient for accurate fitting of 3D humans. Shape parameters also play a crucial role in this process. It would be helpful to hear an explanation for the decision to exclude shape parameters from the motion encoder.\n\nThere are typos in Function (7), which are inconsistent with the function presented in the DDPM paper [16]. It is important to address these inconsistencies and ensure the correctness and coherence of the equations.\n\nThe loss function (8) differs from the one described in the DDPM paper [16]. The original function aimed to estimate noise from V_t instead of directly estimating the ground truth. It would be beneficial to provide an explanation for this modification and discuss its implications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. We summarize and answer your questions as follows:\n\n**Q1: The reasons behind the omission of delta translation and delta rotation.**\n\nThanks for your comments. There are two reasons for predicting delta scale factor only: 1) better generalizability to novel poses, 2) efficient decoder-free rendering. We empirically found that it is only beneficial to include delta translation and rotation for reconstruction quality. However, it does harm to the generalizability of novel poses as the model now implicitly depends on training pose distribution. When we drive the primitives with out-of-distribution poses once the denoising process is done, it requires queries of delta translation and rotation decoder. However, such a decoder-dependent pipeline prevents the model from generalizing to novel poses and slows the rendering speed. \n\n**Q2: Clarification on the discrepancy between Figure 2 and text on whether or not to optimize rotation(R), translation(T).**\n\nThanks for your careful reading and outstanding review! The $R_t$ and $T_t$ should always equal $R_0$ and $T_0$. We will clarify this to be consistent with the caption of Figure 2 in the revised version.\n\n**Q3: The reason for excluding shape parameter from motion encoder.**\n\nThank you. The reason for excluding the shape parameter from the motion encoder is that the shape parameter has been explicitly considered through the process of linear blending skinning, which takes as input the pose $\\theta$, shape $\\beta$, and the neutral SMPL model template.\n\n**Q4: Clarifications of Equation (7) and (8).**\n\nThank you for the comment. We empirically found that predicting $\\mathcal{V}_0$ offers better convergence of the denoiser compared with predicting the noise $\\epsilon$ in our case, which is also observed in other 3D diffusion models [1]. However, the formulation presented in the original DDPM paper is predicting the noise itself (Eq. 11 and Eq. 14 in their paper). Thus, we derive our formulation (Eq. 7 and Eq. 8) from the original one while keeping them mathematically equivalent.\n\n**Q5: Volumetric diffusion v.s. triplane diffusion?**\n\nThis is a great point. In fact, our initial attempt started from triplane diffusion which is similar to Rodin [2]. However, we found that it is not straightforward to fit triplanes and train the diffusion model on top of it for 3D human bodies. As shown in **Figure 4 of the attached figure-only file**, triplane diffusion can easily generate floating artifacts around the human body. We attribute the failure of triplane diffusion on 3D humans to the inefficiency of triplane in representing the human body which is a highly articulated object. The human body only occupies a small portion of the space modeled by the triplane, thus the network wastes most of the parameters in modeling empty space. It poses challenges for both the fitting of triplanes and the convergence of diffusion models.\n\n**Q6: Detailed explanation of the difference between predefined rotation and translation and an identity-specific one.**\n\nSorry for the confusion. Conceptually, the translation of each volumetric primitive is defined according to the positions of mesh vertices. And the rotation is defined in the TBN space over the human mesh. For the predefined rotation and translation, we compute them over a human mesh template shared across identities. For the identity-specific one, we compute the rotation and translation on the human mesh after the linear blending skinning with personalized shape and pose parameters. The intuition for this ablation is to evaluate whether or not using the delta scale factor only is enough to compensate for the inaccuracy of rotation and translation. \n\n[1] T. Anciukevicius et al. RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation. In CVPR, 2023.\n\n[2] T. Wang et al. Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion. In CVPR, 2023."}, {"review_id": "3i2lKy4g1x", "reviewer": "Reviewer_wB9H", "summary": "The paper presents, PrimDiffusion, an approach for generative modeling of 3D humans. PrimDiffusion is a diffusion-based model that operates on a set of volumetric primitives representing the human body. Each volumetric primitive consists of rotation, translation, scale, radiance, and color information. PrimDiffusion is a multi-stage approach. Given a training dataset with multiview images, the first stage fits volumetric primitives on training images using reconstruction losses. The second stage then uses the fitted volumetric primitives to train a diffusion-based generative model where the results from the first stage are used as the ground truth for the diffusion process.  The use of volumetric primitives allows decoder-free rendering of the images which allows real-time rendering once the diffusion process is done. It also provides explicit control over the body pose and viewpoint. The paper also shows its application for texture transfer and unconditional infilling. Experiments are performed on the synthetic RenderPeople dataset where PrimDiffusion is shown to outperform other baselines.", "questions": "### Missing Experiments \n- The paper learns 3D primitives using reconstruction losses and multiview images. Since the ground-truth meshes are available for the RenderPeople dataset, an interesting comparison would be to use them to obtain volumetric primitives. It will help us understand how much information is lost during the learning of volumetric primitives using multi-view images. \n- I would have liked to see a baseline that is similar to NeuralBody i.e., replace volumetric primitives with learnable latent codes. \n- Please explain how the baseline methods are trained.", "limitations": "- As mentioned above, reliance on 3D ground-truths is a significant limitation of the proposed method as compared to existing methods. Hence, it should be discussed in the `Limitations`.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses the challenging problem of generative modeling of 3D humans. \n- The use of volumetric primitives to represent complex human bodies, clothing, and hairs makes sense and is intuitive. \n- The main challenge for learning 3D diffusion models is the availability of 3D ground truths. The paper overcomes this challenge by first fitting volumetric primitives on multiview images. \n- The paper inherits the advantages of volumetric primitives including decoder-free rendering and explicit control over viewpoint and body pose. \n- The paper demonstrates two interesting applications of the proposed method including texture transfer and infilling. \n- The paper is well-written and easy to read.", "weaknesses": "### Novelty \n- The main novelty of the proposed method comes from the combination of existing bits and pieces. The volumetric primitives have already been used in [25] to model human faces. The paper extends it to the full body by incorporating the SMPL body model. The idea of anchoring neural primitives to the SMPL body model is similar to NeuralBody [a] `(missing citation)`, and generative modeling using diffusion models is standard practice these days. While there is no technical novel contribution of the paper, I acknowledge that the proposed combination is not straightforward and this is not the most crucial limitation of the paper. \n\n### Comparison with existing methods \n- This is the most crucial limitation of the paper. The paper compares with EVA3D, StyleSDF, and EG3D.  The main advantage of these methods is that they can learn 3D representation from a large collection of 2D images. However, it seems that this paper has retrained them from scratch using only the multi-view images from the RenderPeople dataset `(please correct if my understanding is wrong)`. Hence, their results are significantly worse than what is shown in the original paper, in particular the results of EVA3D. I find this comparison unfair for these methods since it suppresses their advantages and conceals the limitations of the proposed method. For a fair comparison, these methods should be fine-tuned on the RenderPeople dataset rather than training from scratch. It is the limitation of the proposed method that it requires 3D ground truths for training and hence can only be trained on multiview data. Other methods, on the other hand, can use a large collection of 2D images hence they should be used by default for these methods. \n\n### Blurry results\n- The qualitative results are quite blurry. Would be great if the authors can comment on this during the rebuttal. \n\n### Missing references\n[a] Implicit Neural Representations with Structured Latent Codes, CVPR'21 \n\n### Conclusion\nOverall, it is a good paper with a reasonable approach. However, the experiments section of the paper is a bit weak, in particular, the comparison with existing methods is unfair. Hence, my current rating for the paper is `Weak Reject`. I would be happy to upgrade my rating if the authors can address my concerns during the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. We summarize and answer your questions as follows:\n\n**Q1: Novelty of the proposed method.**\n\nThanks for your comments. We highlight our key contributions in the above post named \"Response to the common issue of the novelty and motivation\". \n\nMoreover, we argue that our human representation is fundamentally different from NeuralBody. **First**, we anchor explicit primitives with color and kinematic information to the human model while NeuralBody leverages implicit latent codes. **Second**, NeuralBody relies on decoders (consisting of sparse convolution layers and MLPs) to 1) diffuse latent features off the surface of SMPL topology and 2) render latent features into color and density. It provides no generalizability to novel poses as the decoder overfits to the training observations. Instead, the volumetric primitive in our case is decoder-free, which enables efficient rendering and generalizing to novel poses. Moreover, our model is capable of handling off-body topology like loose garments while the latent codes of NeuralBody cannot.\n\n\n**Q2: Unfair comparisons for baseline methods.**\n\nThanks for your great suggestion. We finetuned EVA3D which is pretrained on DeepFashion dataset according to your suggestion. The numerical results are summarized below. We found that it indeed provides a slight improvement compared with the one trained from scratch. However, it still has a significant gap compared with our method. Moreover, we present qualitative results in **Figure 1 of the attached figure-only file**. Will add this result in Table 1 and Figure 5 of our revised version.\n\nBesides, we argue that stage 1 of our method can be treated as an inverse rendering step that recovers 3D representation from 2D observations. It can further benefit from techniques like sparse views reconstruction which can reduce the reliance on multiview images. The key insight of our paper is proposing a way to inherit both the capacity of the diffusion model and the efficiency of volumetric primitives.\n\n|Methods | FID$_{\\mathrm{CLIP}} \\downarrow$ | FID $\\downarrow$ | KID$\\times 10^2 \\downarrow$ | PCK$\\uparrow$ | Depth$\\times 10^2 \\downarrow$| \n|:------:|:------:|:------:|:------:|:------:|:------:|\n|EVA3D|15.03|44.37|2.68 $\\pm$ 0.13|91.84|3.24 $\\pm$ 9.93|\n|EVA3D (Finetuned)|14.58|40.40|2.99 $\\pm$ 0.14|91.30|2.60 $\\pm$ 7.95|\n|Ours|12.11|17.95|1.63 $\\pm$ 0.09|97.62|1.42 $\\pm$ 1.78|\n\n\n**Q3: Blurry results.**\n\nThanks for your comments. We observe that one of the reasons for some blurry results is that we remove the view condition from the model to force it to learn a truly 3D representation. As shown in Figure 1 and Section B.3 of the supplementary material, the UV-aligned view conditions help volumetric primitives to capture more details. However, such a way of conditioning makes the model relies on queries on the view decoder at inference time. It breaks the principle of decoder-free rendering, leading to slower rendering speed for novel poses and views. \n\n\n**Q4: Using GT mesh from RenderPeople to obtain volumetric primitives.**\n\nThis is a good point! In the context of single identity reconstruction, the ground truth mesh undoubtedly offers better geometry than the estimated SMPL model, resulting in a more accurate initialization of rotation and translation for each primitive. \nHowever, as a generative model, our objective is not solely focused on precisely reconstructing a single identity. Instead, we aim to establish a unified parameter space that can be shared across identities for effective representation learning by diffusion model. The ground truth meshes obtained from the RenderPeople dataset contain varying numbers of vertices, preventing the establishment of a unified UV space and primitives parameter space suitable for training.\n\n**Q5: A baseline similar to Neuralbody, replacing volumetric primitives with latent codes.**\n\nThanks for your suggestion. Note that, NeuralBody is an approach for single identity reconstruction while our method aims at human generation. It takes 14 hours per subject to converge which makes it unscalable to large-scale datasets like RenderPeople. Thus, we replace the payload of volumetric primitives to latent codes according to your suggestion and leverage sparse convolution layers as the decoder to output color and density. All kinematic parameters of primitives are frozen. This implementation of a NeuralBody version of our method keeps the ability to learn cross-identity 3D representation, getting rid of per-subject fitting. We present quantitative results below and show qualitative results in the **Figure 3 of the attached figure-only file**. Our method outperforms the NeuralBody (latent codes) variant. We will add this baseline as additional ablation studies to Table 2 in our revised version.\n\n|Methods | FID $\\downarrow$ | KID$\\times 10^2 \\downarrow$ | Depth$\\times 10^2 \\downarrow$| \n|:------:|:------:|:------:|:------:|\n|NeuralBody| 37.70 | 3.44 $\\pm$ 0.11 | 1.76 $\\pm$ 1.91|\n|Ours|17.95|1.63 $\\pm$ 0.09|1.42 $\\pm$ 1.78|"}, {"review_id": "GjNxdqIwYr", "reviewer": "Reviewer_PWaT", "summary": "The paper presents a diffusion-based approach, PrimDiffusion, for 3D human generation. The method leverages the representation of MVP (mixture of volumetric primitives) and a design of volumetric primitives diffusion to generate high-quality 3D human models using multi-view images as training data. The authors propose a two-stage training process: the first stage involves learning generalizable primitives from multi-view images, and the second stage involves a diffusion process to generate/sample these primitives. The paper also presents an encoder design tailored with cross-modal attention for learning volumetric primitives from images across identities. This encoder, which consists of an appearance branch and a motion branch, is trained end-to-end across identities, eliminating the need for per-subject optimization. The cross-modal attention module within the encoder effectively captures dependencies between appearance and motion, contributing to the generation of high-quality 3D humans. The paper includes extensive experimental results demonstrating the effectiveness of the proposed method in comparison with existing techniques.", "questions": "- In Figure 3, top-left, the input is denoted as \"Body Model $M(\\beta, \\theta)$\", but the figure appears to depict a detailed model rather than a SMPL model. Could the authors clarify this discrepancy?\n\n- When projecting image features and pose features to the SMPL UV space, how are the **\"empty\"** UV regions handled? What are the initial values for the primitive positions corresponding to these empty regions?\n\n- What is the memory consumption of the proposed method during training and testing?\n\n- The training data includes 20 different human poses. How were these poses sampled? What criteria were used to ensure a comprehensive coverage of possible poses?", "limitations": "The proposed method requires the use of multi-view images as training data. This requirement, when compared to previous works that do not have such a constraint, should be discussed in detail as a limitation of the current approach.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) A novel way of generating 3D human models using volumetric primitives diffusion.\n\n(+) The paper is well-structured and clearly written. The authors provide a clear explanation of their method. The supplementary document provides additional technical details, and the supplementary video also helps to showcase the quality that the proposed method achieves.\n\n(+) The proposed method achieves superior results compared to existing techniques on the RenderPeople multiview dataset.", "weaknesses": "(-) The proposed method is a 2-stage approach, compared to previous GAN-based methods such as EVA3D, it requires **multi-view images** as training data to learn generalizable primitives in the first stage. This is a significant difference as collecting large-scale real-world multi-view human data can be challenging, while 2D image collections are relatively easier to gather. This distinction is not clearly articulated in the paper. Considering a single-stage solution might address this issue.\n\n(-) The experiments lack a comparison with a recent baseline: Jiang, Suyi, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. \"Humangen: Generating human radiance fields with explicit priors.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12543-12554. 2023.\n\n(-) Please refer to the questions section for more points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. We summarize and answer your questions as follows:\n\n**Q1: The proposed method is a 2-stage approach that requires multi-view images.**\n\nThanks for your feedback. Indeed, this is a limitation of our method which we will discuss in our revised version. Besides, as discussed in L311 of our paper, the current two-staged approach is a design choice for training stability given the high degrees of freedom of primitives. It is fruitful to explore a single-stage approach like SSDNeRF [1] for 3D humans in the future. Yet, efficient 3D representation for highly articulated objects like humans is still an open problem for single-staged methods. We believe our method paves the way for solving this challenging problem.\n\n**Q2: Lack a comparison with HumanGen.**\n\nThanks for your constructive comments. \nHumanGen [3] is a great work for 3D human generation that leverages explicit image priors from strong pretrained 2D human image generators and geometry priors from PIFu [4]. Unfortunately, it is published in CVPR 2023 which was held after the submission date of NeurIPS. To the best of our knowledge, the source code has not been released so far. Therefore, we tried to reproduce according to our understanding. However, we failed to achieve reasonable results in our setting. Undoubtedly, we will cite and discuss HumanGen as a concurrent work in Section 2 of our revised version.\n\n**Q3: Clarification of Figure 3 top-left on why it is a more detailed model than SMPL.**\n\nSorry for the confusion. The top-left of Figure 3 is only a principled illustration of parametric human models. It could be any parametric human model, like SMPL or SMPL-X. In our implementation, we choose SMPL. Will replace it with the reposed SMPL mesh for coherence in the revised version.\n\n\n**Q4: Memory consumption during training and testing.**\n\nThanks for the question. Please kindly refer to the above post named \"Response to the common issue of the computational cost\".\n\n**Q5: How the 20 poses are sampled for training data?**\n\nThanks for the question. As introduced in the supplementary material (Section C.2), we randomly sample human poses from AMASS dataset [2], which covers sufficient diversity of real-world human poses. Moreover, to bridge the gap between SMPL models from AMASS dataset and arbitrary human meshes from the RenderPeople data, we use the motion retargeting technique which allows for the transfer of motion data between heterogeneous sources.\n\n**Q6: How are the empty UV regions handled?**\n\nLet us elaborate on this process in detail. The pose features are padded to align the UV space so there is no projection process. We choose an \"inverse\" way to project the image features. Specifically, we project the vertices of the SMPL model to the corresponding conditioned camera view to get pixel coordinates and perform grid sampling on the conditioned 2D image to get the corresponding projected image as input. Since the UV mapping of the SMPL model is predefined, there is no need to handle empty UV space in this process. As you correctly pointed out, we need to handle empty UV regions to initialize the primitives' positions. As presented in L131 of our paper, The 3D positions of primitives are initialized by uniformly sampling UV space and mapping each primitive to the closest surface point on SMPL that corresponds to the UV coordinates of the grid points. Specifically, we handle empty UV regions by inpainting the vertices index map with the nearest neighbors in the UV space. Intuitively, the primitive initialized to those \"empty\" UV regions will be placed on the nearest SMPL vertex.\n\n[1] H. Chen et al. Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction. In ICCV, 2023.\n\n[2] N. Mahmood et al. AMASS: Archive of motion capture as surface shapes. In ICCV, 2019.\n\n[3] S. Jiang et al. HumanGen: Generating Human Radiance Fields with Explicit Priors. In CVPR, 2023.\n\n[4] S. Saito et al. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In ICCV, 2019."}, {"review_id": "xbCfRXqG0h", "reviewer": "Reviewer_aNCP", "summary": "The paper proposes a generative model for 3D humans. The work is based on volumetric rendering of geometric primitives and diffusion model. Given multi-view images of humans as training data, the method first learns a volumetric primitives representation, and then applies diffusion model framework to model the representation distribution. The authors evaluate their method and compare to SOTA approaches on a relatively small synthetic multi-view dataset. Results show favor towards proposed approach.", "questions": "I would like the authors to answer Weaknesses 2, 3, 4, and 5. The answers may not change my final decision as long as no major flaw is found later. The lack of details in the generated results prevent me from giving strong support of the submission. But the attempt to address 3D generative human models with diffusion model and volumetric primitives should be encouraged.", "limitations": "Yes, both are tackled in the submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an interesting, practical and high-impact problem: generating clothed 3D humans.\n\n2. The paper is overall well written, easy to follow.\n\n3. The use of cross attention to fuse pose info and texture info is appreciated and demonstrated good performance.\n\n4. I'm glad to see the attempt of using volumetric primitives and diffusion model to create generative models of 3D humans.", "weaknesses": "1. The bibliography needs to be better formatted. \na) The same conference is referred in different format, e.g. CVPR vs IEEE Conference on Computer...\nb) Capitalization of words, e.g. 3d -> 3D, Shapenet -> ShapeNet\nc) Others, e.g. [30] two commas, [31] where is it published?\n\n2. As a reference, I'd like to have a fidelity evaluation of \"ground truth\" volumetric primitives, better added into Table 1 in the updated version. This helps to understand the upper limit of the diffusion process in the proposed method and see where the bottle neck lies.\n\n3. It looks like all the generated results are blurry. Is it due to the memory limit? What is the GPU memory consumption for W=32 (assuming it is the setting of all the experiments)? Is the memory consumption a limitation for the proposed method compared to EVA3D? In other words, if one had a large size of multi-view human data, would it be likely that EVA3D performs much better than the proposed method given current limit of computational power?\n\n4. Most of the generated results presented in the submission (including the video) contain mono-color clothing. There are only two or three exceptions. Is it a bias of the dataset or is it a result of low resolution volumetric primitives? Could the authors provide a statistics on the ratio of mono-color clothing in the Renderpeople dataset?\n\n5. What is the difference of texture transfer and reposing? It looks to me that transfer the source texture to the target subject is the same as repose (and reshape the body of) the target subject to the source pose (and source body shape). If this is true, please remove this application as it is basically in 4.4 already.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. We summarize and answer your questions as follows:\n\n**Q1: The bibliography needs to be better formatted.**\n\nThank you for the great suggestion. We will revise all references in a unified format.\n\n**Q2: Fidelity evaluations of ground truth volumetric primitives (Table 1) as an upper bound for diffusion.**\n\nThanks for this constructive comment! We provide all corresponding metrics in Table 1 for ground truth volumetric primitives learned from the first stage as follows. As you correctly pointed out, this serves as the upper bound of our diffusion model. We will add these numerical results in Table 1 of our revised version.\n\n\n|Methods | FID$_{\\mathrm{CLIP}} \\downarrow$ | FID $\\downarrow$ | KID$\\times 10^2 \\downarrow$ | PCK $\\uparrow$ | Depth$\\times 10^2 \\downarrow$| \n|:------:|:------:|:------:|:------:|:------:|:------:|\n|GT|10.34|12.66|1.67 $\\pm$ 0.10|97.70|1.20 $\\pm$ 1.65|\n|Ours|12.11|17.95|1.63 $\\pm$ 0.09|97.62|1.42 $\\pm$ 1.78|\n\n\n**Q3: Are blurry results caused by GPU memory limit? Provide the GPU memory consumption for W=32.**\n\nThanks for your question. We first provide the GPU memory usage during training and inference of the diffusion model. Please kindly refer to the above post named \"Response to the common issue of the computational cost\". Ideally, we could have more detailed results by increasing the resolution of W. We observe that one of the reasons for some blurry results is that we remove the view condition from the model to force it to learn a truly 3D representation. As shown in Figure 1 of the supplementary material and Section B.3, the UV-aligned view conditions help volumetric primitives to capture more details. However, such a way of conditioning makes the model relies on queries on the view decoder at inference time. It breaks the principle of decoder-free rendering, leading to slower rendering speed for novel poses and views.\n\n**Q4: The statistics of mono-color clothing in the dataset.**\n\nThanks for your question. We provide the statistics of mono-color clothing in our dataset, which shows a strong bias towards mono-color clothes. Therefore, the results are caused by dataset bias. In the future, one may seek larger scale data with more diversity for training or use ControlNet [1] to augment the dataset.\n\n| Body Part| Mono-color   | Complex Texture |\n| :----: | :------: | :------: |\n| Upper (shirts, jacket etc.)| 698 | 98 | \n| Lower (pants, trousers etc.)| 754 | 42 |\n\n[1] L. Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. In arXiv, 2023.\n\n**Q5: The difference of texture transfer and reposing.**\n\nLet us elaborate on the difference between texture transfer and pose-driven synthesis (reposing). They are only equivalent if we want to transfer all textures (shoes, shirts, and pants) from the source to the target. In most cases, the goal of texture transfer is to transfer a part of the textures from the source to the target, e.g., changing a shirt for the person. As shown in Figure 8 of the paper (changing shirt), we leverage the dense correspondence provided by volumetric primitives to transfer the shirt texture from the source to the target. Reposing and reshaping the target person to the source one cannot offer the flexibility for controlling specific body parts or clothing."}, {"review_id": "Y4hv8NQrcw", "reviewer": "Reviewer_GMsy", "summary": "This paper introduces a method to generate various 3D human geometry and appearance. To this end, it introduces a new design of diffusion model that learns to generate the parameters of 3D primitives of humans: a generalizable encoder constructs motion and appearance features and their attentions are fed to predict scale, color, and density of each primitive. The evaluation is performed on the render people dataset, and the authors demonstrate a number of applications such as reposing and 3D inpainting.", "questions": "Please address the questions in the weakness section.", "limitations": "This paper sufficiently discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper effectively combines existing 3D diffusion model and existing 3D representation to perform generalizable 3D human generation. It is the first attempt to combine diffusion model and primitive representation for 3D generative humans. It can serve as good initial baseline for future research.\n+ The 3D inpainting application is interesting, which allows reconstructing a complete 3D human from a partial body.\n", "weaknesses": "1] The motivation of this paper is not clear. \nIn particular, it is not clear why the volume primitive representation is ideal (other than efficiency) for handling complex human articulations. Other than computational efficiency, why does this representation fundamentally enables better quality of 3D human generation than other representation (e.g., point clouds, sdf, occupancy, and so on)? \nSimply showing the ablation study (without thorough analysis) does not account for this fundamental motivation, and I would like to request the authors to explain above question with fundamental reasons. Without strong motivation, this work will fall into the work that just simply combines existing 3D diffusion with existing representation as one try of a baseline. \n\n2] Misleading claims and overclaims: \n2-1) In L45, existing methods (nerf or tri-plane) do not model humans as coarse volumes. They model the geometry in the continuous space, which can lead to the fine-results. In fact, the coarseness is quite proportional to the resolution (e.g., ray resolution, volume resolution). In that sense, the proposed method could be considered as coarse reconstruction method (also, the geometry and texture quality seem coarse). \n2-2) Claiming the proposed method as real-time approach sounds overstated since the generation time of diffusion model is quite slow. Please claim the computational performance in light of the whole pipeline.\n\n\n3] Weak clarity in implementation. \n3-1) It is not clear how the authors repose the human from the generated 3D volume primitives while preserving identity. \n3-2) While the number of parameters are five, the authors describe it as six-degree of freedom. Why six-degree of freedom? \n3-3) Figure 2 is not clear. The {input, output} modality of Unet should be 2D image space. However, the current modality is not image but the parameters of {T,R,s,c,sigma}. How did you handle the modality for such parameter prediction?\n3-4) In L165, it is not clear how the method can handle the texture outside the SMPL body parts. \n\n4] Low-quality results. \n4-1) The quality of the generated 3D human geometry and texture is low. In particular, the high-frequency details are highly lacking for both geometry and texture. \n4-2) Also, the texture is not reflective of geometry. For example, in Figure 1, the shade in the red dress of the women (of the upper right corner) does not reflect the depth information at all. \n4-3) Compared to existing work [1], the 3D generation quality is low. \nAre they all mainly due to the weakness of the 3D primitive representation?\n\n\n5] Weak validation, comparison, demonstration, and analysis: \n5-1) missing baseline for 3D human generation. In particular, this paper “must” be compared to gDNA [1] paper, which can generate 3D human geometry conditioned by SMPL model whose objective is exactly same as the one in this paer. Without this comparison, this paper should not claim “state-of-the-art” method; \n5-2) Since this paper claims good “computational efficiency”, it should provide the measure of inference time including denoising process, not only the rendering part; \n5-3) Due to the use of synthetic data, the human generation results look always synthetic and fake. It would be nice if the authors provide the results of the model trained on real multiview data, e.g., Thuman data [2].\n5-4) In L263 and Figure 6, it is unexpected that the voxel-based approach fails that much. Please provide a thorough insight and analysis why the voxel-based approach fails. \n\n\n[1] gDNA: Towards Generative Detailed Neural Avatars\n[2] Function4D: Real-time Human Volumetric Capture from Very Sparse RGBD Sensors\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive suggestions. We summarize and answer your questions as follows:\n\n**Q1: Fundamental motivation of the proposed method.**\n\nPlease kindly refer to the above post named \"Response to the common issue of the novelty and motivation\".\n\n**Q2-1: Clarification for the coarseness.**\n\nThanks for your question. We argue that the coarseness of 3D representations indicates the ratio of effective parameters in modeling the human body. As pointed out in L40, the human body exists within the vast expanse of 3D space, occupies only a sparse portion of it. An ideal representation should compactly model the human body without wasting parameters on modeling empty space. We present the intuition behind in the experiment below. As the accurate volume of the human body is hard to measure, we define the \"Efficiency\" of a certain 3D representation by projecting them onto camera views. For example, given a rendered human image, we render the occupied space of a 3D representation by setting the volume density to 1. Then we calculate the ratio between the foreground of the human and the projected 3D representation. This efficiency is calculated over 200 views where the camera orbits around the target human body. We present the result below, where the volumetric primitive has the highest efficiency, indicating it is the finest one to represent 3D humans given the computational budget.\n\n|Methods|Efficiency(\\%)|FPS|\n|:-:|:-:|:-:|\n|StyleSDF (SDF)|12.02$\\pm$1.96|2.53|\n|EG3D (Tri-plane)|12.02$\\pm$1.96|22.97|\n|EVA3D (Compositional voxels)|75.27$\\pm$1.51|6.14|\n|Ours (Volumetric primitives)|87.24$\\pm$1.25|88.24|\n\n**Q2-2: The computational performance in light of the whole pipeline.**\n\nPlease refer to the above post named \"Response to the common issue of the computational cost\".\n\n**Q3: Clarifications of implementation.**\n\nLet us elaborate on details case by case as follows. \n\n3-1) We repose the human by explicitly reposing the SMPL model through linear blending skinning. Our key insight is that the decoder-free rendering of volumetric primitives enables all primitives to learn the color and density and kinematic parameters in a physically correct way instead of an implicit way. Therefore, reposing only changes the kinematic parameter of primitives while keeping the identity unchanged. \n\n3-2) The actual degree of freedom is 13 since $T,R,s,c$ are three-dimensional vectors. The description of six degrees of freedom aims to highlight that each primitive has independent rotation and translation which differentiate from prior work where the voxel cannot rotate freely. Will polish L116 in our revised version. \n\n3-3) As you correctly pointed out, the input and output of Unet should be in 2D image space. Therefore, it is not straightforward to apply diffusion models to volumetric primitives. As we iterated in L140, the learnable parameters contain color $c$, density $s$, and delta scale factor $\\delta s$. Then, we clearly provide the detail in L190-L192 on how to form a parameter space of regular shape as the input of Unet. Specifically, we first pad the delta scale $\\delta s$ to the same size of color $c$, and concatenate them together into a tensor as $\\mathcal{V}_0 \\in R^{[W\\cdot S]\\times[W\\cdot S]\\times [7S]}$. \n\n3-4) In L165, we describe how to obtain image-based input feature in the UV space for the appearance branch. Note that, there is no need for this input feature to align with off-body topology like loose garments. We handle loose garments by the independent delta scale factor of each primitive, which is optimized through rendering reconstruction loss.\n\n**Q4: Quality of results.**\n\nThanks for your comments. Indeed, modeling shadows that are consistent with 3D geometry is an important and interesting future work for 3D human generative models. It requires the underlying 3D human representation to be as efficient as possible for simulating complex light transport, where the volumetric primitive has a chance for its efficiency. Moreover, we demonstrate an impressive ability to model off-body topology (shown as the red dress of the women in Figure 1), which is impossible for prior works like EG3D and EVA3D. \n\n**Q5: Comparisons with gDNA.**\n\nThanks for your reference. To the best of our knowledge, gDNA is an awesome model for generating a large variety of animatable 3D human shapes. However, our task is to generate textured 3D humans. The modality gap between RGB renderings and human shapes leads to different modeling between the two tasks. Therefore, the objective of our approach has a significant difference compared with gDNA. Moreover, we have cited and properly discussed gDNA in Section 2 of our paper (L88) as part of 3D human generation, where we believe gDNA will shed light on 3D human generative model on modeling more detailed geometry.\n\n**Q6: Experiments on THuman dataset.**\n\nThanks for your constructive comments. We train our model on THuman 2.0 dataset. Overall, we render 500 identities from THuman 2.0 with 36 camera views for each identity. We keep the training configuration unchanged and retrain our model from scratch on the rendered images. Note that, no explicit 3D supervision, e.g., normal or mesh, is used in our training. We present the results in the **Figure 2 of the attached figure-only file**, where the renderings show promising results.\n\n**Q7: Provide analysis for the failure of the voxel-based approach.**\n\nThanks for your question. The voxel-based approach densely allocates voxel grids with a resolution of $64^3$ to model 3D humans. As discussed in L263-L265, it wastes lots of parameters to densely model the entire scene instead of the human body itself, leading to the low spatial resolution of the representation. This is the root cause of its blurry renderings, which validates the importance of primitive-based representation for 3D human diffusion model."}]

Author rebuttal:
**Common response**

We sincerely thank all reviewers for the constructive comments and recognition of our work. We are encouraged that reviewers find our proposed method "innovative" (Reviewer oDDi), "novel" (Reviewer PWaT), and "effective" (Reviewer GMsy); our results "have superior performance" (Reviewer oDDi, PWaT) and "interesting" (Reviewer wB9H, aNCP). We have posted responses for each reviewer respectively.

**Response to the common issue of the novelty and motivation**

Both Reviewer wB9H and GMsy are concerned about the novelty and motivation of our method. Admittedly, the volumetric primitives, for their efficiency and effectiveness, have been proposed to model human faces. Nevertheless, it is not trivial to make it work on high-resolution and efficient 3D human generation. Specifically, the idea of volumetric primitives is proposed for identity-specific 3D reconstruction. However, it is intractable to fit every single human given a large-scale 3D human dataset. Instead, we propose a scalable framework with cross-modal attention for learning volumetric primitives without per-subject optimization. Moreover, our work first shows how to unleash the power of the diffusion model on 3D human generation while keeping real-time performance for novel pose and novel view synthesis. It is not a simple application of the diffusion model on volumetric primitives. Instead, we carefully choose the target domain for the denoiser as the color, density, and delta scale factor. It facilitates decoder-free rendering, which is critical for efficiency and generalizability to novel poses. 

Moreover, we summarize the motivations for leveraging volumetric primitives for 3D human generation compared to other 3D representations as follows:

**Flexibility and Adaptability**: Volumetric primitives offer a versatile and adaptive way to represent the articulated human body. By decomposing the human body into a collection of volumetric primitives, it can efficiently handle diverse poses and articulations within a compact parameter space. The effectiveness of offering compact parameter space for 3D generative model is also validated in EVA3D, where the authors propose a compositional representation to achieve a high-resolution 3D human generative model without the super-resolution module. This adaptability is challenging to achieve using other representations like point clouds and occupancy grids.

**Expressiveness**: The volumetric primitive representation allows for capturing the articulated structure of the human body by incorporating parametric human prior. Unlike point clouds, which lack explicit surface representations and may lead to loss of information, volumetric primitives provide a dense and detailed representation of the human body. Moreover, the volumetric primitive offers dense 3D correspondence of the human body, facilitating flexible controls on downstream generative tasks such as 3D in-painting and texture transfer. Other representations like point clouds, SDF, and occupancy grids do not have such flexibility. 

**Efficiency and Real-time Interactivity**: The efficiency of volumetric primitive representation contributes to real-time capabilities, making it suitable for interactive applications and systems that require quick response times. Note that, we only claim the real-time performance for novel view and novel pose synthesis once the denoising process is done (as introduced in L60 of the paper). Besides, it enables tractable training with a heavy denoiser backbone to unleash the capacity of the diffusion model and the high performance of volumetric primitives at the same time.

**Response to the common issue of the computational cost**

In response to the concerns from Reviewer PWaT, aNCP, and GMsy, we report the computational cost of our model. First, we provide the GPU memory consumption for both training and testing of the denoiser as follows. We will add this table to the supplementary material in our revised version.

| |GPU Memory (GB)| Batch size |
| :----: | :----: | :------: |
| Training | 27428 | 4 | 
| Inference | 21258 | 1 |

Moreover, we report the computational performance in light of the whole pipeline at inference time as follows. The FPS stands for novel view and pose synthesis of 300 frames. The amortized FPS indicates the FPS by considering the time of the denoising process. Our average inference FPS still outperforms baselines (the best baseline is only 22.97). More importantly, we only claim the real-time performance for novel view and novel pose synthesis once the denoising process is done (as introduced in L60 of the paper). In most cases, the generative backbones account for identity-specific information. Since we disentangle the pose and view control from the generative backbone in a physically explicit way, we only need to call the denoiser once for identity-specific appearance. Thanks to our decoder-free rendering, we do not need any forward pass through the denoiser for novel view and novel pose synthesis, which is the fundamental reason for real-time rendering. However, existing 3D generative models like EG3D, EVA3D, and StyleSDF implicitly condition view and pose as input features, which forces them to call the forward pass of heavy generative backbones upon the view and pose changes. We will add this table to the supplementary material in our revised version.

| Inference Mode| Denoising time | FPS | Amortized FPS |
| :----: | :----: | :------: | :------: | 
| DDIM, 100 steps | 2.86 s | 88.24 | 47.93 |
| DDIM, 50 steps | 1.45 s | 88.24 | 61.86 |

Please do not hesitate to let us know if you have any additional comments or if there are more clarifications that we can offer.

Meta-review: The manuscript received two weak accepts, two borderline accepts, and one borderline reject. It combines the diffusion model with volumetric primitives to facilitate 3D human generation. A common concern among reviewers was the experimental comparison, which the authors addressed in their rebuttal. Although the generated results tend to be smooth, the AC also supports the effort to use volumetric primitives and the diffusion model in developing generative models and, therefore, recommends acceptance.
Decision: Accept (poster)