Title: PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural rendering

3D generative model

diffusion model

volumetric primitives

3D human generation
Abstract: We present PrimDiffusion, the first diffusion-based framework for 3D human generation. Devising diffusion models for 3D human generation is difficult due to the intensive computational cost of 3D representations and the articulated topology of 3D humans. To tackle these challenges, our key insight is operating the denoising diffusion process directly on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This volumetric primitives representation marries the capacity of volumetric representations with the efficiency of primitive-based rendering. Our PrimDiffusion framework has three appealing properties: **1)** compact and expressive parameter space for the diffusion model, **2)** flexible representation that incorporates human prior, and **3)** decoder-free rendering for efficient novel-view and novel-pose synthesis. Extensive experiments validate that PrimDiffusion outperforms state-of-the-art methods in 3D human generation. Notably, compared to GAN-based methods, our PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of $512\times512$ once the denoising process is done. We also demonstrate the flexibility of our framework on training-free conditional generation such as texture transfer and 3D inpainting.

PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation

Abstract

We present PrimDiffusion , the first diffusion-based framework for 3D human generation. Devising diffusion models for 3D human generation is difficult due to the intensive computational cost of 3D representations and the articulated topology of 3D humans. To tackle these challenges, our key insight is operating the denoising diffusion process directly on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This volumetric primitives representation marries the capacity of volumetric representations with the efficiency of primitive-based rendering. Our PrimDiffusion framework has three appealing properties: 1) compact and expressive parameter space for the diffusion model, 2) flexible 3D representation that incorporates human prior, and 3) decoder-free rendering for efficient novel-view and novel-pose synthesis. Extensive experiments validate that PrimDiffusion outperforms state-of-the-art methods in 3D human generation. Notably, compared to GAN-based methods, our PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of 512 × 512 once the denoising process is done. We also demonstrate the flexibility of our framework on training-free conditional generation such as texture transfer and 3D inpainting.

1 Introduction

Generating 3D humans with high-resolution and 3D consistency is a central research focus in computer vision and graphics. It has wide-ranging applications in virtual reality, telepresence, and the film industry. These applications are becoming increasingly important as technology advances and the demand for realistic virtual experiences grows. Consequently, a flexible and high-quality 3D human generation system possesses not only immense scientific merit but also practical relevance.

For a high-quality and versatile 3D human generative model, we argue that there are two key factors: 1) high-capacity generative modeling, and 2) efficient 3D human representation. Recent advances in 3D-aware generative models [5, 18, 35, 37] have made significant progress. By learning from 2D images using generative adversarial networks (GAN) [14], these methods can synthesize images across different viewpoints in a 3D-aware manner. However, their results still leave significant gaps w.r.t. rendering quality, resolution, and inference speed. Besides, these GAN-based methods often require task-specific adaptations to be conditioned on a particular task, limiting their scope of application. Meanwhile, diffusion-based models [17, 51] have recently surpassed GANs on generative tasks with unprecedented quality, suggesting their great potential in 3D generation. Several recent works [1, 33, 57] have made initial attempts to generate 3D rigid objects based on the diffusion model. Nevertheless, how to integrate the denoising and diffusion processes into a generation framework of articulated 3D humans remains an open problem.

𝑡𝑡

=

𝑇𝑇

𝑡𝑡

= 0

……

……

Pose / View / Shape

Well-defined Depth

+

Specifically, applying diffusion models to 3D human generation is a non-trivial task due to the articulated topology, diverse poses, and various identities, among which an efficient human representation is the most important. Unlike human faces [23] and rigid objects [6, 22] where most existing 3D-aware generative models [1, 5, 15, 33] succeed, the human body exists within the vast expanse of 3D space, occupies only a sparse portion of it, characterized by an intricate, articulated topology. An ideal representation should compactly model the human body without wasting parameters on modeling empty space while also enabling explicit view and pose control. The most widely used representation for 3D-aware generative models is based on neural radiance field (NeRF) [32], which can be parameterized by coordinate-based MLP [15, 37] and hybrid representation like tri-plane [5], voxel grids [50] or compositional subnetworks [18]. However, these methods model humans as coarse volumes or planes, which limits their generation quality. In addition, they also suffer from slow inference speeds due to the need for dense MLP decoder queries during rendering.

In this paper, we propose PrimDiffusion , the first diffusion model for 3D human generation. Our key insight is that the denoising and diffusion process can be directly operated on a set of volumetric primitives, which models the 3D human body as a number of tiny volumes with radiance and kinematic information. Each volume parameterizes the spatially-varied color and density with six degrees of freedom. This representation fuses the capacity of volumetric representations with the efficiency of primitive-based rendering, which enables: 1) compact and expressive parameter space for the diffusion model, 2) flexible representation that inheres human prior, and 3) efficient and straightforward decoder-free rendering. Furthermore, these primitives naturally capture dense correspondence, facilitating downstream human-centric tasks such as texture transfer and inpainting with 3D consistency. Notably, previous GAN-based methods [18, 61] implicitly encode control signals ( e.g., view directions, poses) as inputs, indicating that they need extra forward passes upon condition changes ( e.g., novel view/pose synthesis). In contrast, PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of 512 × 512 once the denoising process is done.

Extensive experiments are conducted both qualitatively and quantitatively, demonstrating the superiority of PrimDiffusion over state-of-the-art methods for 3D human generation. We summarize our contributions as follows: 1) To the best of our knowledge, we introduce the first diffusion model for 3D human generation. 2) We propose to represent 3D humans as volumetric primitives in a generative context, which enables efficient training and high-performance rendering. 3) We design an encoder tailored with cross-modal attention, which accounts for learning volumetric primitives from images across identities without per-subject optimization. 4) We demonstrate applications of PrimDiffusion, including texture transfer and 3D inpainting, which can be naturally done without retraining.

2 Related Work

3D-aware Generation. With recent advances in neural radiance field (NeRF) [32], many studies [5, 9, 11, 15, 18, 35, 37, 49, 50, 59] have incorporated NeRF as the key inductive bias to make GANs be

{𝑻𝒕 , 𝑹

𝒕

, 𝒔

𝒕

}

𝑠𝑥

𝑘

𝑠𝑧

𝑘

𝑠𝑦

𝑘

𝑇 𝑘

, 𝑅 𝑘

𝓥𝟎

𝒌

= {𝑻 𝟎

𝒌

, 𝑹

𝟎

𝒌

, 𝒔

𝟎

𝒌

, 𝒄

𝟎

𝒌

, 𝝈

𝟎

𝒌

}

𝜖

{𝒄𝒕

, 𝝈

𝒕

}

𝓥𝒕

{𝑻𝟎, 𝑹 𝟎

, 𝒔

𝟎 }

{𝒄𝟎 , 𝝈

𝟎 }

෩

𝓥𝟎

𝑡

𝐾

Primitives

U-Net

𝑐 𝑘

, 𝜎 𝑘

3D-aware, which enables GANs to learn 3D-aware representations from 2D images. Researchers have proposed to use efficient representations like voxel grids [50] and triplanes [5] to improve the spatial resolution of the scene representation, which is a crucial factor for the quality of geometry and images. However, these techniques necessitate an implicit decoder ( e.g., a Multilayer Perceptron (MLP) or Convolutional Neural Network (CNN)) following the 3D representation to obtain the rendered images, which results in substantial computational overheads associated with neural volume rendering. Consequently, neither the image quality nor the rendering speed are satisfactory.

Diffusion Model. Despite the remarkable success of GAN-based models, diffusion-based models [17, 51, 52] have recently exhibited impressive performance in various generative tasks, especially for 2D tasks like text-to-image synthesis [34, 41, 44, 46]. However, their potential for 3D generation remains largely unexplored. A few attempts have been made on shape [25, 30, 60], point cloud [65], and text-to-3D [39] generation. Recently, some works have succeeded in learning diffusion models from 2D images for unconditional 3D generation [1, 33, 57] of human faces and objects. Still, given the articulated topology of the human body, efficiently training 3D diffusion models is challenging.

Human Generation. Extensive efforts have been made to generate human-centric assets [8, 19, 47, 62, 64], especially for 2D human images [12, 13, 21, 24, 48]. For 3D human generation, initial attempts have been made to generate human shapes [7, 56] using scanned data. Several works learn 3D-aware GANs from 2D image collections driven by differentiable rendering. However, they are either in low-resolution [36] or rely on super-resolution module [3, 61] which does not guarantee 3D consistency. EVA3D [18] succeeds in high-resolution 3D human generation within a clean framework. As a concurrent work, HumanGen [20] leverages explicit image priors from strong pretrained 2D human image generators and 3D geometry priors from PIFu [47]. However, these 3D-aware GAN-based methods implicitly condition the viewpoints and poses, which causes additional forward calls when conditions change, preventing high-resolution real-time performance.

3 Methodology

The goal of PrimDiffusion is learning to generate 3D humans from multi-view images built upon denoising and diffusion probabilistic models. It learns to revert the diffusion process that gradually corrupts 3D human representations by injecting noise at different scales. In this paper, 3D humans are represented as volumetric primitives, facilitating feasible training of diffusion models and efficient rendering. We first introduce this 3D human representation and corresponding rendering algorithm (Sec. 3.1). Then, a generalizable encoder is employed to robustly fit primitive representations from multi-view images across identities without per-subject optimization (Sec. 3.2). Once the volumetric primitives are learned from images, the problem of 3D human generation is reduced to learning the distribution of 3D representations. We thereby present our denoising and diffusion modeling operated on the parameter space of volumetric primitives to generate high-quality 3D humans (Sec.3.3). An overview of PrimDiffusion is presented in Fig. 2.

3.1 Primitive-based 3D Human Representation

To learn a denoising and diffusion model with explicit 3D supervision, we need an expressive and efficient 3D human representation that accounts for 1) compact parameter space for diffusion models, 2) articulated topology and motion of 3D humans, and 3) fast rendering with explicit 3D controls.

Volumetric Primitives. Inspired by previous work [28, 43] in reconstructing dynamic human heads and actors, we propose to represent 3D humans as a set of volumetric primitives which acts as the target space for diffusion models. Note that, whereas previous research has centered on the per-subject reconstruction, we enable learning volumetric primitives across multiple identities. This paves the way for a shared latent space conducive to generative modeling. In specific, this representation consists of a set of K primitives that jointly parameterize the spatially varied color and density:

Each primitive V k is a tiny volume, parameterized by a translation vector T k ∈ R 3 , an orientation R k ∈ SO(3) , a three-dimensional scale factor s k ∈ R 3 , a spatially-varied color c k ∈ R 3 × S × S × S , and density σ k ∈ R S × S × S , where S is the per-axis base resolution. In practice, we set the base resolution of one primitive to S = 8 , which we find the best to strike a balance between reconstruction quality and memory consumption.

To model humans with articulated topology and motion, we attach primitives to the vertices of SMPL [29]. SMPL is a parametric human model, defined as M ( β, θ ) ∈ R 6890 × 3 , where β ∈ R 10 controls body shapes and θ ∈ R 72 controls body poses. In this paper, we leverage this parametric human model for the purposes of 1) serving as a geometry proxy to weakly constrain volumetric primitive representations, 2) offering geometry cues for 3D representation fitting, and 3) generalizing to novel poses through linear blending skinning (LBS) algorithm tailored with decoder-free rendering.

Specifically, the kinematic parameters of primitives are modeled relative to the base transformation from SMPL. Namely, we generate a W × W 2D grid in the UV space of SMPL. The 3D positions of primitives are initialized by uniformly sampling UV space and mapping each primitive to the closest surface point on SMPL that corresponds to the UV coordinates of the grid points. The orientation of each primitive is set to the local tangent frame of the 3D surface point. The per-axis scale is calculated based on the gradient of the UV coordinates at the corresponding grid point. All primitives are given an initial scale that is proportional to the distances between them and their neighbors. Furthermore, to model off-body topologies like loose clothes and fluffy hair, the scale of primitives is allowed to deviate from the base scale ˆ s k . The scale factor is defined as s k = ˆ s k · δ s k , where ˆ s k is the aforementioned initialized base scale and δ s k is the delta scaling prediction.

Therefore, the learnable parameters of each person modeled by its volumetric primitives V are color c ∈ R W × W × 3 × S × S × S , density σ ∈ R W × W × 1 × S × S × S , and delta scale factor δ s ∈ R W × W × 3 .

Efficient Decoder-free Rendering. Once the radiance and kinematic information of primitives are determined, we can leverage differentiable ray marching [27] to render the corresponding camera view. For a given camera ray r p ( t ) = o p + t d p with origin o p ∈ R 3 and ray direction d p ∈ R 3 , the corresponding pixel value I p is calculated as an integral:

where t min and t max are the near and far bound for points sampling. The c ( · ) and σ ( · ) are global color and density fields derived from trilinear interpolation of color and density attributes among primitives hit by the camera ray. Notably, in contrast to NeRF-based methods [32] which often require evaluating MLP decoders per camera rays, this rendering process is decoder-free which can be directly rendered into pixels. Thus, it is far more efficient for both training and rendering.

To animate the volumetric primitives given a pose sequence { θ j } N frame j =1 , we explicitly employ the LBS algorithm on top of SMPL [29] to produce per-frame human mesh M ( β, θ j ) , which serve as driving signal to determine the kinematic information of the moving volumetric primitives.

Ƹ

Multi-view Images

𝐼

𝑖

𝑖

𝑁cam

SMPL Model

ℳ(𝛽,𝜃)

Motion Branch

LBS(

𝜃

)

Pose feature

UV Space

𝐹𝜃

𝑻, 𝑹, ො 𝒔

Unwrap

Image Feature

𝐹𝐼

Motion feature

𝑊

𝑊

Appearance feature

𝑊

𝑊

Cross-modal Attention Layer

𝐹 s

𝐹rgb

𝐹 𝜎

𝛿𝒔

𝒄

𝝈

𝑇, 𝑅

𝑠 =

𝑠 ⋅ 𝛿𝑠

Color

Density

Scale factor

Decoder-free Rendering

3.2 Generalizable Primitive Learning from Multi-View Images

In order to leverage explicit 3D supervision from multi-view images, it is essential to reconstruct 3D representations for each individual to provide ground truth for the diffusion model. While earlier research [57] on 3D diffusion models adhered to a per-identity fitting fashion, which could be timeconsuming on a large-scale dataset. To overcome this limitation, we propose a generalizable encoder to reconstruct 3D human representation across diverse identities without per-subject optimization.

Our setting is learning 3D representations from multi-view images {I i } N cam i =1 , where N cam denotes the number of camera views and i is the corresponding index. We first estimate the SMPL [29] parameters M ( β, θ ) using off-the-shelf tools [10]. Then, we employ an encoder with dual branches to parameterize the coarse human mesh as well as the multi-view images, which outputs the radiance and kinematic information of volumetric primitives. The architecture is illustrated in Figure 3.

Specifically, the encoder consists of two branches, one is for appearance while the other is for motion. For the appearance branch, we back-project the pixels of input images corresponding to all visible vertices of the SMPL model to the UV space. The unwrapped image textures will be averaged across multiple views and fed to the image encoder F I ( I i ; Φ I ) parameterized by Φ I , providing texel-aligned appearance features. For the motion branch, we take as input the local pose parameters of SMPL model, i.e., θ excluding the first three elements that denote global orientation. The local pose parameters are padded to a W × W 2D map and fed to the motion encoder F θ ( θ ; Φ θ ) parameterized by Φ θ , providing motion features. Instead of fusing the features from two branches via naive concatenation, we propose a cross-modal attention module to effectively capture the dependencies between appearance and motion, which significantly improves the reconstruction quality of 3D humans (Tab. 3 and Fig. 7). Intuitively, this module captures the interdependence of color and motion, e.g., clothes wrinkles and shadows caused by different human poses. Then, the radiance and kinematic information (color, density, and delta scale) of volumetric primitives are separately predicted by three mapping networks ( F rgb , F σ , F s ) conditioned on the fused feature.

The encoder comprises { F I , F θ , F rgb , F σ , F s } is trained in an end-to-end manner across identities instead of per-subject optimization. The learning objective is formulated as a reconstruction task between ground truth images and rendered images:

where L rgb is the image reconstruction loss in L2-norm, L sil is the silhouette loss, L vol is the volume regularization term [28], and λ ∗ are loss weights. The volume regularization is defined as L vol = ∑ K i =1 Prod( s i ) , where Prod( · ) denotes the product of scale factor along three axes. It aims to prevent large primitives from occupying empty space that leads to loss of resolution.

3.3 Primitive Diffusion Model

Once the volumetric primitives are reconstructed, the problem of 3D human generation is reduced to learning the distribution p ( V ) with diffusion models. In specific, to generate a 3D human with its

representation as V 0 , the diffusion model learns to denoise V T ∼ N ( 0 , I ) progressively closer to the data distribution through denoising steps {V T -1 , . . . , V 0 } . Note that, the targets of denoising and diffusion process are color c ∈ R W × W × 3 × S × S × S , density σ ∈ R W × W × 1 × S × S × S , and delta scale factor δ s ∈ R W × W × 3 . To form a parameter space with a regular shape, we first pad the delta scale δ s to the same size of color c . Then, we concatenate and reshape them into a 3D tensor. With slight abuse of notation, we denote this tensor as V 0 ∈ R [ W · S ] × [ W · S ] × [7 · S ] in the following subsections.

Forward Process. Given a data point, i.e., volumetric primitives of a human, sampled from a real data distribution V 0 ∼ q ( V ) , we define the forward diffusion process in which we add Gaussian noise to the sample in T steps, producing a sequence of noisy samples {V 1 , . . . , V T } :

where the step sizes are controlled by a variance schedule { β t ∈ (0 , 1) } T t =1 . We use a linear scheduler [17] in our experiments. Denoting α t = 1 -β t and ¯ α t = ∏ t i =1 α i , we can directly sample V t at any arbitrary time step t in a closed form through reparameterization:

Reverse Process. The reverse process, namely the denoising process, aims to generate a true sample V 0 from a Gaussian noise V T ∼ N ( 0 , I ) , which requires an estimation of the posterior distribution:

where ˜ β t = (1 -¯ α t -1 ) / (1 -¯ α t ) β t . Notably, V 0 is the generation target that is unknown. Thus, we need to learn a denoiser g Φ parameterized by Φ to approximate these conditional probabilities p Φ ( V t -1 |V t ) ≈ q ( V t -1 |V t , V 0 ) , i.e., estimate the mean ˜ µ ( · ) . In practice, we train the denoiser g Φ to predict V 0 such that:

where the posterior probabilities can be approximated as p Φ ( V t -1 |V t ) = N ( V t -1 ; µ Φ ( V t , t ) , ˜ β t I ) . During inference, this posterior is sampled at each time step t to gradually denoise the noisy sample V t . The denoiser g Φ is trained to minimize the difference from ˜ µ which can be simplified [17] as :

4 Experiments

Dataset. We obtain 796 high-quality 3D humans from RenderPeople [55] with diverse identities and clothes. For each person, we repose the mesh with 20 different human poses [31] to ensure pose diversity. For each pose instance of a person, we render 36 multi-view images with known camera poses. All methods are trained from scratch (except mentioned) on this dataset for fair comparisons.

a) StyleSDF

b) EG3D

d) Ours

Implementation Details. We use K = W 2 = 1024 primitives to represent each 3D human. The denoiser g Φ is implemented as a 2D U-Net [45] with intermediate attention layers. We first train the generalizable encoder using all available images. Then, the denoiser g Φ is trained according to the primitives produced by the frozen encoder. Please check the supplementary material for more details.

Evaluation Metrics. Weadopt Fréchet Inception Distance (FID) [16] and Kernel Inception Distance (KID) [4] to evaluate the quality of rendered images. Note that, we use different backbones to evaluate FID in different latent spaces in order to get a thorough evaluation, i.e., FID CLIP employs the CLIP image encoder [40] while FID employs the Inception-V3 model [54]. To evaluate the 3D geometry, we use an off-the-shelf tool [42] to estimate depth maps from renderings and compute L2 distance against rendered depths. We further adopt a human-centric metric, Percentage of Correct Keypoints (PCK) [2], to evaluate the pose controllability of 3D human generative models.

Comparison Methods. As the first diffusion model for 3D human generation, we compare with three GAN-based methods. EVA3D [18] learns to generate 3D human from 2D image collections. EG3D [5] and StyleSDF [37] are approaches for 3D-aware generation from 2D images, succeeding in generating human faces and objects. All of these methods enable view control by implicitly conditioning on the camera pose, which leads to extra forward passes upon viewpoint changes.

4.1 Qualitative Results

We show the RGB renderings and corresponding depth maps generated by PrimDiffusion and baselines in Fig. 5. StyleSDF and EG3D achieve coarse results in terms of appearance and geometry. However, due to the lack of human prior, most of the capacity is wasted on modeling the empty space that is not occupied by the human body, which leads to blurry facial details. Furthermore, the inefficient 3D representations make them hard to scale up to high resolution without super-resolution modules, which also limits their geometry to low resolution ( 64 2 or 128 2 ). EVA3D succeeds in high-resolution rendering with much better geometry by incorporating human prior. However, it fails to generate enough facial details. We attribute this to its part-wise human representation. For example, EVA3D models the human head as a single volume, assigning a difficult learning problem of distinguishing facial details within one uniform representation. In contrast, PrimDiffusion divides humans into many primitives tailed with powerful diffusion modeling, which is not only capable of

Voxel

Primitive

(Ours)

Latent Codes

N/A

N/A

N/A

N/A

modeling enough details ( e.g., faces and textures) but also supporting 512 × 512 resolution for both RGB and depth without any super-resolution decoder.

4.2 Quantitative Results

The results of numerical comparisons are presented in Tab. 1. As general 3D generation approaches, StyleSDF and EG3D get fair scores in terms of generation quality, but the geometries are much worse than others. EVA3D generates good geometries, indicated by its comparable depth error with us. However, the image quality is still far behind us. Note that, we observe the vanilla FID might be inaccurate, which leads to discrepancies between visualizations and numerical results. For example, the images rendered by EG3D are of poor quality but the FID is much lower than other baselines. Therefore, we report FID CLIP metric which we found more consistent with qualitative results. Nevertheless, the proposed method still outperforms baselines by a large margin.

Moreover, we also compare our model with a finetuned EVA3D that is first pre-trained on DeepFashion [26] dataset to establish 3D representations of humans and then finetuned on our RenderPeople dataset. The motivation for this experiment is to fully leverage the ability to learn 3D representations on both multiview images and collections of images for methods like EV A3D instead of training from scratch with multiview images only. However, as shown in Tab. 1, the finetuned EVA3D still has a performance gap compared with our method.

4.3 Ablation Studies

Different Representations for 3D Diffusion. Apparently, the volumetric primitive is not the only representation for 3D human diffusion models. We also explore the effectiveness of other representations, like voxel grids [53] and latent codes [38]. The 'voxel' baseline differentiates itself

Ours

w/o

attention

w/o

𝑅, 𝑇

GT

w/o

varying

𝛿𝑠

from our method in two ways: 1) it utilizes voxel grids with a resolution of 64 × 64 × 64 for fitting 3D representation from multi-view images, and 2) it leverages 3D convolution layers instead of 2D for instantiation of g Φ ( · ) . Note that, as 3D convolution layers occupy much more memories than 2D, the feasible resolution of voxel grids is rather limited. Moreover, the 'latents' baseline inherits the idea of anchoring latent codes [38] to the vertices of the SMPL model for 3D human representation learning. However, the vanilla NeuralBody [38] takes 14 hours per subject to converge which makes it unscalable to large-scale datasets like RenderPeople which contains lots of identities. Thus, we implement the 'latents' baseline in the following way. We replace the payload of volumetric primitives with latent codes and leverage sparse convolution layers as the decoder to output the color and density of radiance fields. This implementation of a NeuralBody version of our method keeps the ability to learn cross-identity 3D representation through our generalizable primitive learning framework while getting rid of per-subject fitting.

Quantitative results are presented in Tab. 2. The voxel-based representation fails to achieve a reasonable result both in rendering quality and geometry. The latent codes baseline achieves plausible generated results but worse quality compared with ours. We further visualize the qualitative comparisons in Fig. 6. The voxel-based approach wastes many parameters to densely model the scene, leading to the low spatial resolution of the representation. This is the root cause of its blurry renderings, which validates the importance of our primitive-based representation for 3D human diffusion model.

Ablation on Design Choices for Primitives Fitting. The quality of primitives fitting significantly affects the generation quality, as it provides 3D ground truth for the diffusion model. Therefore, we ablate the effectiveness of different designs for generalizable primitives fitting, presented in Fig 7 and Tab. 3. Note that, we use reconstruction metrics for evaluation. We denote 'w/o varying δs ' as the baseline that replaces the spatially-varied delta scale factor δs with a uniformly distributed scale factor, i.e., all primitives share the same delta scale factor. It fails to reconstruct reasonable 3D humans. The 'w/o R,T ' denotes the method that uses predefined rotation and translation calculated from a template human mesh instead of an identity-specific one, which leads to a slight drop in quality. And 'w/o attention' denotes the method trained without our proposed cross-modal attention module, which is important for detailed textures and motion-dependent effects.

4.4 Further Analysis

Real-time Inference. Thanks to our efficient representation and decoder-free rendering procedure, PrimDiffusion can render 3D consistent humans with varying viewpoints and poses in a resolution of 512 × 512 at 88.24 FPS once the denoising process is done. As shown in Tab. 1, due to the use of the decoder which decodes the 3D representation to images, all baselines fail to high-resolution real-time rendering at a comparable quality with us. Furthermore, these methods implicitly model 3D-aware conditions, i.e., viewpoints and poses, which causes extra forward pass calls upon conditions changes.

Generalizability to Novel Poses. It is worth mentioning that a decoder ( e.g., MLP for NeRF decoding or CNN for super-resolution) may lead to overfitting to the pose distribution of the training dataset, which prohibits pose generalization to out-of-distribution. Another merit of our decoder-free rendering is that we can generalize to novel poses without test-time optimization, post-processing, and rigging (Fig. 1 and Fig. 4). We refer readers to the supplementary material for video results.

Texture Source

Target

Correspondence

Transfer

Mask

Guidance

a) Texture Transfer

b) 3D Inpainting

Add Noise

4.5 Applications

Texture Transfer. Based on the dense correspondence offered by volumetric primitives, we can transfer the texture from one human body to another, as shown in Fig. 8 a). Since the entire process is operated in 3D space, we can render the human with transferred texture with free viewpoints.

3D Inpainting. Thanks to the flexibility of primitive-based representation and diffusion model, PrimDiffusion can inpaint the masked region on the human body with new textures as shown in Fig. 8 b). By perturbing the masked region with noise, we take the volumetric primitives as the mask guidance to the denoiser g Φ ( · ) . Note that, in contrast to GAN-based methods, this conditional generation is done without retraining the model.

5 Conclusion

In this paper, we propose the first diffusion model for 3D human generation. To offer a compact and expressive parameter space with human prior for diffusion models, we propose to directly operate the diffusion and denoising process on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This flexible yet efficient representation enables high-performance rendering for novel views and poses, with a resolution of 512 × 512 in real-time. Furthermore, we also propose a cross-modal attention module tailored with an encoder-based pipeline for fitting volumetric primitives from multi-view images without per-subject optimization. We believe our method will pave the way for further exploration in 3D human generation.

Limitation. While our method shows promising results for unconditional 3D human generation, several limitations remain. First, though we can model off-body topology like loose garments and long hair, how to animate them remains a challenge. Second, our renderings contain artifacts for complex textures. Leveraging 2D pre-trained generative models would be a possible solution. Last, due to the high degree of freedom of primitives, our method is a two-staged pipeline to prevent training instability. Further explorations on single-stage 3D diffusion models would be fruitful. In addition, compared with GAN-based approaches, our method requires a sufficient number of posed views to get good 3D representations for training diffusion models. In this context, it would be interesting to explore few-shot inverse rendering techniques to reduce the reliance on multiview images for the first stage of our method.

Broader Impacts. The generated 3D humans of PrimDiffusion might be misused to create misleading content or fake media. It can also edit personal imagery, leading to privacy concerns.

Acknowledgments and Disclosure of Funding

This work is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-019), NTU NAP, MOE AcRF Tier 2 (T2EP202210012), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

References

Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero. RenderDiffusion: Image diffusion for 3D reconstruction, inpainting and generation. arXiv , 2022. 1, 2, 3

Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 3686-3693, 2014. 7

Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. Advances in Neural Information Processing Systems , 35:19900-19916, 2022. 3

Mikołaj Bi´ nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMDGANs. In International Conference on Learning Representations , 2018. 7

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR , 2022. 1, 2, 3, 7

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012 , 2015. 2

Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avatars. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Zhaoxi Chen and Ziwei Liu. Relighting4D: Neural relightable human from videos. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV , pages 606-623. Springer, 2022. 3

Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Scenedreamer: Unbounded 3D scene generation from 2D image collections. In arXiv , 2023. 2

MMHuman3D Contributors. Openmmlab 3D human parametric model toolbox and benchmark. https://github.com/open-mmlab/mmhuman3d , 2021. 5

Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3D-aware image generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. 2

Anna Frühstück, Krishna Kumar Singh, Eli Shechtman, Niloy J. Mitra, Peter Wonka, and Jingwan Lu. Insetgan for full-body image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) , pages 77237732, June 2022. 3

Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen-Change Loy, Wayne Wu, and Ziwei Liu. Stylegan-human: A data-centric odyssey of human generation. arXiv preprint , arXiv:2204.11823, 2022. 3

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. 1

Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3D aware generator for high-resolution image synthesis. In International Conference on Learning Representations , 2022. 2

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. 7

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020. 1, 3, 6

Fangzhou Hong, Zhaoxi Chen, Yushi LAN, Liang Pan, and Ziwei Liu. EVA3D: Compositional 3D human generation from 2D image collections. In International Conference on Learning Representations , 2023. 1, 2, 3, 7

Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: zero-shot text-driven generation and animation of 3D avatars. ACM Transactions on

Graphics (TOG) , 41(4):1-19, 2022. 3

Suyi Jiang, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12543-12554, June 2023. 3

Yuming Jiang, Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACMTransactions on Graphics (TOG) , 41(4):1-11, 2022. 3

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017. 2

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019. 2

Kathleen M Lewis, Srivatsan Varadharajan, and Ira Kemelmacher-Shlizerman. Tryongan: Body-aware try-on via layered interpolation. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2021) , 40(4), 2021. 3

Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3D mesh modeling. In International Conference on Learning Representations , 2023. 3

Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 7, 8

Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph. , 38(4):65:1-65:14, July 2019. 4

Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph. , 40(4), jul 2021. 4, 5

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) , 34(6):248:1-248:16, Oct. 2015. 4, 5

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3D point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021. 3

Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision , pages 5442-5451, Oct. 2019. 6

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021. 2, 4

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3D radiance field diffusion. In arxiv , 2022. 1, 2, 3

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning , pages 16784-16804, 2022. 3

Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2021. 1, 2

Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. arXiv:2204.08839 , 2022. 3

Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira KemelmacherShlizerman. StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13503-13513, June 2022. 1, 2, 7

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9054-9063, 2021. 8, 9

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. arXiv preprint arXiv:2209.14988 , 2022. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 7

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022. 3

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1217912188, 2021. 7

Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu, Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao, Fabian Prada, Jason Saragih, et al. Drivable volumetric avatars using texel-aligned features. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1-9, 2022. 4

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022. 3

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015. 7

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 3

Shunsuke Saito, , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. arXiv preprint arXiv:1905.05172 , 2019. 3

Kripasindhu Sarkar, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. Humangan: A generative model of human images. In 2021 International Conference on 3D Vision (3DV) , pages 258-267. IEEE, 2021. 3

Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , 2020. 2

Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3D-aware image synthesis with sparse voxel grids. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 2, 3

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. 1, 3

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. 3

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022. 8

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2818-2826, 2016. 7

https://renderpeople.com/3d-people/ . Renderpeople, 2018. 6

Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning animatable clothed human models from few depth images. In Advances in Neural Information Processing Systems , 2021. 3

Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3D digital avatars using diffusion. In Proceedings of the IEEE/CVF Conference on Computer

Vision and Pattern Recognition , pages 4563-4573, 2023. 1, 3, 5

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004. 9

Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds. Technical Report arXiv:2206.07255, arXiv, June 2022. 2

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3D shape generation. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 3

Jianfeng Zhang, Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: A 3D generative model for animatable human avatars. In Arxiv , 2022. 2, 3

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001 , 2022. 3

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018. 9

Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence , 44(6):3170-3184, 2021. 3

Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5826-5835, October 2021. 3

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to generate various 3D human geometry and appearance. To this end, it introduces a new design of diffusion model that learns to generate the parameters of 3D primitives of humans: a generalizable encoder constructs motion and appearance features and their attentions are fed to predict scale, color, and density of each primitive. The evaluation is performed on the render people dataset, and the authors demonstrate a number of applications such as reposing and 3D inpainting.\n\nSTRENGTHS:\n+ This paper effectively combines existing 3D diffusion model and existing 3D representation to perform generalizable 3D human generation. It is the first attempt to combine diffusion model and primitive representation for 3D generative humans. It can serve as good initial baseline for future research.\n+ The 3D inpainting application is interesting, which allows reconstructing a complete 3D human from a partial body.\n\nWEAKNESSES:\n1] The motivation of this paper is not clear. \nIn particular, it is not clear why the volume primitive representation is ideal (other than efficiency) for handling complex human articulations. Other than computational efficiency, why does this representation fundamentally enables better quality of 3D human generation than other representation (e.g., point clouds, sdf, occupancy, and so on)? \nSimply showing the ablation study (without thorough analysis) does not account for this fundamental motivation, and I would like to request the authors to explain above question with fundamental reasons. Without strong motivation, this work will fall into the work that just simply combines existing 3D diffusion with existing representation as one try of a baseline. \n\n2] Misleading claims and overclaims: \n2-1) In L45, existing methods (nerf or tri-plane) do not model humans as coarse volumes. They model the geometry in the continuous space, which can lead to the fine-results. In fact, the coarseness is quite proportional to the resolution (e.g., ray resolution, volume resolution). In that sense, the proposed method could be considered as coarse reconstruction method (also, the geometry and texture quality seem coarse). \n2-2) Claiming the proposed method as real-time approach sounds overstated since the generation time of diffusion model is quite slow. Please claim the computational performance in light of the whole pipeline.\n\n\n3] Weak clarity in implementation. \n3-1) It is not clear how the authors repose the human from the generated 3D volume primitives while preserving identity. \n3-2) While the number of parameters are five, the authors describe it as six-degree of freedom. Why six-degree of freedom? \n3-3) Figure 2 is not clear. The {input, output} modality of Unet should be 2D image space. However, the current modality is not image but the parameters of {T,R,s,c,sigma}. How did you handle the modality for such parameter prediction?\n3-4) In L165, it is not clear how the method can handle the texture outside the SMPL body parts. \n\n4] Low-quality results. \n4-1) The quality of the generated 3D human geometry and texture is low. In particular, the high-frequency details are highly lacking for both geometry and texture. \n4-2) Also, the texture is not reflective of geometry. For example, in Figure 1, the shade in the red dress of the women (of the upper right corner) does not reflect the depth information at all. \n4-3) Compared to existing work [1], the 3D generation quality is low. \nAre they all mainly due to the weakness of the 3D primitive representation?\n\n\n5] Weak validation, comparison, demonstration, and analysis: \n5-1) missing baseline for 3D human generation. In particular, this paper “must” be compared to gDNA [1] paper, which can generate 3D human geometry conditioned by SMPL model whose objective is exactly same as the one in this paer. Without this comparison, this paper should not claim “state-of-the-art” method; \n5-2) Since this paper claims good “computational efficiency”, it should provide the measure of inference time including denoising process, not only the rendering part; \n5-3) Due to the use of synthetic data, the human generation results look always synthetic and fake. It would be nice if the authors provide the results of the model trained on real multiview data, e.g., Thuman data [2].\n5-4) In L263 and Figure 6, it is unexpected that the voxel-based approach fails that much. Please provide a thorough insight and analysis why the voxel-based approach fails. \n\n\n[1] gDNA: Towards Generative Detailed Neural Avatars\n[2] Function4D: Real-time Human Volumetric Capture from Very Sparse RGBD Sensors\n\nQUESTIONS:\nPlease address the questions in the weakness section.\n\nLIMITATIONS:\nThis paper sufficiently discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generative model for 3D humans. The work is based on volumetric rendering of geometric primitives and diffusion model. Given multi-view images of humans as training data, the method first learns a volumetric primitives representation, and then applies diffusion model framework to model the representation distribution. The authors evaluate their method and compare to SOTA approaches on a relatively small synthetic multi-view dataset. Results show favor towards proposed approach.\n\nSTRENGTHS:\n1. The paper tackles an interesting, practical and high-impact problem: generating clothed 3D humans.\n\n2. The paper is overall well written, easy to follow.\n\n3. The use of cross attention to fuse pose info and texture info is appreciated and demonstrated good performance.\n\n4. I'm glad to see the attempt of using volumetric primitives and diffusion model to create generative models of 3D humans.\n\nWEAKNESSES:\n1. The bibliography needs to be better formatted. \na) The same conference is referred in different format, e.g. CVPR vs IEEE Conference on Computer...\nb) Capitalization of words, e.g. 3d -> 3D, Shapenet -> ShapeNet\nc) Others, e.g. [30] two commas, [31] where is it published?\n\n2. As a reference, I'd like to have a fidelity evaluation of \"ground truth\" volumetric primitives, better added into Table 1 in the updated version. This helps to understand the upper limit of the diffusion process in the proposed method and see where the bottle neck lies.\n\n3. It looks like all the generated results are blurry. Is it due to the memory limit? What is the GPU memory consumption for W=32 (assuming it is the setting of all the experiments)? Is the memory consumption a limitation for the proposed method compared to EVA3D? In other words, if one had a large size of multi-view human data, would it be likely that EVA3D performs much better than the proposed method given current limit of computational power?\n\n4. Most of the generated results presented in the submission (including the video) contain mono-color clothing. There are only two or three exceptions. Is it a bias of the dataset or is it a result of low resolution volumetric primitives? Could the authors provide a statistics on the ratio of mono-color clothing in the Renderpeople dataset?\n\n5. What is the difference of texture transfer and reposing? It looks to me that transfer the source texture to the target subject is the same as repose (and reshape the body of) the target subject to the source pose (and source body shape). If this is true, please remove this application as it is basically in 4.4 already.\n\nQUESTIONS:\nI would like the authors to answer Weaknesses 2, 3, 4, and 5. The answers may not change my final decision as long as no major flaw is found later. The lack of details in the generated results prevent me from giving strong support of the submission. But the attempt to address 3D generative human models with diffusion model and volumetric primitives should be encouraged.\n\nLIMITATIONS:\nYes, both are tackled in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a diffusion-based approach, PrimDiffusion, for 3D human generation. The method leverages the representation of MVP (mixture of volumetric primitives) and a design of volumetric primitives diffusion to generate high-quality 3D human models using multi-view images as training data. The authors propose a two-stage training process: the first stage involves learning generalizable primitives from multi-view images, and the second stage involves a diffusion process to generate/sample these primitives. The paper also presents an encoder design tailored with cross-modal attention for learning volumetric primitives from images across identities. This encoder, which consists of an appearance branch and a motion branch, is trained end-to-end across identities, eliminating the need for per-subject optimization. The cross-modal attention module within the encoder effectively captures dependencies between appearance and motion, contributing to the generation of high-quality 3D humans. The paper includes extensive experimental results demonstrating the effectiveness of the proposed method in comparison with existing techniques.\n\nSTRENGTHS:\n(+) A novel way of generating 3D human models using volumetric primitives diffusion.\n\n(+) The paper is well-structured and clearly written. The authors provide a clear explanation of their method. The supplementary document provides additional technical details, and the supplementary video also helps to showcase the quality that the proposed method achieves.\n\n(+) The proposed method achieves superior results compared to existing techniques on the RenderPeople multiview dataset.\n\nWEAKNESSES:\n(-) The proposed method is a 2-stage approach, compared to previous GAN-based methods such as EVA3D, it requires **multi-view images** as training data to learn generalizable primitives in the first stage. This is a significant difference as collecting large-scale real-world multi-view human data can be challenging, while 2D image collections are relatively easier to gather. This distinction is not clearly articulated in the paper. Considering a single-stage solution might address this issue.\n\n(-) The experiments lack a comparison with a recent baseline: Jiang, Suyi, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. \"Humangen: Generating human radiance fields with explicit priors.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12543-12554. 2023.\n\n(-) Please refer to the questions section for more points.\n\nQUESTIONS:\n- In Figure 3, top-left, the input is denoted as \"Body Model $M(\\beta, \\theta)$\", but the figure appears to depict a detailed model rather than a SMPL model. Could the authors clarify this discrepancy?\n\n- When projecting image features and pose features to the SMPL UV space, how are the **\"empty\"** UV regions handled? What are the initial values for the primitive positions corresponding to these empty regions?\n\n- What is the memory consumption of the proposed method during training and testing?\n\n- The training data includes 20 different human poses. How were these poses sampled? What criteria were used to ensure a comprehensive coverage of possible poses?\n\nLIMITATIONS:\nThe proposed method requires the use of multi-view images as training data. This requirement, when compared to previous works that do not have such a constraint, should be discussed in detail as a limitation of the current approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents, PrimDiffusion, an approach for generative modeling of 3D humans. PrimDiffusion is a diffusion-based model that operates on a set of volumetric primitives representing the human body. Each volumetric primitive consists of rotation, translation, scale, radiance, and color information. PrimDiffusion is a multi-stage approach. Given a training dataset with multiview images, the first stage fits volumetric primitives on training images using reconstruction losses. The second stage then uses the fitted volumetric primitives to train a diffusion-based generative model where the results from the first stage are used as the ground truth for the diffusion process.  The use of volumetric primitives allows decoder-free rendering of the images which allows real-time rendering once the diffusion process is done. It also provides explicit control over the body pose and viewpoint. The paper also shows its application for texture transfer and unconditional infilling. Experiments are performed on the synthetic RenderPeople dataset where PrimDiffusion is shown to outperform other baselines.\n\nSTRENGTHS:\n- The paper addresses the challenging problem of generative modeling of 3D humans. \n- The use of volumetric primitives to represent complex human bodies, clothing, and hairs makes sense and is intuitive. \n- The main challenge for learning 3D diffusion models is the availability of 3D ground truths. The paper overcomes this challenge by first fitting volumetric primitives on multiview images. \n- The paper inherits the advantages of volumetric primitives including decoder-free rendering and explicit control over viewpoint and body pose. \n- The paper demonstrates two interesting applications of the proposed method including texture transfer and infilling. \n- The paper is well-written and easy to read.\n\nWEAKNESSES:\n### Novelty \n- The main novelty of the proposed method comes from the combination of existing bits and pieces. The volumetric primitives have already been used in [25] to model human faces. The paper extends it to the full body by incorporating the SMPL body model. The idea of anchoring neural primitives to the SMPL body model is similar to NeuralBody [a] `(missing citation)`, and generative modeling using diffusion models is standard practice these days. While there is no technical novel contribution of the paper, I acknowledge that the proposed combination is not straightforward and this is not the most crucial limitation of the paper. \n\n### Comparison with existing methods \n- This is the most crucial limitation of the paper. The paper compares with EVA3D, StyleSDF, and EG3D.  The main advantage of these methods is that they can learn 3D representation from a large collection of 2D images. However, it seems that this paper has retrained them from scratch using only the multi-view images from the RenderPeople dataset `(please correct if my understanding is wrong)`. Hence, their results are significantly worse than what is shown in the original paper, in particular the results of EVA3D. I find this comparison unfair for these methods since it suppresses their advantages and conceals the limitations of the proposed method. For a fair comparison, these methods should be fine-tuned on the RenderPeople dataset rather than training from scratch. It is the limitation of the proposed method that it requires 3D ground truths for training and hence can only be trained on multiview data. Other methods, on the other hand, can use a large collection of 2D images hence they should be used by default for these methods. \n\n### Blurry results\n- The qualitative results are quite blurry. Would be great if the authors can comment on this during the rebuttal. \n\n### Missing references\n[a] Implicit Neural Representations with Structured Latent Codes, CVPR'21 \n\n### Conclusion\nOverall, it is a good paper with a reasonable approach. However, the experiments section of the paper is a bit weak, in particular, the comparison with existing methods is unfair. Hence, my current rating for the paper is `Weak Reject`. I would be happy to upgrade my rating if the authors can address my concerns during the rebuttal.\n\nQUESTIONS:\n### Missing Experiments \n- The paper learns 3D primitives using reconstruction losses and multiview images. Since the ground-truth meshes are available for the RenderPeople dataset, an interesting comparison would be to use them to obtain volumetric primitives. It will help us understand how much information is lost during the learning of volumetric primitives using multi-view images. \n- I would have liked to see a baseline that is similar to NeuralBody i.e., replace volumetric primitives with learnable latent codes. \n- Please explain how the baseline methods are trained.\n\nLIMITATIONS:\n- As mentioned above, reliance on 3D ground-truths is a significant limitation of the proposed method as compared to existing methods. Hence, it should be discussed in the `Limitations`.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework called PrimDiffusion, which utilizes diffusion models to generate 3D human models. The framework leverages a Mixture of Volumetric Primitives (MVP) to represent 3D humans, enabling efficient rendering and incorporating human priors for retargeting. This representation offers a compact and expressive 2D parameter space, making it well-suited for the diffusion operation. Experimental results demonstrate that the proposed model outperforms previous approaches in terms of both quality and efficiency. Furthermore, the proposed method exhibits flexibility in training-free conditional generation tasks, including texture transfer and 3D inpainting.\n\nSTRENGTHS:\nThe paper clearly proposes to represent 3D humans using volumetric primitives, effectively transforming the 3D representation into a 2D parameter space that is well-suited for 2D diffusion. This method outperforms previous approaches for 3D human face/body generation, which mostly rely on triplane representation. Further, volumetric primitives technique also provides easier animation and conditional generation capabilities.\n\nThrough experiments, the proposed method demonstrates superior qualitative and quantitative performance compared to previous works. These experiments highlight the advantages of using volumetric primitives representation for 3D human generation, specifically, the paper compared it with voxel-based representations.\n\nWEAKNESSES:\nIn lines 139-140, the proposed method only considers color, density, and delta scale factor as optimized parameters, while delta translation and delta rotation are not included. It would be beneficial to understand the reasons behind this omission, especially considering that the original MVP paper optimized these parameters.\n\nFigure 2 illustrates the diffusion process of rotation (R), translation (T), and scale (s) during training, which appears to contradict the claim that the learnable parameters are solely color, density, and delta scale factor. And it seems that there is a discrepancy, as the expected parameters should be delta rotation, delta translation, and delta scale. Clarification on this point would be appreciated.\n\nThe motion encoder in the proposed method solely considers joint rotations, which may not be sufficient for accurate fitting of 3D humans. Shape parameters also play a crucial role in this process. It would be helpful to hear an explanation for the decision to exclude shape parameters from the motion encoder.\n\nThere are typos in Function (7), which are inconsistent with the function presented in the DDPM paper [16]. It is important to address these inconsistencies and ensure the correctness and coherence of the equations.\n\nThe loss function (8) differs from the one described in the DDPM paper [16]. The original function aimed to estimate noise from V_t instead of directly estimating the ground truth. It would be beneficial to provide an explanation for this modification and discuss its implications.\n\nQUESTIONS:\nThe paper conducted a comparison between volumetric primitives and voxel representations in diffusion-based 3D human generation. However, I am also interested in understanding how volumetric primitives compare to triplane representations in diffusion. It would greatly enhance the experimental analysis if this comparison were included.\n\nRegarding the ablation study, the paper compared predefined rotation and translation values derived from a template human mesh with identity-specific ones. However, the paper does not provide a detailed explanation of the identity-specific rotation and translation. It would be beneficial to include a thorough explanation of this approach to improve the understanding of the study.\n\nLIMITATIONS:\nThe proposed method is innovative as it combines volumetric primitives with diffusion model. However, there is room for improvement in terms of the writing. There is still a noticeable disparity between the ground truth and the fitting results. It seems that there is significant room for improvement in stage 1 of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}