Title: XAGen: 3D Expressive Human Avatars Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Human Avatar

3D-aware GAN
Abstract: Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available at https://showlab.github.io/xagen.

XAGen: 3D Expressive Human Avatars Generation

Abstract

Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available at https://showlab.github.io/xagen.

1 Introduction

3D avatars present an opportunity to create experiences that are exceptionally authentic and immersive in telepresence [10], augmented reality (AR) [22], and virtual reality (VR) [50]. These applications [1, 52, 3, 35] require the capture of human expressiveness, including poses, gestures, expressions, and others, to enable photo-realistic generation [65, 70], animation [56], and interaction [33] in virtual environments.

Traditional methods [11, 60, 4, 20, 23] typically create virtual avatars based on template registration or expensive multi-camera light stages in well-controlled environments. Recent efforts [69, 43, 5, 26, 16] have explored the use of generative models to produce 3D human bodies and clothing based on input parameters, such as SMPL [38], without the need of 3D supervision. Despite these advancements, current approaches are limited in their ability to handle expressive attributes of the human body, such

Multi-view Consistent Appearance and Geometry

as facial expressions and hand poses, as they primarily focus on body pose and shape conditions. Yet, there exist scenarios where fine-grained control ability is strongly desired, e.g. , performing social interactions with non-verbal body languages in Metaverse, or driving digital characters to talk with various expressions and gestures, etc. Due to the lack of comprehensive modeling of the full human body, existing approaches [43, 5, 26] fail to provide control ability beyond the sparse joints of major body skeleton, leading to simple and unnatural animation.

In this work, our objective is to enhance the fine-grained control capabilities of GAN-based human avatar generation model. To achieve this, we introduce the first e X pressive 3D human A vatar Gen eration model ( XAGen ) that can (1) synthesize high-quality 3D human avatars with diverse realistic appearances and detailed geometries; (2) provide independent control capabilities for finegrained attributes, including body poses, hand poses, jaw poses, shapes, and facial expressions.

XAGen is built upon recent unconditional 3D-aware generation models for static images [7, 44]. One straightforward approach to implement fully animatable avatar generation is extending 3D GAN models to condition on expressive control signals, such as SMPL-X [47]. Though conceptually simple, such a direct modification of conditioning signal cannot guarantee promising appearance quality and control ability, particularly for two crucial yet challenging regions, i.e. , the face and hands. This is because (1) Compared with body, face and hands contain similar or even more articulations. In addition, their scales are much smaller than arms, torso, and legs in a human body image, which hinders the gradient propagation from supervision. (2) Face and hands are entangled with the articulated human body and thus will be severely affected by large body pose deformation, leading to optimization difficulty when training solely on full-body image collections.

To address the above challenges, we decompose the learning process of body, face, and hands by adopting a multi-scale and multi-part 3D representation and rendering multiple parts independently using their respective observation viewpoints and control parameters. The rendered images are passed to multi-part discriminators, which provide multi-scale supervision during the training process.

With these careful designs, XAGen can synthesize photo-realistic 3D human avatars that can be animated effectively by manipulating the corresponding control parameters for expressions and poses, as depicted in Figure 1. We conduct extensive experiments on a variety of benchmarks [18, 68, 14, 36], demonstrating the superiority of XAGen over state-of-the-arts in terms of appearance, geometry, and controllability. Moreover, XAGen supports various downstream applications such as text-guided avatar creation and audio-driven animation, expanding its potential for practical scenarios.

Our contributions are three-fold: (1) To the best of our knowledge, XAGen is the first 3D GAN model for fully animatable human avatar generation. (2) We propose a novel framework that incorporates

multi-scale and multi-part 3D representation together with multi-part rendering technique to enhance the quality and control ability, particularly for the face and hands. (3) Experiments demonstrate XAGen surpasses state-of-the-art methods in terms of both quality and controllability, which enables various downstream applications, including text-guided avatar synthesis and audio-driven animation.

2 Related work

Generative models for avatar creation. Generative models [27, 28, 51] have demonstrated unprecedented capability for synthesizing high-resolution photo-realistic images. Building upon these generative models, follow-up works [7, 44, 55, 59, 63] have focused on extending 2D image generation to the 3D domain by incorporating neural radiance field [42] or differentiable rasterization [29]. Although enabling 3D-aware generation, these works fail to provide control ability to manipulate the synthesized portrait images. To address this limitation, recent research efforts [64, 26, 43, 69, 57, 16, 71] have explored animatable 3D avatar generation leveraging parametric models for face [32] and body [38]. These works employ inverse [31] or forward [9] skinning techniques to control the facial attributes or body poses of the generated canonical avatars [69, 71]. For human body avatars, additional challenges arise due to their articulation properties. Consequently, generative models for human avatars have explored effective 3D representation designs. Among them, ENARF [43] divides an efficient 3D representation [7] into multiple parts, with each part representing one bone. EVA3D [26] employs a similar multi-part design by developing a compositional neural radiance field. Despite enabling body control, such representation fails to generate the details of human faces or hands since these parts only occupy small regions in the human body images.

Our method differs in two aspects. First, existing works can either control face or body, whereas ours is the first 3D avatar generation model with simultaneous fine-grained control over the face, body, and hands. Second, we devise a multi-scale and multi-part 3D representation, allowing for generating human body with high fidelity even for small regions like face and hands.

Expressive 3D human modeling. Existing 3D human reconstruction approaches can be categorized into two main categories depending on whether explicit or implicit representations are used. Explicit representations mainly utilize the pre-defined mesh topology, such as statistical parametric models [38, 62, 45, 2] or personalized mesh registrations [19, 12], to model naked human bodies with various poses and shapes. To enhance the expressiveness, recent works have developed expressive statistical models capable of representing details beyond major human body [47, 46, 17] or introduced the surface deformation to capture fine-grained features [30, 58]. On the other hand, leveraging the remarkable advances in implicit neural representations [41, 42], another line of research has proposed to either rely purely on implicit representations [53] or combine it with statistical models [61, 48, 8] to reconstruct expressive 3D human bodies. The most recent work [15, 54] proposed to learn a single full-body avatar from multi-part portrait videos or 3D scans. In contrast, our approach focuses on developing 3D generative model for fully animatable human avatars, which is trainable on only unstructured 2D image collections.

3 Method

In this section, we introduce XAGen, a 3D generative model for synthesizing photo-realistic human avatars with expressive and disentangled controllability over facial expression, shape, jaw pose, body pose, and hand pose. Figure 2 depicts the pipeline of our method.

Given a random noise z sampled from Gaussian distribution, XAGen first synthesizes a human avatar with canonical body, face, and hand configurations. In this work, we use X-pose [34] and neutral shape, face, and hand as canonical configurations. We leverage Tri-plane [7] as the fundamental building block of 3D representation in our canonical generator. To increase the capability of 3D representation for the smaller-scale face and hands, we introduce multi-part and multi-scale designs into the canonical Tri-plane (Sec. 3.1). A mapping network first encodes z and the camera viewpoint of body c b into latent code w . The canonical generator then synthesizes three Tri-planes F k conditioned on w , where k ∈ { b , f , h } which stands for { body , face , hand } .

Based on the generated canonical avatar, we deform it from canonical space to observation space under the guidance of control signal p b parameterized by an expressive statistical full body model, i.e. , SMPL-X [47]. We adopt volumetric rendering [39] to synthesize the full body image. However, due

z

Mapping

Network

Face

Disc

Body

Disc

Hand

Disc

c

b

p

b

c

f

p

f

c

h

p

h

Super

Resolution

Multi-Part Rendering

c

h

p

h

c

f

p

f

c

h

p

h

c

f

p

f

c

b

p

b

Real

or

Fake

Real

or

Fake

Real

or

Fake

w

w

Generator

c

b

p

b

IS

IS

IS

Canonical Generator

Discriminators

{

c

b

, c

f

, c

h

}

Camera Poses

Control Params

{

p

b

, p

f

, p

h

}

F

b

F

f

F

h

F

b

F

f

F

h

F

b

F

f

F

h

to the scale imbalance between the face/hands and body, rendering only the full body image cannot guarantee quality for these detailed regions. To address this issue, we propose a multi-part rendering technique (Sec. 3.2). Specifically, we employ part-aware deformation and rendering based on the control parameters ( p f and p h) and cameras ( c f and c h). Accordingly, to ensure the plausibility and controllability of the generated avatars, we develop multi-part discriminators to critique the rendered images (Sec. 3.3).

3.1 Multi-scale and Multi-part Representation

XAGen is designed for expressive human avatars with an emphasis on the high-quality face and hands. However, the scale imbalance between face/hands and body may hamper the fidelity of the corresponding regions. To address this issue, we propose a simple yet effective multi-scale and multi-part representation for expressive human avatar generation. Our multi-scale representation builds upon the efficient 3D representation, i.e. , Tri-plane [7], which stores the generated features on three orthogonal planes. Specifically, we design three Tri-planes for body, face, and hands, denoted as F b ∈ R W b × W b × 3 C , F f ∈ R W f × W f × 3 C , and F h ∈ R W h × W h × 3 C , respectively. The size of the face and hand Tri-planes is set to half of the body Tri-plane, with W f = W h = W b / 2 .

As depicted in Figure 2, our canonical generator first synthesizes a compact feature map F ∈ R W b × W b × 9 C/ 2 , where C represents the number of channels. We then separate and reshape F into F k , where k ∈ { b , f , h } , representing the canonical space of the generated human avatar. Furthermore, to save computation cost, we exploit the symmetry property of hands to represent both left and right hands using one single F h through a horizontal flip operation (refer to Appendix for details).

3.2 Multi-part Rendering

Our method is trainable on unstructured 2D human images. Although this largely reduces the difficulty and cost to obtain data, the training is highly under-constrained due to the presence of diverse poses, faces, and clothes. To facilitate the training process and improve the appearance quality, we propose a multi-part rendering strategy. This strategy allows XAGen to learn each part based on the independent camera poses, which further enhances the geometry quality of the face and hands. Specifically, for each training image, we utilize a pretrained model [17] to estimate SMPL-X parameters { p b , p f , p h } and camera poses { c b , c f , c h } for body, face, and hands, respectively. In the rendering stage, we shoot rays using { c b , c f , c h } and sample points { x b o , x f o , x h o } along the rays in the observation space. To compute the feature for each point, we employ inverse linear-blend skinning [31], which finds the transformation of each point from observation space to canonical space produced by the canonical generator. Based on the parameter p k , where k ∈ { b , f , h } , SMPL-X yields an expressive human body model ( v , w ) , where v ∈ R N × 3 represents N vertices, and w ∈ R N × J represents the skinning

weights of each vertex with respect to joint J . For each point x k,i o , where i = 1 · · · M k and M k is the number of sampled points, we find its nearest neighbour n from vertices v . We then compute the corresponding transformation from observation space to canonical space

where j = 1 · · · J , R j and t j are derived from p k with Rodrigues formula [6], and ∆ n represents the offset caused by pose and shape for vertex n , which is calculated by SMPL-X. Based on this inverse transformation, we can calculate the coordinates for each point in canonical space x k,i c as

where we apply homogeneous coordinates for the calculation.

For the face and hands rendering, i.e. , k ∈ { f , h } , we directly interpolate their corresponding Triplane F f and F h to compute the feature f f ,i c and f h ,i c . Regarding the body rendering, we first define three bounding boxes B f , B lh , B rh for face, left and right hands in canonical body space. Then, we query canonical body points that are outside these bounding boxes from body Tri-plane F b, while the canonical points inside these boxes from F f and F h. The query process for body point x b ,i c is mathematically formulated as

where Q denotes querying the feature for the given point from the corresponding Tri-planes.

Once the features f k,i c are obtained, they are encoded into color c and geometry d via two lightweight multi-layer perceptrons (MLP), where c = MLPc ( f k,i c ) . Inspired by prior works [44, 26, 69], we employ signed distance field (SDF) as a proxy to model geometry. Additionally, following [26, 69], we also query a base SDF d c in the canonical space, and predict delta SDF, such that d = d c + MLP d ( f k,i c , d c ) . We then convert the SDF value into density σ = 1 α Sigmoid ( -d α ) for volume rendering, where α is a learnable parameter.

To handle the body features queried from multiple Tri-planes, we apply feature composition on RGB and density using a window function [37] for smoothness transition. Specifically, if point x k,i c , b is located in the overlapping region between the body and other parts (face, right hand, and left hand), their features are sampled from both Tri-planes and linearly blended together. More details on the feature composition can be found in the Appendix. Finally, volume rendering is applied to synthesize raw images for body, face, and hands, denoted as { I raw b , I raw f , I raw h } . These raw images are then upsampled into high-resolution images { I b , I f , I h } by a super-resolution module.

3.3 Multi-part Discriminators

Based on the images synthesized by XAGen generator, we design a discriminator module to critique the generation results. To ensure both the fine-grained fidelity of appearance and geometry as well as disentangled control over the full body, including face and hands, we introduce multi-part discriminators to encode images { I b , I f , I h } into real-fake scores for adversarial training. As depicted in Figure 2, these discriminators are conditioned on the respective camera poses to encode 3D priors, resulting in improved geometries as demonstrated in our experiments. To enhance the control ability of the face and hands, we further condition face discriminator on expression and shape parameters [ p ψ f , p β f ] , and condition hand discriminator on hand pose p θ h . Weencode the camera pose and condition parameters into intermediate embeddings by two separate MLPs and pass them to the discriminators. The multi-part discriminator is formulated as

Here s k denotes the probability of each image I k being sampled from real data, and D k refers to the discriminator corresponding to the specific body part k . For body part, no conditioning parameters are used because we empirically find that the condition for body hinders the learning of appearance.

3.4 Training Losses

The non-saturating GAN loss [21] is computed for each discriminator, resulting in L b, L f, and L h. We also regularize these discriminators using R1 regularization loss [40] L R1. To improve the plausibility and smoothness of geometry, we compute minimal surface loss L Minsurf, Eikonal loss L Eik, and human prior regularization loss L Prior as suggested in previous works [44, 69].

Due to the occlusion in the full body images, some training samples may not contain visible faces or hands. Thus, we balance the loss terms for both generator and discriminator based on the visibility of face M f and hands M h, which denote whether face and hands are detected or not. The overall loss term of XAGen is formulated as

where ⊙ means instance-wise multiplication, and λ ∗ are the weighting factors for each term.

4 Experiments

Weevaluate the performance of XAGen on four datasets, i.e. , DeepFashion [36], MPV [68], UBC [14], and SHHQ [18]. These datasets contain diverse full body images of clothed individuals. For each image in the dataset, we process it to obtain aligned body, face and hand crops, and their corresponding camera poses and SMPL-X parameters. Please refer to Appendix for more details.

4.1 Comparisons

Baselines. We compare XAGen with four state-of-the-art 3D GAN models for animatable human image generation: ENARF [43], EVA3D [26], AvatarGen [69], and AG3D [16]. All these methods utilize 3D human priors to enable the controllability of body pose. ENARF conditions on sparse skeletons, while others condition on SMPL [38] model. Additionally, AvatarGen and AG3D incorporate an extra face discriminator to enhance face quality. We adopt the official implementations of ENARF and EVA3D, and cite results from AG3D directly. As for AvatarGen, it is reproduced and conditioned on SMPL-X to align with the setup of our model.

Quantitative comparisons. The fidelity of synthesized image is measured by Frechet Inception Distance (FID) [24] computed between 50 K generated images and all the available real images in each dataset. To study the appearance quality for face and hands, we further crop face (resolution 64 2 ) and hands (resolution 48 2 ) regions from the generated and real images to compute FIDf and FIDh. To evaluate pose control ability, we compute Percentage of Correct Keypoints (PCK) between 5 K real images and images generated using the same pose condition parameters of real images under a distance threshold of 0.1. To evaluate this ability in face and hand regions, we also report PCKf and PCKh. Another critical evaluation for a fully controllable generative model is the disentangled control of fine-grained attributes. Inspired by previous works [13, 64], we select one attribute from { expression , shape , jaw pose , body pose , hand pose } , and modify the selected attribute while keeping others fixed for each synthesis. We then estimate the SMPL-X parameters for 1 K generated images using a pre-trained 3D human reconstruction model [17] and compute the Mean Square Error (MSE) for the selected attribute between the input and estimated parameters.

Table 1 summarizes the results for appearance quality and pose control ability for body, face, and hands. It demonstrates that XAGen outperforms existing methods w.r.t. all the evaluation metrics, indicating its superior performance in generating controllable photo-realistic human images with high-quality face and hands. Notably, XAGen shows significant improvements over the most recent method AG3D, achieving more than 20% improvement in FID and FIDf on both DeepFashion and UBC datasets. Additionally, XAGen achieves state-of-the-art pose control ability, with substantial performance boost in PCKf, e.g. , a relative improvement of 40.90% on MPV dataset against baseline.

Table 2 presents the results for the disentangled control ability of XAGen compared to the baseline methods. It is worth noting that ENARF and EVA3D are not fully controllable, but we still report all the evaluation metrics for these two methods to show the controllability lower bound. Notably, the generated images of ENARF are blurry. Thus, our pose estimator cannot estimate precise jaw poses, which leads to an outlier on UBC jaw pose. In general, XAGen demonstrates state-of-the-art

performance for fine-grained controls, particularly in expression, jaw, and hand pose, improving upon baseline by 38.29%, 25.93%, and 33.87% respectively on SHHQ dataset which contains diverse facial expressions and hand gestures. These results highlight the effectiveness of XAGen in enabling disentangled control over specific attributes of the generated human avatar images.

Qualitative comparisons. Figure 3 provides qualitative comparisons between XAGen and baselines. From the results, we observe that ENARF struggles to produce reasonable geometry or realistic images due to the limitations of low training resolution. While EV A3D and AvatarGen achieve higher quality, they still fail to synthesize high-fidelity appearance and geometry for the face and hands. In contrast, XAGen demonstrates superior performance with detailed geometries for face and hands regions, resulting in more visually appealing human avatar images. The improvement of XAGen against baseline models is also confirmed by the perceptual user study, which is summarized in Table 3. Notably, XAGen achieves the best perceptual preference scores for both image appearance ( ≥ 57 . 2% ) and geometry ( ≥ 48 . 3% ) on all the benchmark datasets.

Figure 4 showcases qualitative results for fine-grained control ability. We first observe that ENARF fails to generate a correct arm for the given body pose. Although EVA3D demonstrates a better pose condition ability, its shape conditioning ability is limited and the generated face suffers from unrealistic scaling. On the other hand, AvatarGen shows comparable results for pose and shape control. However, when it comes to expression, jaw pose, and hand pose controls, ours significantly

ENARF

AvatarGen

Ours

outperforms AvatarGen, e.g. , AvatarGen produces distortion in mouth region and blurred fingers while XAGen demonstrates natural faces and correct hand poses.

4.2 Ablation studies

To verify the design choices in our method, we conduct ablation studies on SHHQ dataset, which contains diverse appearances, i.e. , various human body, face, and hand poses as well as clothes.

Representation. XAGen adopts a multi-scale and multi-part representation to improve the quality for face and hands regions. We study the necessity of this design by removing Tri-planes for face and hands. Table 4a provides the results, indicating that using only a single full-body Tri-plane (without any specific Tri-planes for face or hands) results in a significant degradation in appearance quality. Adding either face or hand Tri-plane can alleviate this issue and all the FID metrics drop slightly. The best results are achieved when both face and hand Tri-planes are enabled, demonstrating the importance of our multi-scale and multi-part representation.

Multi-part rendering. In our model, we render multiple parts independently in the forward process to disentangle the learning of body, face, and hands. Table 4b demonstrates that independent rendering for face is crucial, as it significantly improves both fidelity (FIDf: 20.63 vs. 10.06) and control ability (Exp: 6.58 vs. 5.56, Jaw: 7.26 vs. 6.57) for face. Similarly, without rendering for hand, FIDh increases from 18.85 to 25.94, and MSE increases from 3.28 to 4.55 (Table 4c). The effectiveness of multi-part rendering is further supported by the qualitative results shown in Figure 5. Without independent rendering, the geometry quality degrades. For example, the eyes and mouth are collapsed without face rendering, and the model also fails to synthesize geometric details for hand when hand rendering is disabled. These highlight the importance of multi-part rendering in facilitating the learning of 3D geometries for different body parts.

Discriminators. To study the effect of multi-part discriminators, we disable each of them during training. As shown in Table 4b, without face discriminator, the overall appearance quality deteriorates.

ENARF

AvatarGen

Ours

Shape

AvatarGen

Ours

Expression

AvatarGen

Ours

AvatarGen

Ours

Hand Pose

AvatarGen

Ours

Repr.

FID

↓

w/o both 11.50

w/ face

11.27

w/ hand

9.64

FIDf

↓

12.57

11.95

11.61

FIDh

↓

20.97

20.10

19.92

w/ both

5.88

10.06

19.23

(a) The effect of multi-scale and multi-part representations.

(b) The effect of face rendering and face discriminator.

Hand

FID

↓

w/o Rend 14.28

7.78

FIDh

↓

26.66

16.74

Hand

↓

4.51

4.46

w/ both

5.88

19.23

3.33

(c) The effect of hand rendering and hand discriminator.

Despite the slight improvement in face appearance, there is a drop in the control ability, as evidenced by the increase in the MSE values for expression and jaw pose. A similar observation can be made for hand discriminator in Table 4c. Furthermore, the qualitative results shown in Figure 5 provide visual evidence of the impact of the face and hand discriminators on the 3D geometries. When they are removed, the geometries for face and hand collapse.

4.3 Applications

Text-guided avatar synthesis. Inspired by recent works [25, 69, 67] on text-guided avatar generation, we leverage a pretrained vision-language encoder CLIP [49] to guide the generation process using the given text prompt. The text-guided avatar generation process involves randomly sampling a latent code z and a control parameter p b from the dataset, and optimizing z by maximizing the CLIP similarities between the synthesized image and text prompt. As shown in Figure 6a, the generated human avatars exhibit the text-specified attributes, i.e. , hair and clothes adhere to the given text prompt ( e.g. , brown hair and red T-shirt). The generated avatar can be re-targeted by novel SMPL-X parameters, allowing for additional control and customization of the synthesis.

Audio-driven animation. The ability of XAGen to generate fully animatable human avatars with fine-grained control (Figure 1) opens up possibilities for audio-driven animation. The 3D avatars can be driven by arbitrary SMPL-X motion sequences generated by recent works such as [66] given audio inputs. Specifically, we sample an audio stream and SMPL-X sequence from TalkSHOW [66] and use it to animate the generated avatars. As shown in Figure 6b, XAGen is able to synthesize temporally consistent video animations where the jaw poses of the avatars are synchronized with the audio stream (highlighted in red box). Additionally, the generated avatars are generalizable given novel body poses and hand gestures, allowing diverse and expressive animations.

5 Limitations

Although XAGen is able to synthesize photo-realistic and fully animatable human avatars, there are still areas where improvements can be made: (1) XAGen relies on pre-estimated SMPL-X parameters, the inaccurate SMPL-X may introduce potential errors into our model, which can lead to artifacts

w/o Face Disc

w/ both

w/o Face Rend

w/o Hand Rend

D

brown hair woman, red T-shirt, blue jeans

blonde hair woman, pink T-shirt, black trousers

… Saudi Arabia's first female athlete …

t

(a) Text-guided avatar synthesis.

(b) Audio-driven avatar animation.

and degraded body images. Please refer to Sup. Mat. for the experimental analysis of this issue. We believe our method can benefit from a more accurate SMPL-X estimation method or corrective operations. (2) SMPL-X only represents naked body. Thus, methods built upon SMPL-X could struggle with modeling loose clothing, which is a long-standing challenge for 3D human modeling. We believe an advanced human body prior or independent clothing modeling approach is helpful to alleviate this issue. (3) Face and hand images in existing human body datasets lack diversity and sharpness, which affects the fidelity of our generation results, particularly for the novel hand poses that are out-of-distribution. A more diverse dataset with high-quality face and hand images could help tackle this problem. (4) XAGen utilizes inverse blend skinning to deform the points from canonical space to the observation space. However, this process could introduce errors, particularly when computing nearest neighbors for query points located in the connection or interaction regions. Thus, exploring more robust and accurate techniques, such as forward skinning [9], could open up new directions for future work.

6 Conclusion

This work introduces XAGen, a novel 3D avatar generation framework that offers expressive control over facial expression, shape, body pose, jaw pose, and hand pose. Through the use of multi-scale and multi-part representation, XAGen can model details for small-scale regions like faces and hands. By adopting multi-part rendering, XAGen disentangles the learning process and produces realistic details for appearance and geometry. With multi-part discriminators, our model is capable of synthesizing high-quality human avatars with disentangled fine-grained control ability. The capabilities of XAGen open up a range of possibilities for downstream applications, such as text-guided avatar synthesis and audio-driven animation.

References

O. Alexander, M. Rogers, W. Lambeth, J.-Y. Chiang, W.-C. Ma, C.-C. Wang, and P. Debevec. The digital emily project: Achieving a photorealistic digital actor. IEEE Computer Graphics and Applications , 2010.

D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH , 2005.

T. Bagautdinov, C. Wu, T. Simon, F. Prada, T. Shiratori, S.-E. Wei, W. Xu, Y. Sheikh, and J. Saragih. Driving-signal aware full-body avatars. ACM Trans. on Graphics , 2021.

T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross. High-quality passive facial performance capture using anchor frames. In ACM SIGGRAPH , 2011.

A. Bergman, P. Kellnhofer, W. Yifan, E. Chan, D. Lindell, and G. Wetzstein. Generative neural articulated radiance fields. In NeurIPS , 2022.

C. Bregler, J. Malik, and K. Pullen. Twist based acquisition and tracking of animal and human kinematics. Int'l. J. Computer Vision , 2004.

E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR , 2022.

M. Chen, J. Zhang, X. Xu, L. Liu, Y. Cai, J. Feng, and S. Yan. Geometry-guided progressive nerf for generalizable and efficient neural human rendering. In ECCV , 2022.

X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In CVPR , 2021.

H. Chu, S. Ma, F. De la Torre, S. Fidler, and Y. Sheikh. Expressive telepresence via modular codec avatars. In ECCV , 2020.

A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. ACM Trans. on Graphics , 2015.

E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun. Performance capture from sparse multi-view video. In ACM SIGGRAPH , 2008.

Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In CVPR , 2020.

H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin. Towards multi-pose guided virtual try-on network. In CVPR , 2019.

J. Dong, Q. Fang, Y. Guo, S. Peng, Q. Shuai, X. Zhou, and H. Bao. Totalselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies. In NeurIPS , 2022.

Z. Dong, X. Chen, J. Yang, M. J. Black, O. Hilliges, and A. Geiger. Ag3d: Learning to generate 3d avatars from 2d image collections. In ICCV , 2023.

Y. Feng, V. Choutas, T. Bolkart, D. Tzionas, and M. J. Black. Collaborative regression of expressive bodies using moderation. In 3DV , 2021.

J. Fu, S. Li, Y . Jiang, K.-Y . Lin, C. Qian, C. C. Loy, W. Wu, and Z. Liu. Stylegan-human: A data-centric odyssey of human generation. In ECCV , 2022.

J. Gall, C. Stoll, E. De Aguiar, C. Theobalt, B. Rosenhahn, and H.-P. Seidel. Motion capture using joint skeleton tracking and surface estimation. In CVPR , 2009.

A. Ghosh, G. Fyffe, B. Tunwattanapong, J. Busch, X. Yu, and P. Debevec. Multiview face capture using polarized spherical gradient illumination. ACM Trans. on Graphics , 2011.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 2020.

K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR , 2022.

K. Guo, P. Lincoln, P. Davidson, J. Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-Escolano, R. Pandey, J. Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Trans. on Graphics , 2019.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 2017.

F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Trans. on Graphics , 2022.

F. Hong, Z. Chen, Y. LAN, L. Pan, and Z. Liu. EVA3d: Compositional 3d human generation from 2d image collections. In ICLR , 2023.

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In CVPR , 2019.

T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In NeurIPS , 2021.

H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In CVPR , 2018.

N. Kolotouros, G. Pavlakos, and K. Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. In CVPR , 2019.

J. P. Lewis, M. Cordner, and N. Fong. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In CGIT , 2000.

T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero. Learning a model of facial shape and expression from 4D scans. ACM Trans. on Graphics , 2017.

J.-W. Liu, Y.-P. Cao, T. Yang, Z. Xu, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou. Hosnerf: Dynamic human-object-scene neural radiance fields from a single video. In ICCV , 2023.

L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. on Graphics , 2021.

T. Liu, J. Zhang, X. Nie, Y . Wei, S. Wei, Y . Zhao, and J. Feng. Spatial-aware texture transformer for high-fidelity garment transfer. IEEE Transactions on Image Processing , 2021.

Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR , 2016.

S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. on Graphics , 2021.

M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: A skinned multiperson linear model. ACM Trans. on Graphics , 2015.

N. Max. Optical models for direct volume rendering. IEEE Trans. on Visualization and Computer Graphics , 1995.

L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In ICLR , 2018.

L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR , 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

A. Noguchi, X. Sun, S. Lin, and T. Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. In ECCV , 2022.

R. Or-El, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Kemelmacher-Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR , 2022.

A. A. Osman, T. Bolkart, and M. J. Black. Star: Sparse trained articulated human body regressor. In ECCV , 2020.

A. A. Osman, T. Bolkart, D. Tzionas, and M. J. Black. Supr: A sparse unified part-based human representation. In 3DV , 2022.

G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR , 2019.

S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , 2021.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

E. Remelli, T. Bagautdinov, S. Saito, C. Wu, T. Simon, S.-E. Wei, K. Guo, Z. Cao, F. Prada, J. Saragih, et al. Drivable volumetric avatars using texel-aligned features. In ACM SIGGRAPH , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

J. Romero, D. Tzionas, and M. J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Trans. on Graphics , 2017.

S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV , 2019.

K. Shen, C. Guo, M. Kaufmann, J. J. Zarate, J. Valentin, J. Song, and O. Hilliges. X-avatar: Expressive human avatars. In CVPR , 2023.

Y. Shi, D. Aggarwal, and A. K. Jain. Lifting 2d stylegan for 3d-aware face generation. In CVPR , 2021.

A. Siarohin, O. J. Woodford, J. Ren, M. Chai, and S. Tulyakov. Motion representations for articulated animation. In CVPR , 2021.

J. Sun, X. Wang, L. Wang, X. Li, Y. Zhang, H. Zhang, and Y. Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In CVPR , 2023.

S. Tang, F. Tan, K. Cheng, Z. Li, S. Zhu, and P. Tan. A neural network for detailed human depth estimation from a single image. In ICCV , 2019.

T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR , 2023.

D. Xiang, F. Prada, T. Bagautdinov, W. Xu, Y. Dong, H. Wen, J. Hodgins, and C. Wu. Modeling clothing as a separate layer for an animatable human avatar. ACM Trans. on Graphics , 2021.

Y. Xiu, J. Yang, D. Tzionas, and M. J. Black. Icon: implicit clothed humans obtained from normals. In CVPR , 2022.

H. Xu, E. G. Bazavan, A. Zanfir, W. T. Freeman, R. Sukthankar, and C. Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR , 2020.

H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR , 2023.

H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR , 2023.

Z. Xu, J. Zhang, J. Liew, W. Zhang, S. Bai, J. Feng, and M. Z. Shou. Pv3d: A 3d generative model for portrait video generation. In ICLR , 2023.

H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black. Generating holistic 3d human motion from speech. In CVPR , 2023.

K. Youwang, K. Ji-Yeon, and T.-H. Oh. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In ECCV , 2022.

P. Zablotskaia, A. Siarohin, B. Zhao, and L. Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. In BMVC , 2019.

J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng. Avatargen: a 3d generative model for animatable human avatars. In ECCV Workshop , 2023.

J. Zhang, H. Yan, Z. Xu, J. Feng, and J. H. Liew. Magicavatar: Multimodal avatar generation and animation. arXiv , 2023.

X. Zhang, J. Zhang, C. Rohan, H. Xu, G. Song, Y. Yang, and J. Feng. Getavatar: Generative textured meshes for animatable human avatars. In ICCV , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generative model for 3D expressive human avatars. Based on the backbone of the recent works, tri-plane 3D feature representation, volumetric rendering, transformation from pose space to canonical space, parametric body models and GAN, the authors propose to improve the expressive of the model by representing the hand and face with extra tri-planes and supervising them with extra adversarial loss. The result achieves state of the art performance according to various evaluation metrics on various dataset. Moreover, one can animate the hand and jaw of the generated avatar, surpassing most SOTA approaches. Two practical applications of such a generative model are demonstrated in the paper.\n\nSTRENGTHS:\n1. The paper presents a state of the art generative model for 3D clothed human avatar with expressive control.\n\n2. The paper is solid, well written and easy to follow.\n\n3. The evaluation is extensive and the results are convincing.\n\n4. Two practical applications are demonstrated.\n\nWEAKNESSES:\n1. Lack of technical novelty, multi-tri-plane and part focused supervision are quite weak when claimed as novelty. But I'm relatively ok with this since the performance and the control of the model surpass the SOTA. Although in my opinion, this paper fits much more for SIGGRAPH/Asia, where technical novelty is less expected compared to NeurIPS, I believe the overall quality of the submission clearly reaches the acceptance standard for NeurIPS.\n\n2. While I appreciate the effort on making consistent and good format of references (I notice the authors use consistent form of names of the same conference across the reference, which is rare nowadays in most of the submissions), there is still room to improve the reference, such as \na. Capitalization of words. For example: 3d -> 3D, Icon: implicit... -> ICON: Implicit..., Avartargen: a 3d... -> AvartarGen: A 3D..., \nb. the URL in the references is not necessary.\nc. [65] In ECCVw, 2023 -> In ECCV Workshops, 2022\n\n3. In the supplementary video, \na. the voice over contains environment noise and echoes. It would be great to remove them.\nb. the bottom left animation has a white background, occluding the slide contents. It could be better if those animations were rendered with transparent background.\nc. the demonstration of controlling jaw motion and hand poses are a bit long and not eye catching. It would be more appealing if they can be shortened a bit and shown with a close-up / zoom-in view.\n\n4. On the project page, the RGB video and the geometry visualization are asynchronized regardless of refreshing the page. \n\n5. Suggestion. Please consider incorporating some concurrent work into related work for the updated version to make the paper more inclusive, for example: Chupa (https://snuvclab.github.io/chupa/), a generative model for 3D clothed human geometry; and SCARF (https://yfeng95.github.io/scarf/), creating an expressive avatar from monocular video.\n\nQUESTIONS:\nThe submission is in a good state, method clearly stated, experiment is extensive and convincing. I don't have further questions to the authors regarding reviewing this paper.\n\nLIMITATIONS:\nLimitations and societal impact are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for the generation of high quality, articulable 3D avatars of humans. The method proposed builds on top of the 3D GAN framework which has been used to learn to generate 3D articulable human bodies from collections of 2D images of humans. Additionally, the method adopts the proposed high-level methodology of generating bodies in a canonical pose and then using an explicit deformation guided by a parametric body model in order to render images in a desired pose. The novel contribution of the method is unifying this generation architecture not only for bodies, but for faces, bodies, and hands, and demonstrating that generating, articulating, and discriminating these body parts individually results in a higher quality result than handling them all simultaneously. The paper describes the architecture which accomplishes this, and demonstrates that this results in a state-of-the-art generation quality for 3D human avatars. The contributions of modeling and applying losses to these body parts separately are ablated, demonstrating that they are responsible for improvement in the quality of generated avatars.\n\nAFTER REBUTTAL:\n\nI have read the authors' rebuttal. I believe the additional comparisons provided for the hand and jaw quality address my weakness there. As I mentioned, I still have some questions about the novelty of each of the components, but I think the method performs state-of-the-art overall and combining all of these components into a working method is important for the community. Thus, I am leaning positively.\n\nSTRENGTHS:\nIn my opinion, the main strengths of the paper are that:\n1. The paper is well written and is clear to read and follow, and presentation is extremely polished. As a result, I feel like the method will be impactful and those working in the field of generative 3D articulable avatars will want to build on top of it.\n2. The proposed method demonstrates state-of-the-art results. The evaluations are extremely convincing both qualitatively and quantitatively, and compared to the baselines EVA3D and AvatarGen which are currently generating the state-of-the-art, I believe the generated results are significantly better. This is extremely important in pushing towards photorealistic generative 3D human avatars.\n3. The method is ablated well. The main contributions: separately modeling faces and hands, and using a separate discriminator for each of them, are both ablated in a very clear way showing how they improve the performance of the method. This is extremely important to understand that the contributions proposed are actually responsible for the increase in quality, rather than just hyperparameter tuning.\n\nWEAKNESSES:\nIn my opinion, there are two weaknesses of the paper.\n1. The evaluation accuracy of the facial/hand articulation is not entirely convincing. \n    - While Table 2 shows that there are improvements in “Jaw” and “Hand” when a facial and hand pose estimator is applied to the generated results, it is only compared to the modified AvatarGen with SMPL-X (which it outperforms). In order to actually ground these numbers with a baseline, it would be insightful to apply these pose estimators to the results of EVA3D and ENARF. I understand these methods don’t model hand or facial poses, but it would be an important comparison to understand how much improvement is gotten from even modeling these at all.\n    - Quantitative comparisons don’t ever show multiple different identities with modified facial or hand poses (for example, in the supplementary video). Because of this, I don’t know if the examples provided (for example, mouth open), are cherry-picked results or consistent. It would be immensely helpful to show a result where the facial/hand pose is held constant in a deformed state (for example, mouth open), and then the identity latent space is walked through.\n2. The amount of method novelty is relatively limited. While the results are very nice and this is an important contribution, the method is based off of the same framework as ENARF/GNARF/EVA3D/AvatarGen where bodies are generated in the canonical pose and then deformed, and the tri-plane representation is used for volume rendering. While explicitly modeling hands and faces is important, it seems like a combination of methods like GNARF which have already been applied to heads, and just putting them in the same model. The additional results with text-driven generation or audio-driven manipulation are very cool, but are not technically novel and are instead a combination of open-source models (and can be applied to any other generative articulable 3D representation).\n\nQUESTIONS:\n1. It is mentioned that the left and right hand are represented with the same set of tri-planes but mirrored. Does this limit the capacity in the generation, i.e. when there are different accessories on either hand?\n2. Is truncation used when sampling the generated results from the method? Is it used for sampling for the baseline methods, such as AvatarGen and EVA3D? This should be standard for evaluating qualitative results.\n3. I assume this is using the same representation as AvatarGen, but why is it not described in the methodology. Is the representation a NeRF? An SDF? How is volume rendering done?\n\nLIMITATIONS:\nThe limitations of the method have not been explicitly addressed in the paper, although they are addressed in the supplementary information. In my opinion, including some of the limitations, especially surrounding the quality of the datasets and SMPL-X estimators (and how this actually affects the quality of the generated results) would be insightful (potentially more-so than the added applications, such as audio-driven acting or text-driven generation).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed novel part-aware sampling and feature parametrization strategies to improve the fidelity of the avatar model, especially for smaller body parts. These techniques s enable the efficient learning of diverse fashion shapes with a focus on the hand and facial details. Through experimental evaluation of several benchmarks, the method surpasses all baselines including the recent work AG3D for a huge margin. The results emphasize the necessity of introducing discriminator techniques. Moreover, the authors demonstrate control with text and audio-generated poses as a possible application of their model.\n\nSTRENGTHS:\nI appreciate the authors' effort in presentation results and getting fair comparisons with baselines.\n\n- Undoubtedly, the proposed technique for discriminative learning is important and interesting - easily transferable to other models, which is clearly well studied\n- The authors suggest using separate representations for significant parts - hands, face, and body as a simple technique to drastically improve the capacity of the model\n- Comparison presented on three data unrelated to the training corpus.\n\nWEAKNESSES:\n- some important architectural choices are not studied. The original idea of sepration face and body doesn’t have cleare explanation or intuition to separate face from the body for features (not in discriminator)\n- The ideas lying in the core of the method are quite unpretentious, which is not bad but it decreases the novelty and impactful of the model itself.\n- ethical and limitation sections are missing in the main text\n- Animations are only shown for synthetic pose sequences. Try to extend it with real sequences (e.g. AMASS), the body movements can improve the credibility of the method.\n- No images from the competitor, who utilizes the part-based discriminator as well - AG3D - ask authors to sample more images for you.\n- SMPL-based deformation model will affect loose clothing that is not covered in the main text\n\nQUESTIONS:\n- Why do you condition the discriminator for shape and expression (and not pose to handle hands/skeleton deformations or just clear this moment in the main text)? Is it important if so, could you add reference numbers for that?\n- Is it possible to compare the methods for human preferences score, since metrics here are not ideal measures? At least a more careful qualitative evaluation can be a good extension it is difficult to see the difference in Figure 4. Is it possible that FID results here do not correlate well with visual ones?\n- What if the size of the face and hands tri-plane will be downsampled twice more?\n- Does geometry contain any facial attributes related to expression or fingers?\n- What is happening with loose clothing since you have quite a simple deformation model?\n- How good is this part-based scheme for image inversion - do you have any results of intuition in comparison with EVA3D or AvatarGen.\n\nLIMITATIONS:\nIt is difficult to handle wrinkles or accurate textures as well as loose cloth. Moreover, the demonstrated method is less efficient due to the extended representation. Due to the fact that the model is trained without video data, it is not possible to definitively determine the correspondence of pixels can be incorrect. The trained model has a huge bias to fashion poses due to training data that should be taken into account with all conclusions from the model.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for human 3D avatar generation. The whole model is trained on a set of 2D images, thus can support large variations in terms of shape. The framework is based on EG3D with several important modifications:\n1) Incorporation of the inverse LBS, that will deform the canonical representation of the object.\n2) Modeling hands and face with separate triplane branches.\n3) Additional part based discriminators for face and hands.\nThe paper demonstrates good results, that include modeling of the face and hands motion. Although the motion of these parts are rather weak and contain a lot of artifacts.\n\n\nPOST Rebuttal Summary: The answers in the rebuttal makes sense, in the rebuttal authors provided additional results on sensitivity to SMPL-X errors. To this end I increased the rating by one, and encourage the authors to improve the presentation quality of the method and results.\n\nSTRENGTHS:\n- The paper proposed a meaningful framework that combines several existing ideas into a single unified 3D framework.\n- The paper demonstrates good results for the full body modeling.\n- The paper contains rigorous evaluation of the different modeling aspects, as well as detailed comparison with sota methods.\n\nWEAKNESSES:\n- The quality of presentation is not the best, the method is presented in such a way that it looks like a combination of InsetGan[1] and EG3D. Also visual results presented only on a single dataset, or it is not clear where the results on other datasets.  \n\n- Some architectural choices seem questionable. For example why nearest neighbor LBS, was used, instead of the more advanced LBS approximation from HumanNerf[2] or even more advanced SNARF[3]? Also EG3D, because the upsample produces visible inconsistencies in the generated results, why a more advanced training scheme, such as EpiGRAF is not used? The current LBS approximation will struggle with loose clothing that is not well represented with SMPL-X.\n\n- The quality of hands and face is extremely poor. Why it is bad is not analyzed in the paper. Is this a dataset issue? Is it possible to use specific dedicated face/hand datasets in addition to human bodies dataset, and training only face and hand discriminators with them? Or is it an SMPL-X fitting problem? Again is it possible to use more specific hand/face models to improve this aspect?\n\n[1] InsetGAN for Full-Body Image Generation https://arxiv.org/pdf/2203.07293.pdf\n\n[2] HumanNeRF:Free-viewpoint Rendering of Moving People from Monocular Video https://grail.cs.washington.edu/projects/humannerf/\n\n[3] SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes https://arxiv.org/abs/2104.03953\n\n[4] EpiGRAF: Rethinking training of 3D GANs https://arxiv.org/abs/2206.10535\n\nQUESTIONS:\nQuestions listed in weaknesses.\n\nLIMITATIONS:\n- Inconsistency in the results.\n- Poor quality of the hand and face regions.\n- Not very good presentation. Visual results only on a single dataset, no visual comparison with other methods in supplementary videos.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of 3D full body avatar generation, going beyond prior work primarily on more detailed hand and face generation quality and controllability. \nThe pipeline comprises a 3D-aware GAN where the generator generates tri-plane feature maps from noise vector, followed by a 3D decoder that generates the human geometry represented by SDFs, in a canonical pose. The canonical SDF is then posed (with the help of SMPL-X body model) and rendered to be discriminated about the image discriminators. \nThe key contributions of the work is a complete pipeline to model the face and hand regions separately, equip them with controllability (e.g. hand pose and facial expression), via the body model, and finally unify them to the full body generation. This effectively addressed the limitation of prior work, namely the compromised hand/face quality due to their relatively small region in most training images. Extensive experiments show the edge of this work over other recent human generators. The authors also demonstrated the usefulness of a full-body avatar generative model in the applications of test/audio-driven avatar synthesis.\n\nSTRENGTHS:\n- This work provides a complete pipeline for controllable 3D avatar generation, where the components in this pipeline are combined in an intuitive and effective way. Speaking of novelty, although the tri-plane-based 3D GANs are common nowadays and the body-model-aided reposing mechanism is also well-known, I can't come up with other work in this line that well fixes the problem of hand/face quality and controllability. Therefore, the full method makes a good technical contribution to the avatar generative modeling.\n\n- Experiment section is strong, and the results are impressive. It incorporates major recent methods on controllable avatar generation and show a clear performance edge in terms of the general coherency of geometry, rendered image fidelity, pose controllability and sharpness on the face/hand region, validating the effectiveness major technical contribution. \n\n- The paper is well-written, being clear and easy to follow. The main paper is mostly self-contained, and the SupMat is also very thorough.\n\nWEAKNESSES:\n- The generated geometry has limited resolution, are prone to artifacts, and in most of them, the face/hand region are missing details that are present in the textured renderings. Despite the careful handling of hand/face regions, the geometry of these areas generally show a lower quality than the texture. In other words, although the work targets geometry generation, the model often \"covers up\" what should be geometric structures with texture. \n\n- Although the generated avatars can indeed be animated (via SMPL-X) as claimed, there seems to be no mechanism that ensures the physical correctness (e.g. semantics of body parts and cloth) or visual plausibility (e.g. pose-aware cloth deformation) of the animated results. For example, when the generated character has self-contact (e.g. Fig 3, lower right \"Ours\", the subject that touches the face), the contact area are merged as if they are connected (please correct me if wrong); consequently, when animating this subject with novel poses (e.g. arms-open), visual artifacts can appear.\n\nQUESTIONS:\n- The hand, face and body feature maps are generated together in a channel-stack manner. Intuitively, the features of these three different parts are not spatially aligned, and generating them with a same generator might not have other benefits than saving computational cost. Is it investigated whether training separated generators for these 3 parts brings better quality?\n\n- For the face region specifically, AvatarGen also employs a dedicated descriminator, similar to the one used in this work, and yet performs poorly in terms of plausibility of facial detail and expressions. What is the key differentiator in the method that brings such salient improvement?\n\nLIMITATIONS:\nThe techinical limitations and potential societal impacts are discussed in detail in the SupMat, I really appreciate that.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}