XAGen: 3D Expressive Human Avatars Generation
Authors: Zhongcong Xu
Jianfeng Zhang
Jun Hao Liew
Jiashi Feng
Mike Zheng Shou
Venue: NeurIPS 2023 poster (2023)
Keywords: Human Avatar
3D-aware GAN

Abstract: Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available at https://showlab.github.io/xagen.

XAGen: 3D Expressive Human Avatars Generation
Zhongcong Xu
Jianfeng Zhang
Show Lab
ByteDance
National University of Singapore
j'anfengghanglbyycddeee.ccm
zhongeongxuuu. nus edu
Jun Hao Liew
Jiashi Feng
ByteDance junhao cer.oo.lewwbbbb www.:@byteddnnee...... com
ByteDance
jshfeng@bytedance..c com
Mike Zheng Shou
Show Lab
National University of Singapore mike.
mike.zhenge zheng shou@gmail. com
Abstract
Recent advances 3D-aware GAN models have enabled the generation of re- alistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capa- ble of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise multi-scale and multi-part 3D representation that models fine details. Based this representation, we propose multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, design multi-part discriminators that evaluate the quality the generated avatars with respect their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-an methods in terms of realism, diver- sity, and expressive control abilities. Code and data will be made available https://showlabggigg http:://shoollb..giibboooeeee wwt.ub.io/oaaeee....
Introduction
3D avatars present an opportunity create experiences that are exceptionally authentic and immersive in telepresence [10], augmented reality (AR) [22], and virtual reality (VR) [50]. These applications [1, 52, 3, 35] require capture of human expressiveness, including poses, gestures, expressions, and others, to enable photo-realistic generation [65, 70], animation [56], and interaction [33] in virtual environments.
Traditional methods [11, 60, 4, 20, 23] typically create virtual avatars based on template registration or expensive multi-camera light stages in well-controlled environments. Recent efforts [69, 43, 26, 16] have explored the use of generative models to produce 3D human bodies and clothing based on input parameters, such SMPL [38], without the need of 3D supervision. Despite these advancements, current approaches are limited in their ability to handle expressive attributes of the human body, such
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Multi-view Consistent Appearance and Geometry
Varying Facial Expressions
Varying Jaw Poses
Varying Body Poses
Varying Hand Poses
Figure 1: XAGen synthesize realistic 31) avatars with detailed geometry, while providing disentangled control over expressive attributes, ie.,, facial expressions, jaw, body, and hand poses.
as facial expressions and hand poses, as they primarily focus on body pose and shape conditions. Yet, there exist scenarios where fine-grained control ability is strongly desired, e.g., performing social interactions with non-verbal body languages in Metaverse, or driving digital characters to talk with various expressions and gestures, etc. Due to the lack of comprehensive modeling of the full human body, existing approaches [43, 5, 26] fail to provide control ability beyond the sparse joints of major body skeleton, leading to simple and unnatural animation.
In this work, our objective avatar generation model. To achieve this, we introduce the first eXpressive 3D human A Avatar Generation model (XAGen) that can (1) synthesize high-quality 3D human avatars with diverse realistic appearances and detailed geometries; (2) provide independent control capabilities for fine- grained attributes, including body poses, hand poses, jaw poses, shapes, and facial expressions.
enhance the fine- nne-grained control capabilities of GAN-based human
XAGen is built upon recent unconditional 3D-aware generation models for static images [7, 44]. One straightforward approach to implement fully animatable avatar generation is extending 3D GAN models to condition on expressive control signals, such as SMPL-X [47]. Though conceptually simple, such direct modification of conditioning signal cannot guarantee promising appearance quality and control ability. particularly for two crucial yet challenging regions, i.e., the face and hands. This is because (1) Compared with body, face and hands contain similar or even more articulations. In addition, their scales are much smaller than arms, torso, and legs in a human body image, which hinders the gradient propagation from supervision. (2) Face and hands are entangled with the articulated human body and thus will be severely affected by large body pose deformation, leading to optimization difficulty when training solely on full-body image collections.
To address the above challenges, we decompose the learning process of body, face, and hands by adopting multi-scale and multi-part 3D representation and rendering multiple parts independently using their respective observation viewpoints and control parameters. The rendered images are passed to multi-pan discriminators, which provide multi-scale supervision during the training process.
With these careful designs, XAGen can synthesize photo-realistic 3D human avatars that can be animated effectively by manipulating the corresponding control parameters for expressions and poses, as depicted in Figure 1. We conduct extensive experiments on a variety of benchmarks [18, 68, 14, 36), demonstrating the superiority of XAGen over tftte-of-hhe-rtss in terms of appearance, geometry, and controllability. Moreover, XAGen supports various downstream applications such as guided avatar creation and audio-driven animation, expanding its potential for practical scenarios.
Our contributions are three-fold: (1) the best of our knowledge, XAGen is the first 3D GAN model for fully animatable human avatar generation. (2) We propose novel framework that incorporates

---

multi-scale and multi- murti 3D representation together with multi part rendering technique to enhance the quality and control ability, particularly for the face and hands. (3) Experiments demonstrate XAGen surpasses state-of-the-art methods in terms of both quality and controllability, which enables various downstream applications, including text-guided avatar synthesis and audio-driven animation.
Related work
Generative models for avatar creation. Generative models [27, 28, 51] have demonstrated unprece- dented capability for synthesizing high-resolution photo-realistic images. Building upon these genera- tive models, follow-up works [7, 44, 55, 59, 63) have focused on extending 2D image generation to the 3D domain by incorporating neural radiance field [42] or differentiable rasterization [29]. Although enabling 3D-aware generation, these works fail to provide control ability to manipulate the synthe- sized portrait images. To address this limitation, recent research efforts [64, 26, 43, 69, 57, 16, 71] have explored animatable 3D avatar generation leveraging parametric models for face [32] and body [38]. These works employ inverse [31] or forward [9] skinning techniques to control the facial attributes or body poses of the generated canonical avatars [69, 71]. For human body avatars, additional challenges arise due to their articulation properties. Consequently, generative models for human avatars have explored effective 3D representation designs. Among them, ENARF [43] divides an efficient 3D representation [7] into multiple parts, with each part representing one bone. EVA3D [26] employs similar multi-part design by developing a compositional neural radiance field. Despite enabling body control, such representation fails to generate the details of human faces or hands since these parts only occupy small regions the human body images.
Our method differs in two aspects. First, existing works can either control face or body, whereas ours the first 3D avatar generation model with simultancous fine-grained control over the face, body, and hands. Second, we devise a multi-scale and multi-part 3D representation, allowing for generating human body with high fidelity even for small regions like face and hands.
Expressive 3D human modeling. Existing 3D human reconstruction approaches can be categorized into two main categories depending on whether explicit or implicit representations are used. Explicit representations mainly utilize the pre-defined mesh topology, such as statistical parametric models [38, 62, 45, 2] or personalized mesh registrations [19, 12], to model naked human bodies with various poses and shapes. To enhance the expressiveness, recent works have developed expressive statistical models capable of representing details beyond major human body [47, 46, 17] or introduced the surface deformation to capture fine-grained features [30, 58]. On the other hand, leveraging the remarkable advances in implicit neural representationn [41, 42], another line of research has proposed to either rely purely on implicit representations [53] or combine it with statistical models [61, 48, 8] to reconstruct expressive 3D human bodies. The most recent work [15, 54] proposed to learn a single full-body avatar from multi-part portrait videas or 3D scans. In contrast, our approach focuses on developing 3D generative model for fully animatable human avatars, which trainable on only unstructured 2D image collections.
3 Method
In this section, we introduce XAGen, a 3D generative model for synthesizing photo-realistic human avatars with expressive and disentangled controllability over facial expression, shape, jaw pose, body pose, and hand pose. Figure 2 depicts the pipeline of our method.
Given random noise z sampled from Gaussian distribution, XAGen first synthesizes a human avatar with canonical body, face, and hand configurations. In this work, we use X-pose [34] and neutral shape, face, and hand canonical configurations. We leverage Tri-plane [7] as the fundamental building block of 3D representation in our canonical generator. To increase the capability of 3D representation for the smaller-scale face and hands, we introduce multi-part and multi-scale designs into the canonical Tri-plane (Sec. 3.1). ). A mapping network first encodes z and the camera viewpoint of body a into latent code w. The canonical generator then synthesizes three Tri-planes F& conditioned on w, where E (b, f. h} which stands for (body, face, hand).
Based on the generated canonical avatar, we deform it from canonical space to observation space under the guidance of control signal Ph parameterized by an expressive statistical full body model, i.e. SMPL-X [47]. We adopt volumetric rendering [39] to synthesize the full body image. However, due
3

---

Canonical Generator
Multi-Part Rendering
Discriminators
IS
DR B
Real
Mapping Netwark
Fake
oz Pt
Gerernia
Fi
Superr Reesolie
Dee Appy
Real or Fake
Cb
Camera Poses (Ch,, Cry Ch.
IS
Horizontal Flip
Real
Control Params (1r Pr. Ph.
Ph
Fake
or
Ph.
Figure 2: Pipeline of XAGen. Given a random noise z, the canonical generator synthesizes the avatar in the format of canonical multi-part and multi-scale Tri-planes given the corresponding camera pose cb. We then deform the canonical avatar under the guidance of control parameters to render multi-part images using respective camera poses es and upsample the images using super-resolution module. Discriminators encode the output images, camera poses, and control parameters into real or fake probabilities to critique the rendered images. IS represents inverse skinning.
to the scale imbalance between the face/hands and body, rendering only the full body image cannot guarantee quality for these detailed regions. To address this issue, we propose a multi-part rendering technique (Sec. 3.2). Specifically, we employ part-aware deformation and rendering based on the control parameters (pr and ph) and cameras (er and q,). Accordingly, to ensure the plausibility and controllability of the generated avatars, we develop multi-part discriminators to critique the rendered images (Sec. 3.3).
3.1 Multi-scale and Multi-part Representation
XAGen is designed for expressive human avatars with an emphasis on the high-quality face and hands. However, the scale imbalance between face/hands and body may hamper the fidelity of the corresponding regions. To address this issue, we propose a simple yet effective multi-scale and multi-part representation for expressive human avatar generation. Our multi-scale representation builds upon the efficient 3D representation, i.e., Tri-plane [7], which stores the generated features on three orthogonal planes. Specifically, we design three Tri-planes for body, face, and hands, denoted as Fn E I WxxWx Fr E IR Wi WWxxWxx3e and F, E IR W WWxW3 respectively. The size of the face and hand Tri-planes is set half of the body Tri-plane, with Wr W, W,/2.
As depicted in Figure 2, our canonical generator first synthesizes a compact feature map F e R Wix W. x90/2 where C' represents the number of channels. We then separate and reshape into Fe. where h E {b, f, hJ. representing the canonical space of the generated human avatar. Furthermore, to save computation cost, we exploit the symmetry property of hands to represent both left and right hands using one single Fú through horizontal flip operation (refer to Appendix for details).
3.2 Multi-part Rendering
Our method is trainable on unstructured 2D human images. Although this largely reduces the difficulty and cost to obtain data, the training is highly under-constrained due to the presence of diverse poses, faces, and clothes. To facilitate the training process and improve the appearance quality, we propose multi-part rendering strategy. This strategy allows XAGen to learn each part based on the independent camera poses, which further enhances the geometry quality of the face and hands. Specifically, for each training image, we utilize pretrained model [17] to estimate SMPL-X parameters (Po, Pr. Pa] and camera poses {co, Cr, On) for body, face, and hands, respectively. In the rendering stage, we shoot rays using {cp, Cr, Q.) and sample points {x), x x along the rays in the observation space. To compute the feature for each point, we employ inverse linear-blend skinning [31], which finds the transformation of each point from observation space to canonical space produced by the canonical generator. Based on the parameter Pk. where k E {b, f.h), SMPL-X yields an expressive human body model (v, w), where v E Nx3 represents N vertices, and w E R represents the skinning

---

weights of each vertex with respect to joint J. For each point x where i = 1........ M, and Mr is the number of sampled points, we find its nearest neighbour from vertices v. We then compute the corresponding transformation from observation space to canonical space
T Σ
R 0
1
I A 0
)
(I)
where j = = 11..... R3 and i are derived from pk with Rodrigues formula [6], and A represents the offset caused by pose and shape for vertex n, which is calculated by SMPL-X. Based this inverse transformation, we can calculate the coordinates for each point in canonical space x as
x T x
where apply homogeneous coordinates for the calculation.
For the face and hands rendering, i.., {f, h), we directly interpolate their corresponding Tri- plane Fi and Fa to compute the feature f and f... Regarding the body rendering, we first define three bounding boxes Br, Bm, Bn for face, left and right hands in canonical body space. Then, we query canonical body points that are outside these bounding boxes from body Tri-plane Fo. while the canonical points inside these boxes from Fr and Fr. The query process for body point xba is mathematically formulated
[ Q (
Q(xbs, Fr),
if x E Br, if i E {Ba, Bm) if x & (Br, Bi, Bm)
ph
where Q denotes querying the feature for the given point from the corresponding Tri-planes.
Once the features fk. are obtained, they are encoded into color с and geometry via two lightweight MLP: Inspired by prior works [44, 26, 69], we
multi-layer perceptrons (MLP), where employ signed distance field (SDF) as a proxy to model geometry. Additionally, following [26, 69]. we also query a base SDF de in the canonical space, and predict delta SDF such that de + (f.. de)- We then convert the SDF value into density rendering, where a is a learnable parameter.
Sigmoid( =d) for volume a
To handle the body features queried from multiple Tri-planes, apply feature composition RGB and density using window function [37] for smoothness transition. Specifically, if point x is located in the overlapping region between the body and other parts (face, right hand, and left hand), their features are sampled from both Tri-planes and linearly blended together. More details on the feature composition can be found in the Appendix. Finally, volume rendering is applied to synthesize raw images for body, face, and hands, denoted as { {Im _! I!""), These raw images are then upsampled into high-resolution images (Ib, It, In} by super-resolution module.
3.3 Multi-part Discriminators
Based on the images synthesized by XAGen generator, we design discriminator module critique the generation results. To ensure both the fine-grained fidelity of appearance and geometry as well as disentangled control over the full body, including face and hands, we introduce multi-part discriminators to encode images (T,, Ir, In) into real-fake scores for adversarial training. As depicted in Figure 2, these discriminators are conditioned on the respective camera poses to encode 3D priors, resulting in improved geometries demonstrated in experiments. To enhance the control ability of the face and hands, we further condition face discriminator on expression and shape parameters [oft po and condition hand discriminator on hand pose ph- We encode the camera pose and condition parameters into intermediate embeddings by two separate MLPs and pass them to the discriminators. The multi-part discriminator is formulated as
if k b k
s (1r, MLP MLP where p/-
In p
(4)
if k h
Here 8; denotes the probability of each image T being sampled from real data, and Dr refers to the diseriminator corresponding to the specific body part k. For body part, no conditioning parameters used because empirically find that the condition for body hinders the learning of appearance.
5

---

3.4 Training Losses
The non-saturating GAN loss [21] is computed for each discriminator, resulting in Lp, L,, and Lg. We also regularize these discriminators using RI regularization loss [40] LRI. To improve the plausibility and smoothness of geometry, we compute minimal surface loss Minsurf. Eikonal loss Lrak and human prior regularization loss [prrr as suggested in previous works [44, 69].
Due to the occlusion in the full body images, some training samples may not contain visible faces hands. Thus, we balance the loss terms for both generator and discriminator based on the visibility of face Mr and hands Mn. which denote whether face and hands are detected or not. The overall loss term of XAGen is formulated as
Lg Dg Me L + M o L AMinsun + AEIK LEIk + Aprice
(5)
Lp L. + LRi + A (LP Lki) + An @ (LP + LRR):
where 3 means wnst multiplication, and X, are the weighting factors for each term.
4 Experiments
We evaluate the performance of XAGen on four datasets, i.e., DeepFashion [36], MPV [68], UBC [14], and SHHQ [18]. These datasets contain diverse full body images of clothed individuals. For each image in the dataset, we process it obtain aligned body, face and hand crops, and their corresponding camera poses and SMPL.-X parameters. Please refer to Appendix for more details.
4.1 Comparisons
Baselines. We compare XAGen with four state-of-theeart 3D GAN models for animatable human image generation: ENARF [43], EVA3D [26], AvatarGen [69], and AG3D [16]. All these methods utilize 3D human priors to enable the controllability of body pose. ENARF conditions on sparse skeletons, while others condition on SMPL [38] model. Additionally, AvatarGen and AG3D incorpo- rate an extra face discriminator to enhance face quality. We adopt the official implementations of ENARF and EVA3D, and cite results from AG3D directly. As for AvatarGen, it is reproduced and conditioned on SMPL-X to align with the setup of our model.
Quantitative comparisons. The fidelity of synthesized image is measured by Frechet Inception Distance (FID) [24] computed between 50K generated images and all the available real images in each dataset. To study the appearance quality for face and hands, we further crop face (resolution 642) and hands (resolution 48a) regions from the generated and real images to compute FID, and FID,. To evaluate pose control ability, we compute Percentage of Correct Keypoints (PCK) between 5K real images and images generated using the same pose condition parameters of real images under distance threshold of 0.1. To evaluate this ability in face and hand regions, also report PCK₁ and PCK, Another critical evaluation for a fully controllable generative model is the disentangled control of fine-grained attributes. Inspired by previous works [13, 64], we select attribute from (expression, shape, jaw pose, body pose, hand pose), and modify the selected attribute while keeping others fixed for each synthesis. We then estimate the SMPL-X parameters for 1K generated images using a pre-trained 3D human reconstruction model [17] and compute the Mean Square Error (MSE) for the selected attribute between the input and estimated parameters.
Table summarizes the results for appearance quality and pose control ability for body, face, and hands. It demonstrates that XAGen outperforms existing methods w.r.f. all the evaluation metrics, indicating its superior performance in generating controllable photo-realisti human images with high-quality face and hands. Notably, XAGen shows significant improvements over the most recent method AG3D, achieving more than 20% improvement in FID and FID, on both DeepFashion and the-art oftte-of-the-att pose control ability, with substantial
UBC datasets. Additionally, XAGen achieves state-of performance boost in PCK₁, e.g., a relative improvement of 40.90% on MPV dataset against baseline.
Table 2 presents the results for the disentangled control ability of XAGen compared to the baseline methods. worth noting that ENARF and EVA3D are not fully controllable, but we still report all the evaluation metrics for these two methods to show the controllability lower bound. Notably, the generated images of ENARF are blurry. Thus, our pose estimator cannot estimate precise jaw poses, which leads to an outlier on UBC jaw pose. In general, XAGen demonstrates state-of-the-a oftt---rr-

---

Table 1: Quantitative comparisons with baselines in terms of appearance and overall control ability, with best results in bold. F.Ctl. indicates whether the approach generates fully controllable human body or not. "We implement AvatarGen by conditioning it on SMPI.-X.
DecpFashion [36]
MPV [14]
FCC.. FID! FID, FID PCK PCK,t PCK&t FID, FID,, &ID,, PCK PCK,1 PCKa1
ENARF [43]
X
68.62
52.17 17.17 46.86 3.54
3.79
1.34
65.97
47.71
37.08
3.06
3.55
0.67
EVA3D [26]
X
15.91
14.63 48.10
56.36
75.43
23.14
14.98
27.48
32.54
33.00
42.47
19.24
AG3D [16]
X
10.93
14.79
AvatarGen [69]'
9.53
13.96
27.68
60.12
73.38
46.50
10.06
13.08
19.75
38.32
45.26
30.75
XAGen (Ours)
8.55
10.69
24.26 66.04 87.06
47.56
7.94
12.07
17.35 48.84
63.77
32.01
UBC [68]
SHHQ [18]
FCU. FID FID. FID,, PCK PCK/t PCKat
FID, FID, FID. PCK PCKrt PCK&
ENARF [43]
X
36.39
34.27 32.72 6.90
7.44
6.37
79.29
50.19
46.97
4.43
4.62
2.71
EVA3D [26]
12.61
36.87 45.66 36.31
55.31
8.38
11.99
20.04
39.83
31.24
37.60
18.38
AG3D [16]
X
11.04
15.83
AvatarGen [69]*
9.75
13.23 18.09
65.31
77.09
55.09
10.52
12.57
28.21
59.18
78.71
36.29
XAGen (Ours)
8.80 9.82 16.72
69.18
84.18
55.17
5.88 10.06
19.23
65.14
91.44
38.53
performance for fine-grained controls, particularly in expression, jaw, and hand pose, improving upon baseline by 38.29%, 25.93%, and 33.87% respectively SHHQ dataset which contains diverse facial expressions and hand gestures. These results highlight the effectiveness XAGen in enabling disentangled control over specific attributes of the generated human avatar images.
Table Quantitative comparisons with baselines in terms of disentangled control ability measured by MSE. We report Jaw x 10 and others x x10-² for simplicity, with best results in bold. "We implement AvatarGen by conditioning it on SMPL-X.
DeepFashion [36]
MPV [14]
Expl.
Shape Jawl Body| Hand, Exp..
Shapel,
Jaw
Body.
Hand
ENARF [43]
13.47
6.30
5.79
3.14
9.87
11.21
4.91
8.36
2.75
12.90
EVA3D [26]
6.03
2.87
5.11
1.78
3.68
9.97
4.14
13.83
1.80
4.65
AvatarGen [69]'
4.92
3.06
5.05
1.23
3.17
8.98
3.88
15.22
1.11
3.47
XAGen (Ours)
4.46
2.77
3.67
1.26
2.95
6.31
3.88
7.43
0.94
2.23
UBC [68]
SHHQ [18]
Exp4
Shapel
Jawl, Body|. Hand]
Expl.
Shapel. Jaw.. Body] Hand]
ENARF [43]
10.70
6.11
3.62
1.07
8.19
14.51
6.43
8.16
3.27
9.83
EVA3D [26]
7.00
2.98
5.36
1.00
2.78
7.43
4.15
9.26
1.93
5.15
AvatarGen [69]*
9.59
4.50
9.34
1.22
3.01
9.01
3.99
8.87
1.52
4.99
XAGen (Ours)
5.35
2.57
4.76
0.73
1.63
5.56
3.66
6.57
1.24
3,30
Qualitative comparisons. Figure 3 provides qualitative comparisons between XAGen and baselines. From the results, we observe that ENARF struggles to produce reasonable geometry or realistic images due to the limitations of low training resolution. While EVA3D and AvatarGen achieve higher quality, they still fail to synthesize high-fidelity appearance and geometry for the face and hands. In contrast, XAGen demonstrates superior performance with detailed geometries for face and hands regions, resulting in more visually appealing human avatar images. The improvement of XAGen against baseline models is also confirmed by the perceptual user study, which summarized in Table 3. Notably, XAGen achieves the best perceptual preference scores for both image appearance (2 57.2%) and geometry (2 48.3%) on all the benchmark datasets.
Figure showcases qualitative results for fine- -neinnri control ability. We first observe that ENARF fails to generate correct arm for the given body pose. Although EVA3D demonstrates a better pose condition ability, its shape conditioning ability is limited and the generated face suffers from unrealistic scaling. On the other hand, AvatarGen shows comparable results for pose and shape control. However, when it comes to expression, jaw pose, and hand pose controls, ours significantly
7

---

ENARF
EVA3D
AvatarGen
Ours
Figure 3:
Comparisons against baselines in terms of appearance and 3D geometry. Our method produces photo-rcalistic human images with superior detailed geometries.
Table 3: We conduct a perceptual human study and report participants' preferences on images and geometries generated by our method and baselines. It is measured by preference rate (%), with best results in bold. RGB represents image, and Geo represents geometry. *We implement AvatarGen by conditioning it on SMPL-X.
DcepFashion [36]
MPV [14]
UBC [68]
SHHQ [18]
RGBt
Geot
RGB1 Geot
RGBt
Geot
RGBt Geot
ENARF [43]
0.0
0.0
0.0
0.6
0.0
EVA3D [26]
17.3
35.6
15.0
17.2
7.8
34.4
11.3
15.5
AvatarGen [69]*
15.4
16.1
17.2
18.9
34.4
3.9
28.2
28.6
XAGen (Ours)
67.3
48.3
67.8
63.9
57.2
61.7
60.5
55.9
outperforms AvatarGen, e.g., AvatarGen produces distortion in mouth region and blurred fingers while XAGen demonstrates natural faces and correct hand poses.
4.2
Ablation studies
To verify the design choices in our method, conduct ablation studies SHHQ dataset, which contains diverse appearances, i.e., various human body, face, and hand poses as well as clothes.
Representation. XAGen adopts multi-scale and multi-part representation to improve the quality for face and hands regions. We study the necessity of this design by removing Tri- Tri-planes for face and hands. Table 4a provides the results, indicating that using only single full-body Tri-plane (without any specific Tri-planes for face or hands) results a significant degradation in appearance quality. Adding either face or hand Tri-plane can alleviate this issue and all the FID metries drop slightly. The best results are achieved when both face and hand Tri-planes are enabled, demonstrating the importance of our multi-scale and multi-pan -prt representation.
Multi-part rendering. In our model, we render multiple parts independently in the forward process to disentangle the learning of body, face, and hands. Table 4b demonstrates that independent rendering for face is crucial, as significantly improves both fidelity (FID;: 20.63 vs. 10.06) and control ability (Exp: 6.58 vs. 5.56, Jaw: 7.26 6.57) for face. Similarly, without rendering for hand, FID, increases from 18.85 to 25.94, and MSE increases from 3.28 to 4.55 (Table 4c). The effectiveness of multi part rendering is further supported by the qualitative results shown in Figure 5. Without independent rendering, the geometry quality degrades. For example, the eyes and mouth are collapsed without face rendering, and the model also fails to synthesize geometric details for hand when hand rendering disabled. These highlight the importance of multi-part rendering in facilitating the learning of 3D geometries for different body parts.
Discriminators. To study the effect of multi-part discriminators, we disable each of them during training. As shown in Table 4b, without face discriminator, the overall appearance quality deteriorates.

---

Body Pose
ENARF
EVA3D
AvatarGen
Ours
Shape
EVA3D
AvatarGen
Ours
Expression
AvatarGen
Ours
Jaw Pose
AvalarGen
Ours
Hand Pose
AvalarGen
Ours
Figure 4: Qualitative comparisons in terms of disentangled control ability. Our method exhibits
state-of-the-art control abilities for body pose, shape, expression, jaw pose, and hand pose.
Table 4: Ablations of our method on SHHQ dataset. We vary representation, rendering method, and discriminators to investigate their effectiveness.
Repr. FID FID,, FID.
Face FID
FID,
Expl Jaw
Hand
FID4
FID..)
Hand
w/o both 11.50 12.57 20.97
w/o Rend
14.53
20.63
6.58
7.26
w/o Rend
14.28
26.66
4.51
w/ face
11.27 11.95 20.10
w/o Disc 7.40 9.20
6.27
6.58
w/o Disc 7.78
16.74
4.46
hand 9.64
11.61 19.92
w/ both
5.88 10.06 5.56
6.57
w/ both 5.88
19.23
3.33
w/ both 5.88 10.06 19.23
(b) The effect of face rendering and face discriminator.
(c) The effect of hand rendering and hand discriminator.
(a) The effect multi-scale and multi part representations.
Despite the slight improvement in face appearance, there is a drop in the control ability, as evidenced by the increase in the MSE values for expression and jaw pose. similar observation can be made for hand discriminator in Table 4c. Furthermore, the qualitative results shown in Figure 5 provide visual evidence the impact of the face and hand discriminators on the 3D geometries. When they are removed, the geometries for face and hand collapse.
4.3 Applications
Text-guided avatar synthesis. Inspired by recent works [25, 69, 67] on text-guided avatar generation, we leverage pretrained vision-language encoder CLIP [49] to guide the generation process using the given text prompt. The text-guided avatar generation process involves randomly sampling latent code z and a control parameter ps from the dataset, and optimizing z by maximizing the CLII similarities between the synthesized image and text prompt. As shown in Figure 6a, the generated human avatars exhibit the text-specified attributes, i.e., hair and clothes adhere to the given text prompt (e.g., brown hair and red T-shirt). The generated avatar can be re-targeted by novel SMPL-X parameters, allowing for additional control and customization of the synthesis.
Audio-driven animation. The ability of XAGen to generate fully animatable human avatars with fine-grat amn-irrined control (Figure opens up possibilities for audio-driven animation. The 3D avatars can be driven by arbitrary SMPL--X motion sequences generated by recent works such as [66] given audio inputs. Specifically, we sample an audio stream and SMPL--X sequence from TalkSHOW [66] and use it to animate the generated avatars. As shown in Figure 6b, XAGen is able to synthesize temporally consistent video animations where the jaw poses of the avatars are synchronized with the audio stream (highlighted in red box). Additionally, the generated avatars are generalizable given novel body poses and hand gestures, allowing diverse and expressive animations.
5 Limitations
Although XAGen is able to synthesize photo-realistic and fully animatable human avatars, there are still areas where improvements can be made: (1) XAGen relies on pre-estimated SMPL-X parameters, the inaccurate SMPL-X may introduce potential errors into our model, which can lead to artifacts

---

w/o Face Rend
w/o Hand Rend
w/o Face Disc
w/o Hand Disc
w/ both
Figure 5: Qualitative results for the ablations on multi-part rendering and discriminators.
Sequdi Arabia's first female athlete
brown hair woman, T-shirt, blue jeans
blonde hair woman, pink T-shirt, black trousers
(a) Text-guided avatar synthesis.
(b) Audio-driven avatar animation.
Figure 6: Downstream applications of our method.
and degraded body images. Please refer to Sup. Mat. for the experimental analysis of this issue. We believe our method can benefit from a more accurate SMPL-X estimation method or corrective operations. (2) SMPL-X only represents naked body. Thus, methods built upon SMPL-X could struggle with modeling loose clothing, which is long-standing challenge for 3D human modeling. We believe an advanced human body prior or independent clothing modeling approach is helpful to alleviate this issue. (3) Face and hand images in existing human body datasets lack diversity and sharpness, which affects the fidelity of our generation results, particularly for the novel hand poses that are could help tackle this problem. (4) XAGen utilizes inverse blend skinning to deform the points from canonical space to the observation space. However, this process could introduce errors, particularly when computing nearest neighbors for query points located in the connection or interaction regions. Thus, exploring more robust and accurate techniques, such as forward skinning [9], could open up new directions for future work.
A more diverse dataset with high-quality face and hand images
Conclusion
This work introduces XAGen, novel 3D avatar generation framework that offers expressive control over facial expression, shape, body p jaw pose, and hand pose. Through the use of multi-scale and multi-part representation, XAGen can model details for small-scale regions like faces and hands. By adopting multi-part rendering, XAGen disentangles the learning process and produces realistic details for appearance and geometry. With multi-part discriminators, our model is capable of synthesizing high-quality human avatars with disentangled fine-grained control ability. The capabilities of XAGen open up a range of possibilities for downstream applications, such as text-guided avatar synthesis and audio-driven animation.
10

---

Acknowledgement
This project is supported by the National Research Foundation, Singapore under its NRFF Award SAT-NRRFF13-2021-000 and the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2022).
References
[1] O. Alexander, M. Rogers, W. Lambeth, J.-Y. Chiang, W.-C. Ma, C.-C. Wang. and P. Dehevec. The digital emily project: Achieving a photorealistic digital actor. IEEE Computer Graphics and Applications, 2010.
[2] D. Anguelov, Srinivasan, D. Koller, S. Thrun, J. Rodgers, and Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH, 2005.
[3] T. Bagautdinov, C. Wu, T. Simon, F. Prada, T. Shiratori, S.-E. Wei, W. Xu, Y. Sheikh, and J. Saragih. Driving-signal aware full-body avatars, ACM Trans. on Graphics, 2021.
[4] T. Beeler, F. Hahn, D. Bradley, Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross. High-quality passive facial performance capture using anchor frames. In ACM SIGGRAPH, 2011.
[5] A. Bergman, Kellnhofer, W. Yifan, E. Chan, D. Lindell, and G. Wetzstein. Generative neural articulated radiance fields. In NeurIPS, 2022.
[6] C. Bregler, J. Malik, and K. Pullen. Twist based acquisition and tracking of animal and human kinematics. Int'l, J. Computer Vision, 2004.
[7] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks, In CVPR, 2022.
[8] M. Chen, J. Zhang, X. Xu, L. Liu, Y. Cai, J. Feng, and S. Yan. Geometry-guided progressive nerf for generalizable and efficient neural human rendering. ECCV, 2022.
[9] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In CVPR, 2021.
[10) H. Chu, S. Ma, F. De la Torre, S. Fidler, and Y. Sheikh. Expressive telepresence via modular codec avatars. In ECCV, 2020.
[11] A. Collet, Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and
S. Sullivan. High-quality streamable free-viewpoint video. ACM Trans. on Graphics, 2015.
[12] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.P. Seidel, and S. Thrun. Performance capture from sparse multi-view view video. In ACM SIGGRAPH, 2008.
[13] Y. Deng, J. Yang, D. Chen, E. Wen, and X. Tong. Disentangled and controllable face image generation via 3d imitative-contrastiv learning. In CVPR, 2020.
[14] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, Zhu, Z. Hu, and J. Yin. Towards multi-pose guided virtual try-on network. In CVPR, 2019.
[15] J. Dong, Q Fang, Y. Guo, S. Peng, Q. Shuai, X. Zhou, and H. Bao. Totalselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies. In NeurIPS, 2022.
[16] Z. Dong, X. Chen, Yang, M. Black, O. Hilliges, and A. Geiger. Ag3d: Learning to generate
3d avatars from 2d image collections. In ICCV, 2023.
[17] Y. Feng, V. Choutas, T. Bolkart, D. Tzionas, and M. J. Black. Collaborative regression of expressive bodies using moderation. 3DV, 2021.
[18] Fu, S. Li, Y. Jiang, K.-Y. Lin, C. Qian, C. C. Loy, W. Wu, and Liu. Stylegan- data-centric odyssey of human generation. In ECCV, 2022.

---

[19] J. Gall, C. Stoll, E. De Aguiar, C. Theobalt, B. Rosenhahn, and H.-P. Seidel. Motion capture using joint skeleton tracking and surface estimation. In CVPR, 2009.
[20] A. Ghosh, Fyffe, Tunwattanapong. Busch, X. Yu, and Debevec. Multiview face capture using polarized spherical gradient illumination. ACM Trans. on Graphics, 2011.
[21] I. Goodfellow, J. Pouget- Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and
Y. Bengio. Generative adversarial networks. Communications of the ACM, 2020.
[22] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang. M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[23] K. Guo, P. Lincoln, P. Davidson, Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-Escolano, R. Pandey, J. Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Trans. on Graphies, 2019.
[24] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
[25] F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Trans. on Graphics, 2022.
[26] Hong, Z. Chen, Y. LAN, L. Pan, and Z. Liu. EVA3d: Compositional 3d human generation from 2d image collections. In ICLR, 2023.
[27] T. Karras, S. Laine, and T. Aila. A stylc-based generator architecture for generative adversarial networks. In CVPR, 2019.
(28) T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In NeurIPS, 2021.
[29] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In CVPR, 2018.
[30] N. Kolotouros, G. Pavlakos, and K. Danillidis. Convolutional mesh regression for single-image human shape reconstruction. In CVPR, 2019.
[31] P. Lewis, M. Cordner, and N. Fong. Pose space deformation: unified approach to shape interpolation and skeleton-driven deformation. In CG/T, 2000.
[32] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero. Learning a model of facial shape and expression from 4D scans. ACM Trans. Graphics, 2017.
[33] J.-W. Liu, Y.-P. Cao, T. Yang, Z. Xu, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou. Hosnerf: Dynamic human-object-scene neural radiance fields from a single video. In ICCV, 2023.
[34] L. Liu, M. Habermann, Rudnev, K. Sarkar, J. Gu, and C. Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. on Graphics, 2021.
[35] T. Liu, J. Zhang, X. Nie, Y. Wei, S. Wei, Y. Zhao, and J. Feng. -patial-aware texture transformer for high-fidelity garment transfer. IEEE Transactions on Image Processing. 2021.
[36] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016.
[37] S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. on Graphics, 2021.
[38] M. Loper, N. Mahmood, Romero, G. Poll-Moll, and M. J. Black. Smpl: A skinned multi- person linear model. ACM Trans. on Graphics, 2015.
[39] N. Max. Optical models for direct volume rendering. IEEE Trans. Visualization and Computer Graphics, 1995.
[40] I.. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In ICLR, 2018.
12

---

[41] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[42] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[43] A. Noguchi, X. Sun, S. Lin, and T. Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. In ECCV, 2022.
[44] R. Or-EI, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Seiimacher-Shlizermaae Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022.
[45] A. A. Osman, T. Bolkart, and M. J. Black. Star: Sparse trained articulated human body regressor. In ECCV, 2020.
[46] A. A. Osman, T. Bolkart, D. Tzionas, and M. J. Black. Supr: A sparse unified part-based human representation. In 3DV, 2022.
[47] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019.
[48] S. Peng, Y. Zhang, Y. Xu, Q Wang, Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[50] E. Remelli, T. Bagautdinov, S. Saito, C. Wu, T. Simon, S.-E. Wei, K. Guo, Z. Cao, F. Prada, J. Saragih, et al. Drivable volumetric avatars using texel-aligned features. In ACM SIGGRAPHI, 2022.
[51] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[52] J. Romero, D. Tzionas, and M. J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Trans. on Graphics, 2017.
[53] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
[54] K. Shen, C. Guo, M. Kaufmann, J. Zarate, J. Valentin, J. Song, and Hilliges. X-avatar: Expressive human avatars. In CVPR, 2023.
[55] Y. Shi, D. Aggarwal, and A. K. Jain. Lifting 2d stylegan for 3d-aware face generation. In CVPR. 2021.
[56] A. Siarohin, O. Woodford, Ren, M. Chai, and S. Tulyakov. Motion representations for articulated animation. In CVPR, 2021.
[57] J. Sun, X. Wang, I. Wang, X. Li, Y. Zhang, H. Zhang, and Y. Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In CVPR, 2023.
[58] S. Tang, F. Tan, K. Cheng, Z. Li, S. Zhu, and P. Tan. A neural network for detailed human depth estimation from a single image. In 2019.
[59] T. Wang, B. Zhang. T. Zhang. S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR, 2023.
[60] D. Xiang, Prada, Bagautdinov, W. Xu, Y. Dong, H. Wen, Hodgins, and C. Wu. Modeling clothing as a separate layer for an animatable human avatar. ACM Trans. Graphics, 2021.
13

---

[61] Y. Xiu, J. Yang. D. Tzionas, and M. J. Black. Icon: implicit clothed humans obtained from normals. In CVPR, 2022.
[62] H. Xu, E. G. Bazavan, A. Zanfir, W. T. Freeman, R. Sukthankar, and C. Sminchisescu. Ghum ghuml: Generative 3d human shape and articulated pose models. In CVPR, 2020.
[63] H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, Feng, and L.. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR, 2023.
[64] H. Xu, G. Song, 7. Jiang. J. Zhang, Shi, J. Liu, W. Ma, Feng, and L.. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR, 2023.
[65] Z. Xu, Zhang, Liew, W. Zhang, Bai, J. Feng, and M. Shou. Pv3d: A 3d generative model for portrait video generation. In ICLR, 2023.
[66] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. Black. Generating holistic 3d human motion from speech. In CVPR, 2023.
[67] K. Youwang, K. Ji- Yeon, and T.-H. Oh. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In ECCV, 2022.
[68] Zablotskaia, A. Siarohin, B. Zhao, and I.. Sigal. Dwnet: Dense warp-hased network for pose-guided human video generation. In BMVC, 2019.
[69] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng. Avatargen: 3d generative model for animatable human avatars. In ECCV Workshop, 2023.
[70] Zhang, H. Yan, Z. Xu, J. Feng, and H. Liew. Magicavatar: Multimodal avatar generation and animation. arviv, 2023.
[71] X. Zhang, J. Zhang, C. Rohan, H. Xu, G. Song, Y. Yang, and J. Feng. Getavatar: Generative textured meshes for animatable human avatars. In ICCV, 2023.
14


Reviews:
[{"review_id": "T2PLAlHNV0", "reviewer": "Reviewer_3QMX", "summary": "This work addresses the problem of 3D full body avatar generation, going beyond prior work primarily on more detailed hand and face generation quality and controllability. \nThe pipeline comprises a 3D-aware GAN where the generator generates tri-plane feature maps from noise vector, followed by a 3D decoder that generates the human geometry represented by SDFs, in a canonical pose. The canonical SDF is then posed (with the help of SMPL-X body model) and rendered to be discriminated about the image discriminators. \nThe key contributions of the work is a complete pipeline to model the face and hand regions separately, equip them with controllability (e.g. hand pose and facial expression), via the body model, and finally unify them to the full body generation. This effectively addressed the limitation of prior work, namely the compromised hand/face quality due to their relatively small region in most training images. Extensive experiments show the edge of this work over other recent human generators. The authors also demonstrated the usefulness of a full-body avatar generative model in the applications of test/audio-driven avatar synthesis.", "questions": "- The hand, face and body feature maps are generated together in a channel-stack manner. Intuitively, the features of these three different parts are not spatially aligned, and generating them with a same generator might not have other benefits than saving computational cost. Is it investigated whether training separated generators for these 3 parts brings better quality?\n\n- For the face region specifically, AvatarGen also employs a dedicated descriminator, similar to the one used in this work, and yet performs poorly in terms of plausibility of facial detail and expressions. What is the key differentiator in the method that brings such salient improvement?", "limitations": "The techinical limitations and potential societal impacts are discussed in detail in the SupMat, I really appreciate that.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This work provides a complete pipeline for controllable 3D avatar generation, where the components in this pipeline are combined in an intuitive and effective way. Speaking of novelty, although the tri-plane-based 3D GANs are common nowadays and the body-model-aided reposing mechanism is also well-known, I can't come up with other work in this line that well fixes the problem of hand/face quality and controllability. Therefore, the full method makes a good technical contribution to the avatar generative modeling.\n\n- Experiment section is strong, and the results are impressive. It incorporates major recent methods on controllable avatar generation and show a clear performance edge in terms of the general coherency of geometry, rendered image fidelity, pose controllability and sharpness on the face/hand region, validating the effectiveness major technical contribution. \n\n- The paper is well-written, being clear and easy to follow. The main paper is mostly self-contained, and the SupMat is also very thorough.", "weaknesses": "- The generated geometry has limited resolution, are prone to artifacts, and in most of them, the face/hand region are missing details that are present in the textured renderings. Despite the careful handling of hand/face regions, the geometry of these areas generally show a lower quality than the texture. In other words, although the work targets geometry generation, the model often \"covers up\" what should be geometric structures with texture. \n\n- Although the generated avatars can indeed be animated (via SMPL-X) as claimed, there seems to be no mechanism that ensures the physical correctness (e.g. semantics of body parts and cloth) or visual plausibility (e.g. pose-aware cloth deformation) of the animated results. For example, when the generated character has self-contact (e.g. Fig 3, lower right \"Ours\", the subject that touches the face), the contact area are merged as if they are connected (please correct me if wrong); consequently, when animating this subject with novel poses (e.g. arms-open), visual artifacts can appear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and recognition that 1) our pipeline is effective, and it makes a good technical contribution to the avatar generative modeling; 2) our results are expressive and show a clear performance edge. We respond to each of your comments one by one in what follows.\n\n> **Weakness 1**\n\nThis is a very insightful point. In our pipeline, we apply a super-resolution module to enhance the coarse texture results given by the volumetric rendering, while there is no super-resolution module for geometry. Thus, it is true that the generated geometry has a lower resolution. Based on our current pipeline, there may exist two possible solutions. First, we increase the volume rendering resolution for geometry and show the comparison results in the right below figure in Figure 3 of the rebuttal PDF file. It can be observed that, with an increasing rendering resolution, we can see more geometric details on the face and dress. It demonstrates that our model can improve geometry by using a larger rendering resolution. Yet, this technique can not fully address this issue, and the second solution is to apply a real upsampling on the 3D representation to enhance the details of geometry. We would leave this for future work.\n\n> **Weakness 2**\n\n- To study the issue of contact area, we rotate the samples shown in Fig 3 in our main paper and render more results from multiple directions. As shown in Figure 1 (a) of our rebuttal file, we rotate the avatar by 50, 110, 220, and 330 degrees respectively. The results demonstrate that the hand is not connected or merged with the face.\n\n- It is true that when animating the subject with novel poses like arms-open, we could observe visual artifacts. The main reason for this phenomenon is that our training dataset contains only fashion data, whereas arms-open poses are less frequent in the dataset. We believe a more diverse dataset with in-the-wild poses would help alleviate this issue.\n\n- It is our limitation that our model does not incorporate physical correctness to improve visual plausibility. We agree that these constraints are helpful, and we will incorporate physical/penetration constraints to further improve the animation results in future works.\n\n> **Question 1**\n\nThis is a good question. First, we have some design choices to help align the features of different parts: (1) As introduced in our main paper (lines 163-167) and Appendix (Section 1.1), we compute the feature for the query points which are located in the overlap regions from two related Tri-planes, and then composite them to increase the transition smoothness and plausibility of the full body image. (2) Our body discriminator is trained on full body images to critique the generation result. It supervises the learning of full body images and the gradients can be backpropagated to not only the body but also face and hand Tri-planes to increase generation quality and help align the features spatially.\n\nTo study the advantages of using a shared generator branch to generate multi-part features, we conduct additional ablation study experiments. We use separated generator branches for hand, face, and body respectively. The results are summarized in the Table below:\n\n| Generator | FID&#8595; | FID_f&#8595; | FID_h&#8595; |\n|-----|:------------------:|:-------------:|:----------:|\n| separated | 7.65 | 12.20 | 21.03 |\n| shared | 5.88 | 10.06   | 19.23  |\n|||||||\n\nThe results show that using separated generators cannot improve generation quality. We think the reason would be the redundancy in separated generators. The redundancy increases the computation cost and hinders the optimization of the generator.\n\n> **Question 2**\n\nThe differences between ours and AvatarGen are:\n\n- The rendering process of face is different. AvatarGen crops face images from synthesized full body images whereas ours use face camera poses to render the face images directly from face Tri-planes. Compared with AvatarGen, our independent rendering process can disentangle the learning of face and body, which reduces the training difficulty on 2D image datasets.\n- Our multi-part discriminators have stronger supervision on face than AvatarGen. In our pipeline, we have face discriminator and full body discriminator. We, therefore, compute adversarial loss terms for both full body and face. And both full body discriminator and face discriminator critique the face regions and supervise the synthesis of faces, which further enhances face quality.\n- Our face discriminator is conditioned on facial expressions while AvatarGen does not use such conditions. Thus, our discriminator leverages the prior knowledge provided by SMPL-X to supervise the training of generator. It can help enhance facial details (i.e., expressions) of the generated faces.\n\nThe above three reasons work together to bring a large improvement in the face synthesis of our pipeline."}, {"review_id": "dAVP5k7w0j", "reviewer": "Reviewer_F4xj", "summary": "The paper presents a framework for human 3D avatar generation. The whole model is trained on a set of 2D images, thus can support large variations in terms of shape. The framework is based on EG3D with several important modifications:\n1) Incorporation of the inverse LBS, that will deform the canonical representation of the object.\n2) Modeling hands and face with separate triplane branches.\n3) Additional part based discriminators for face and hands.\nThe paper demonstrates good results, that include modeling of the face and hands motion. Although the motion of these parts are rather weak and contain a lot of artifacts.\n\n\nPOST Rebuttal Summary: The answers in the rebuttal makes sense, in the rebuttal authors provided additional results on sensitivity to SMPL-X errors. To this end I increased the rating by one, and encourage the authors to improve the presentation quality of the method and results.\n\n\n", "questions": "Questions listed in weaknesses.", "limitations": "- Inconsistency in the results.\n- Poor quality of the hand and face regions.\n- Not very good presentation. Visual results only on a single dataset, no visual comparison with other methods in supplementary videos.\n\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper proposed a meaningful framework that combines several existing ideas into a single unified 3D framework.\n- The paper demonstrates good results for the full body modeling.\n- The paper contains rigorous evaluation of the different modeling aspects, as well as detailed comparison with sota methods. \n\n", "weaknesses": "- The quality of presentation is not the best, the method is presented in such a way that it looks like a combination of InsetGan[1] and EG3D. Also visual results presented only on a single dataset, or it is not clear where the results on other datasets.  \n\n- Some architectural choices seem questionable. For example why nearest neighbor LBS, was used, instead of the more advanced LBS approximation from HumanNerf[2] or even more advanced SNARF[3]? Also EG3D, because the upsample produces visible inconsistencies in the generated results, why a more advanced training scheme, such as EpiGRAF is not used? The current LBS approximation will struggle with loose clothing that is not well represented with SMPL-X.\n\n- The quality of hands and face is extremely poor. Why it is bad is not analyzed in the paper. Is this a dataset issue? Is it possible to use specific dedicated face/hand datasets in addition to human bodies dataset, and training only face and hand discriminators with them? Or is it an SMPL-X fitting problem? Again is it possible to use more specific hand/face models to improve this aspect?\n\n[1] InsetGAN for Full-Body Image Generation https://arxiv.org/pdf/2203.07293.pdf\n\n[2] HumanNeRF:Free-viewpoint Rendering of Moving People from Monocular Video https://grail.cs.washington.edu/projects/humannerf/\n\n[3] SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes https://arxiv.org/abs/2104.03953\n\n[4] EpiGRAF: Rethinking training of 3D GANs https://arxiv.org/abs/2206.10535\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and helpful feedback. We respond to each of your comments one by one in what follows.\n\n> **Weakness 1.1**\n\nFirst, we agree with the reviewer that our presentation could be further improved. However, there do exist several significant differences between our framework and InsetGan/EG3D:\n\n- InsetGan inserts face/hand images into the full body by optimizing the latent code of each pretrained part GANs to guarantee seamless merging. In contrast, our XAGen does not rely on any optimization or post-processing to merge the images of each part. Instead, our model learns to generate full-body images with plausible faces and hands in one forward pass directly.\n\n- InsetGan is designed for 2D image generation and is not animatable. We believe it is non-trivial to directly apply InsetGan in the animatable 3D avatar generation.\n\n- EG3D is an unconditional 3D-aware GAN model proposed for static face/object modeling, while our model is designed for generating fully-animatable 3D human avatars, which is more challenging due to its high articulation, complex clothes, diverse appearances, and small-scale face and hands. Thus, combining EG3D with InsetGan directly cannot achieve this easily.\n\n- EG3D only synthesizes one single Tri-plane and renders the whole image with one volumetric rendering process. Differently, we propose a multi-part rendering approach together with multi-part discriminators to improve fidelity and controllability for body, face, and hand.\n\n> **Weakness 1.2: visual results**\n\nThere may be a misunderstanding because we didn't add explicit captions in Figure 3 of our main text. Indeed, the first row in Figure 3 are generation results on the UBC dataset, and the second row is the results on DeepFashion dataset. We will modify this in the final version of our submission. To evaluate the performance of the other two datasets (SHHQ and MPV), we report more qualitative results in Figure 2 (a) of the rebuttal PDF file.\n\n> **Weakness 2.1: architectural choices**\n\nThanks for the constructive suggestions. We start from the nearest neighbor LBS to implement our framework, and the experiment results show that this approach is simple yet effective. Since our work is focused on 3D avatar generation instead of a new skinning technique, we didn't further explore advanced techniques such as SNARF or HumanNerf. Moreover, SNARF and HumanNerf were originally designed for single-scene fitting, which may have challenges in terms of generalization ability. Yet, this direction is still worth further exploration, and we will leave this for future work.\n\n> **Weakness 2.2: EpiGRAF**\n\nThis question is very insightful. EpiGRAF discards the super-resolution module and proposes a patch-wise training scheme for 3D generation models. Although patch-wise training can reduce the computation cost of each iteration, it also has disadvantages, such as missing global information of the image. This drawback could be problematic for our task because we rely on the body discriminator to guarantee plausibility and increase the consistency between each part (face, body, and hands). Similar to EpiGRAF, EVA3D, one of our baselines, also skips a super-resolution module and synthesizes 512X256 images directly by volume rendering. However, their fidelity and controllability are not comparable with ours. Yet, it is still meaningful to try this advanced training scheme, and we will leave this for future work.\n\n> **Weakness 2.3: loose clothing**\n\nWe agree that the LBS approximation will struggle with modeling loose clothing. Loose clothing is a long-standing challenge for 3D human modeling, and it has not been perfectly resolved yet. Even HumanNerf and SNARF with carefully designed skinning techniques and working on single-scene fitting scenario cannot tackle this challenge well. In future works, we may adopt a more advanced human body prior or process clothing independently to address this issue.\n\n> **Weakness 3**\n\nThis is a meaningful question. First, we agree that the dataset could be an issue because the quality of the faces and hands cropped from the fashion datasets are less diverse and have a much lower resolution than the commonly used face dataset, such as FFHQ. We believe our method can benefit from a more diverse dataset with higher resolution for face and hands. However, augmenting the training dataset using high-quality face dataset could be challenging because, in this case, there exists a distribution discrepancy between the full body images and face images. Nonetheless, this is an interesting idea and we will explore this direction in future works.\n\nTo investigate the SMPL-X fitting problem, we conducted additional ablation studies. We add Gaussian noises into the clean SMPL-X fitting results and train the model again. The results are reported in the Table below:\n\n| SMPL-X | FID&#8595; | FID_f&#8595; | FID_h&#8595; | PCK&#8593; | PCK_f&#8593; | PCK_h&#8593; | Exp&#8595; | Shape&#8595; | Jaw&#8595; | Body&#8595; | Hand&#8595; |\n|-----|:------------------:|:-------------:|:----------:|:------------------:|:-------------:|:----------:|:------------------:|:-------------:|:----------:|:-------------:|:----------:|\n| noisy | 8.61 | 10.48 | 19.55 | 62.17 | 90.48 | 31.14| 6.78 | 3.88 | 7.45 | 1.51 | 3.75 |\n| clean | 5.88 | 10.06   | 19.23 | 65.14 | 91.44 | 38.53| 5.56 | 3.66 | 6.57 | 1.24 | 3.30 |\n|||||||\n\nAs we can see, noisy SMPL-X will not affect face and hand image quality too much. It will instead affect control ability and full body quality. Therefore, a more precise or specific SMPL-X/face/hand estimation method may improve control ability more than hands/face quality.\n\n> **Limitations**\n\nWe will move the section for limitation discussion from the supplementary material to the main text, and discuss the limitations of our work thoroughly. Please refer to our responses above for the presentation issue and qualitative comparisons on other datasets.\n"}, {"review_id": "uHTkCTjm3J", "reviewer": "Reviewer_YASD", "summary": "The authors proposed novel part-aware sampling and feature parametrization strategies to improve the fidelity of the avatar model, especially for smaller body parts. These techniques s enable the efficient learning of diverse fashion shapes with a focus on the hand and facial details. Through experimental evaluation of several benchmarks, the method surpasses all baselines including the recent work AG3D for a huge margin. The results emphasize the necessity of introducing discriminator techniques. Moreover, the authors demonstrate control with text and audio-generated poses as a possible application of their model.", "questions": "- Why do you condition the discriminator for shape and expression (and not pose to handle hands/skeleton deformations or just clear this moment in the main text)? Is it important if so, could you add reference numbers for that?\n- Is it possible to compare the methods for human preferences score, since metrics here are not ideal measures? At least a more careful qualitative evaluation can be a good extension it is difficult to see the difference in Figure 4. Is it possible that FID results here do not correlate well with visual ones?\n- What if the size of the face and hands tri-plane will be downsampled twice more?\n- Does geometry contain any facial attributes related to expression or fingers?\n- What is happening with loose clothing since you have quite a simple deformation model?\n- How good is this part-based scheme for image inversion - do you have any results of intuition in comparison with EVA3D or AvatarGen.", "limitations": "It is difficult to handle wrinkles or accurate textures as well as loose cloth. Moreover, the demonstrated method is less efficient due to the extended representation. Due to the fact that the model is trained without video data, it is not possible to definitively determine the correspondence of pixels can be incorrect. The trained model has a huge bias to fashion poses due to training data that should be taken into account with all conclusions from the model.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I appreciate the authors' effort in presentation results and getting fair comparisons with baselines.\n\n- Undoubtedly, the proposed technique for discriminative learning is important and interesting - easily transferable to other models, which is clearly well studied\n- The authors suggest using separate representations for significant parts - hands, face, and body as a simple technique to drastically improve the capacity of the model\n- Comparison presented on three data unrelated to the training corpus.", "weaknesses": "- some important architectural choices are not studied. The original idea of sepration face and body doesn’t have cleare explanation or intuition to separate face from the body for features (not in discriminator)\n- The ideas lying in the core of the method are quite unpretentious, which is not bad but it decreases the novelty and impactful of the model itself.\n- ethical and limitation sections are missing in the main text\n- Animations are only shown for synthetic pose sequences. Try to extend it with real sequences (e.g. AMASS), the body movements can improve the credibility of the method.\n- No images from the competitor, who utilizes the part-based discriminator as well - AG3D - ask authors to sample more images for you.\n- SMPL-based deformation model will affect loose clothing that is not covered in the main text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and constructive suggestions. We respond to each of your comments one by one in what follows.\n\n> **Weakness 1**\n\nOur motivation of separating face and hands from body features is to increase the resolution of Tri-planes and improve the model's capacity for these small-scale regions. This design choice has been confirmed by our ablation studies in Table 3(a) of our main text. When we add two separate Tri-planes for face and hand, we achieve the best results in terms of visual fidelity.\n\n> **Weakness 2**\n\nWhile we acknowledge the individual components used in our work have been explored in previous studies, combining and adapting them for generative and expressive 3D human modeling is a challenging and non-trivial task. Our carefully designed XAGen not only generates high-quality 3D fully animatable avatars but also achieves clear improvements in face and hands generation/animation. Thus, we believe integrating all the important modules coherently and demonstrating their effects through a comprehensive ablation study constitutes a valuable contribution, as also highlighted by Reviewer 3QMX. \n\n> **Weakness 3**\n\nThanks for pointing this out, we will move these two sections from supplementary material to the main text.\n\n> **Weakness 4**\n\nWe have generated animation results using the sequences sampled from the AMASS dataset. The image results can be found in Figure 1(b) in rebuttal file. We have also sent the link to video results to the AC. In addition, our animation results in initial submission are driven by the sequences from TalkSHOW dataset, which are the tracking results of realistic videos instead of synthetic.\n\n> **Weakness 5**\n\nWe have contacted the authors for the AG3D results. The qualitative comparisons can be found in Figure 2(b) in the rebuttal file.     \n\n> **Weakness 6**\n\nWe agree that SMPL-based deformation will struggle with modeling loose clothing. Loose clothing is a long-standing challenge for 3D human modeling. We believe an advanced human body prior or independent clothing modeling is helpful to alleviate this issue. We will add the discussion of this point into the limitation section in our main text.\n\n> **Question 1**\n\nWe choose these designs experimentally. To investigate these choices, we conduct additional ablation studies and the results are shown below:\n\n| Expression | FID_f&#8595; | PCK_f&#8593; | Exp&#8595; |\n|-|:-:|:-:|:-:|\n| w/o | 9.57 | 91.43 | 5.86 |\n| w/ | 10.06 | 91.44 | 5.56  |\n|||||\n\n*Expression:* Although adding expression condition slightly decreases the FID of face, it will improve the control ability in terms of facial expression.\n\n| Body Pose | FID&#8595; | PCK&#8593; | Body&#8595; |\n|-|:-:|:-:|:-:|\n| w/ | 14.23 | 66.69 | 1.00 |\n| w/o | 5.88 | 65.14 | 1.24  |\n|||||\n\n*Body Pose:* It can be observed that conditioning on body pose will significantly affect image quality. Although it can improve the control ability for body pose, the quality decrease is too large.\n\n| Hand Pose | FID_h&#8595; | PCK_h&#8593; | Hand&#8595; |\n|-|:-:|:-:|:-:|\n| w/ | 19.57 | 40.07 | 3.59 |\n| w/o | 19.23 | 38.53 | 3.30  |\n|||||\n\n*Hand Pose:* We can see that conditioning on the hand pose will not affect the visual quality too much. Though the PCK drops, the MSE for hand control is improved. Thus, we skip it to save the computation cost.\n\n> **Question 2**\n\nYes, we conducted human studies for qualitative comparisons. The results are summarized in Table 2 of our rebuttal PDF file. We can see that our method outperforms baselines in terms of texture and geometry quality on four datasets. The human preference scores are in line with the FID scores reported in Table 1 of our main text.\n\n> **Question 3**\n\nTo answer this question, we conducted ablation experiments accordingly. The results are shown in the Table below:\n| Triplanes | FID&#8595; | FID_f&#8595; | FID_h&#8595; |Exp&#8595; | Jaw&#8595; | Hand&#8595; |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n| half | 7.05 | 9.59 | 20.33 | 6.17 | 7.03 | 4.11 |\n| full | 5.88 | 10.06 | 19.23 | 5.56 | 6.57 | 3.30 |\n|||||\n\nIt can be observed that reducing the size of face and hand Tri-planes twice will cause a performance decrease in terms of full body and hand quality. Also, we can see a performance drop for expression, jaw pose, and hand pose control. These results show that a smaller Tri-plane will decrease the capacity of our generator and decrease the control abilities.\n\n> **Question 4**\n\nWe tried our best but still couldn't understand this question. We are glad to further answer this question during discussion period.\n\n> **Question 5**\n\nAs shown in the below figure in Figure 6(b) of our main text and the right figure in Figure 1(b) of our rebuttal file, when it comes to loose clothing (e.g., dress), our generator will learn to separate the dress and attach it to the two legs. When the avatar opens its legs, the long dress will also open from the middle. The reason is that our generator adopts the nearest neighbor LBS and each query point will be attached to its nearest SMPL-X vertex in inverse skinning.\n\n> **Question 6**\n\nWe report the inversion results in the right above of Figure 3 in rebuttal PDF file. Compared with EVA3D, our result has a better hand and a more detailed geometry. Compared with AvatarGen, we can inverse the T-shirt collar better and we also exhibit a more detailed geometry.\n\n> **Limitations**\n\nThanks for pointing these limitations out, we will discuss these points in our limitation section and move this section from supplementary material to the main text."}, {"review_id": "0HWAGz5PZm", "reviewer": "Reviewer_GuK3", "summary": "The paper proposes a method for the generation of high quality, articulable 3D avatars of humans. The method proposed builds on top of the 3D GAN framework which has been used to learn to generate 3D articulable human bodies from collections of 2D images of humans. Additionally, the method adopts the proposed high-level methodology of generating bodies in a canonical pose and then using an explicit deformation guided by a parametric body model in order to render images in a desired pose. The novel contribution of the method is unifying this generation architecture not only for bodies, but for faces, bodies, and hands, and demonstrating that generating, articulating, and discriminating these body parts individually results in a higher quality result than handling them all simultaneously. The paper describes the architecture which accomplishes this, and demonstrates that this results in a state-of-the-art generation quality for 3D human avatars. The contributions of modeling and applying losses to these body parts separately are ablated, demonstrating that they are responsible for improvement in the quality of generated avatars.\n\nAFTER REBUTTAL:\n\nI have read the authors' rebuttal. I believe the additional comparisons provided for the hand and jaw quality address my weakness there. As I mentioned, I still have some questions about the novelty of each of the components, but I think the method performs state-of-the-art overall and combining all of these components into a working method is important for the community. Thus, I am leaning positively.", "questions": "1. It is mentioned that the left and right hand are represented with the same set of tri-planes but mirrored. Does this limit the capacity in the generation, i.e. when there are different accessories on either hand?\n2. Is truncation used when sampling the generated results from the method? Is it used for sampling for the baseline methods, such as AvatarGen and EVA3D? This should be standard for evaluating qualitative results.\n3. I assume this is using the same representation as AvatarGen, but why is it not described in the methodology. Is the representation a NeRF? An SDF? How is volume rendering done?", "limitations": "The limitations of the method have not been explicitly addressed in the paper, although they are addressed in the supplementary information. In my opinion, including some of the limitations, especially surrounding the quality of the datasets and SMPL-X estimators (and how this actually affects the quality of the generated results) would be insightful (potentially more-so than the added applications, such as audio-driven acting or text-driven generation).", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "In my opinion, the main strengths of the paper are that:\n1. The paper is well written and is clear to read and follow, and presentation is extremely polished. As a result, I feel like the method will be impactful and those working in the field of generative 3D articulable avatars will want to build on top of it.\n2. The proposed method demonstrates state-of-the-art results. The evaluations are extremely convincing both qualitatively and quantitatively, and compared to the baselines EVA3D and AvatarGen which are currently generating the state-of-the-art, I believe the generated results are significantly better. This is extremely important in pushing towards photorealistic generative 3D human avatars.\n3. The method is ablated well. The main contributions: separately modeling faces and hands, and using a separate discriminator for each of them, are both ablated in a very clear way showing how they improve the performance of the method. This is extremely important to understand that the contributions proposed are actually responsible for the increase in quality, rather than just hyperparameter tuning.", "weaknesses": "In my opinion, there are two weaknesses of the paper.\n1. The evaluation accuracy of the facial/hand articulation is not entirely convincing. \n    - While Table 2 shows that there are improvements in “Jaw” and “Hand” when a facial and hand pose estimator is applied to the generated results, it is only compared to the modified AvatarGen with SMPL-X (which it outperforms). In order to actually ground these numbers with a baseline, it would be insightful to apply these pose estimators to the results of EVA3D and ENARF. I understand these methods don’t model hand or facial poses, but it would be an important comparison to understand how much improvement is gotten from even modeling these at all.\n    - Quantitative comparisons don’t ever show multiple different identities with modified facial or hand poses (for example, in the supplementary video). Because of this, I don’t know if the examples provided (for example, mouth open), are cherry-picked results or consistent. It would be immensely helpful to show a result where the facial/hand pose is held constant in a deformed state (for example, mouth open), and then the identity latent space is walked through.\n2. The amount of method novelty is relatively limited. While the results are very nice and this is an important contribution, the method is based off of the same framework as ENARF/GNARF/EVA3D/AvatarGen where bodies are generated in the canonical pose and then deformed, and the tri-plane representation is used for volume rendering. While explicitly modeling hands and faces is important, it seems like a combination of methods like GNARF which have already been applied to heads, and just putting them in the same model. The additional results with text-driven generation or audio-driven manipulation are very cool, but are not technically novel and are instead a combination of open-source models (and can be applied to any other generative articulable 3D representation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and recognition that 1) our method has SOTA results; 2) our method is ablated well. We respond to each of your comments one by one in what follows.\n\n> **Weakness 1.1**\n\nThanks for the suggestions, we have computed these metrics for ENARF and EVA3D. The results are shown in Table 1 of our rebuttal PDF file. In general, the results for ENARF and EVA3D are worse than ours because they don't have explicit control over these attributes. However, there is an outlier for ENARF MSE Jaw on UBC dataset. We think the reason is that most of the faces in this dataset have a zero jaw pose. And the images generated by ENARF have a very low resolution and the mouth is always blurry.  In this case, the pose estimator would tend to predict zero jaw poses, which leads to the lowest Jaw MSE.\n\n> **Weakness 1.2**\n\nWe visualize more results using different identities and identical mouth open and hand poses. Also, we visualize the random walk results and keep the mouth and hand pose unchanged to verify the consistency of our results. The results can be found on the left of Figure 3 in our rebuttal PDF file.\n\n> **Weakness 2**\n\nWe would like to thank the reviewer for the recognition of our results and downstream applications. Although the frameworks mentioned by reviewer could be applied for different 3D generation tasks, these straightforward extensions cannot guarantee high-quality faces or hands simultaneously in one generation model. While we acknowledge the individual components used in our work have been explored in previous studies, combining and adapting them for generative and expressive 3D human modeling is a challenging and non-trivial task. Our sophisticated design not only generates high-quality 3D fully animatable avatars but also achieves clear improvements in face and hands generation/animation. Moreover, our method facilitates various downstream applications, like text-guided synthesis and video/audio-driven animation. Thus, we believe integrating all the important modules coherently and demonstrating their effects through a comprehensive ablation study constitutes a valuable contribution, as also highlighted by Reviewer 3QMX. \n\n> **Question 1**\n\nWe conduct an ablation study on the hand Tri-planes, and the results are shown below:\n| Hand Triplanes | FID&#8595; | FID_f&#8595; | FID_h&#8595; | PCK_h&#8593; | Hand&#8595; |\n|-|:-:|:-:|:-:|:-:|:-:|\n| double | 8.32 | 10.12 | 20.53 | 39.64 | 3.27 | \n| single | 5.88 | 10.06 | 19.23 | 38.53 | 3.30 |\n|||||\n\nWe can see that using two hand Tri-planes can improve the control ability of hand, but image fidelity decreases. We think the reason is that generator learns to generate an additional hand Tri-plane, which is difficult to optimize. We agree that independent hand Tri-planes can model different accessories, and we observed this during training (Figure 1(c) in our rebuttal PDF file). However, accessories usually have small scales. Compared with the full body image, this point is less important and will not affect the FID score too much. This is confirmed by the FID score for hand.\n\n> **Question 2**\n\nWe didn't use any truncation for qualitative results. To make fair comparisons, we disable truncation for all the baseline models. We will point this out in our paper.\n\n> **Question 3**\n\nIt is correct, we use the same representation as AvatarGen. We describe this in lines 158-159 of our main text. Our 3D representation is Tri-plane, with SDF as the proxy for geometry. We mention this in line 101 of our main text. Our volume rendering process is identical to NeRF. We will modify the main text to explicitly point these out in the final version.\n\n> **Limitations**\n\nThanks for the suggestions, we conducted an ablation study to investigate the effects of SMPL-X estimations. We randomly sample a random noise and add it to the clean SMPL-X parameters to make a noisy version of the training dataset. The experiment results are summarized below:\n\n| SMPL-X | FID&#8595; | FID_f&#8595; | FID_h&#8595; | PCK&#8593; | PCK_f&#8593; | PCK_h&#8593; | Exp&#8595; | Shape&#8595; | Jaw&#8595; | Body&#8595; | Hand&#8595; |\n|-----|:------------------:|:-------------:|:----------:|:------------------:|:-------------:|:----------:|:------------------:|:-------------:|:----------:|:-------------:|:----------:|\n| noisy | 8.61 | 10.48 | 19.55 | 62.17 | 90.48 | 31.14 | 6.78 | 3.88 | 7.45 | 1.51 | 3.75 |\n| clean | 5.88 | 10.06   | 19.23 | 65.14 | 91.44 | 38.53 | 5.56 | 3.66 | 6.57 | 1.24 | 3.30 |\n|||||||\n\nAs we can see, noisy SMPL-X decreases the model's performance in terms of all the evaluation metrics. It will mainly affect the control ability and full body image fidelity. Therefore, a more precise SMPL-X estimation method may largely improve control ability and full body image quality. We will discuss this in the limitation section and move it from supplementary material to the main text."}, {"review_id": "mszjbaRM0y", "reviewer": "Reviewer_7EQQ", "summary": "The paper proposes a generative model for 3D expressive human avatars. Based on the backbone of the recent works, tri-plane 3D feature representation, volumetric rendering, transformation from pose space to canonical space, parametric body models and GAN, the authors propose to improve the expressive of the model by representing the hand and face with extra tri-planes and supervising them with extra adversarial loss. The result achieves state of the art performance according to various evaluation metrics on various dataset. Moreover, one can animate the hand and jaw of the generated avatar, surpassing most SOTA approaches. Two practical applications of such a generative model are demonstrated in the paper.", "questions": "The submission is in a good state, method clearly stated, experiment is extensive and convincing. I don't have further questions to the authors regarding reviewing this paper.", "limitations": "Limitations and societal impact are discussed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a state of the art generative model for 3D clothed human avatar with expressive control.\n\n2. The paper is solid, well written and easy to follow.\n\n3. The evaluation is extensive and the results are convincing.\n\n4. Two practical applications are demonstrated.", "weaknesses": "1. Lack of technical novelty, multi-tri-plane and part focused supervision are quite weak when claimed as novelty. But I'm relatively ok with this since the performance and the control of the model surpass the SOTA. Although in my opinion, this paper fits much more for SIGGRAPH/Asia, where technical novelty is less expected compared to NeurIPS, I believe the overall quality of the submission clearly reaches the acceptance standard for NeurIPS.\n\n2. While I appreciate the effort on making consistent and good format of references (I notice the authors use consistent form of names of the same conference across the reference, which is rare nowadays in most of the submissions), there is still room to improve the reference, such as \na. Capitalization of words. For example: 3d -> 3D, Icon: implicit... -> ICON: Implicit..., Avartargen: a 3d... -> AvartarGen: A 3D..., \nb. the URL in the references is not necessary.\nc. [65] In ECCVw, 2023 -> In ECCV Workshops, 2022\n\n3. In the supplementary video, \na. the voice over contains environment noise and echoes. It would be great to remove them.\nb. the bottom left animation has a white background, occluding the slide contents. It could be better if those animations were rendered with transparent background.\nc. the demonstration of controlling jaw motion and hand poses are a bit long and not eye catching. It would be more appealing if they can be shortened a bit and shown with a close-up / zoom-in view.\n\n4. On the project page, the RGB video and the geometry visualization are asynchronized regardless of refreshing the page. \n\n5. Suggestion. Please consider incorporating some concurrent work into related work for the updated version to make the paper more inclusive, for example: Chupa (https://snuvclab.github.io/chupa/), a generative model for 3D clothed human geometry; and SCARF (https://yfeng95.github.io/scarf/), creating an expressive avatar from monocular video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and recognition that 1) our method achieves state-of-the-art results; 2) our work is solid and its overall quality is good; 3) our experiments are extensive and convincing. We respond to each of your comments one by one in what follows.\n\n> **Weakness 1**\n\nWe would like to thank the reviewer for the recognition that our overall quality is good for this conference. While we acknowledge the individual components used in our work have been explored in previous studies, combining and adapting them for generative and expressive 3D human modeling is a challenging and non-trivial task. Our sophisticated design not only generates high-quality 3D fully animatable avatars but also achieves clear improvements in face and hands generation/animation. Moreover, our method facilitates various downstream applications, like text-guided synthesis and video/audio-driven animation. Thus, we believe integrating all the important modules coherently and demonstrating their effects through a comprehensive ablation study constitutes a valuable contribution, as also highlighted by Reviewer 3QMX. \n\n> **Weakness 2**\n\nThanks for the recognition and suggestions, we also appreciate your careful observation. We have modified the reference section accordingly and will update the final version.\n\n> **Weakness 3**\n\nThanks for the useful suggestions, we will update the supplementary video accordingly in the final version.\n\n> **Weakness 4**\n\nWe think this could be an issue with the browser. It is recommended to wait until all the samples are fully downloaded and then refresh the page. Also, we will further compress the gif files and update the webpage later to enhance the experience.\n\n> **Weakness 5**\n\nThanks for the suggestion. We have already added the above concurrent works into the related work. We will update our final version accordingly."}]

Author rebuttal:
We want to thank all the reviewers for their constructive and insightful feedback. We appreciate the reviewers' time and efforts spent on our submission. 

Please check our rebuttal PDF files uploaded here for the additional figures and tables. We have sent the video results on AMASS data to AC following the official instructions. 

Please check the comments for each review, and feel free to ask if you have any questions about the rebuttal. We are glad to have further discussions with all the reviewers.

Meta-review: The paper proposes a 3D-aware GAN generative model for controllable avatar creation purely from 2D images in the wild. The main contributions of the work are in proposing a unified framework for animatable avatar creation with significantly improved quality and controllability for the face and hand regions versus the state of the art. The proposed method, although comprises of a combination of known components, shows strong improvements in quality over the existing approaches. All novel aspects of the work are thoroughly ablated and the presentation quality of the paper is good.

All reviewers' concerns were sufficiently addressed during the rebuttal phase. There is consensus among the reviewers to accept the paper (3A, WA, BA). 

The AC recommends acceptance. Congratulations!

The authors are strongly encouraged to incorporate the changes that they've promised in the rebuttal into the final manuscript.
Decision: Accept (poster)