SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We compare XAGen with four state-of-the-art 3D GAN models for animatable human image generation: ENARF [43], EVA3D [26], AvatarGen [69], and AG3D [16]. All these methods utilize 3D human priors to enable the controllability of body pose. ENARF conditions on sparse skeletons, while others condition on SMPL [38] model. Additionally, AvatarGen and AG3D incorporate an extra face discriminator to enhance face quality. We adopt the official implementations of ENARF and EVA3D, and cite results from AG3D directly. As for AvatarGen, it is reproduced and conditioned on SMPL-X to align with the setup of our model.
Quantitative comparisons. The fidelity of synthesized image is measured by Frechet Inception Distance (FID) [24] computed between  $50K$  generated images and all the available real images in each dataset. To study the appearance quality for face and hands, we further crop face (resolution  $64^2$ ) and hands (resolution  $48^2$ ) regions from the generated and real images to compute  $\mathrm{FID_f}$  and  $\mathrm{FID_h}$ . To evaluate pose control ability, we compute Percentage of Correct Keypoints (PCK) between  $5K$  real images and images generated using the same pose condition parameters of real images under a distance threshold of 0.1. To evaluate this ability in face and hand regions, we also report  $\mathrm{PCK_f}$  and  $\mathrm{PCK_h}$ . Another critical evaluation for a fully controllable generative model is the disentangled control of fine-grained attributes. Inspired by previous works [13, 64], we select one attribute from  $\{\text{expression}, \text{shape}, \text{jaw pose}, \text{body pose}, \text{hand pose}\}$ , and modify the selected attribute while keeping others fixed for each synthesis. We then estimate the SMPL-X parameters for  $1K$  generated images using a pre-trained 3D human reconstruction model [17] and compute the Mean Square Error (MSE) for the selected attribute between the input and estimated parameters.
Table 1 summarizes the results for appearance quality and pose control ability for body, face, and hands. It demonstrates that XAGen outperforms existing methods w.r.t. all the evaluation metrics, indicating its superior performance in generating controllable photo-realistic human images with high-quality face and hands. Notably, XAGen shows significant improvements over the most recent method AG3D, achieving more than  $20\%$  improvement in FID and  $\mathrm{FID_f}$  on both DeepFashion and UBC datasets. Additionally, XAGen achieves state-of-the-art pose control ability, with substantial performance boost in  $\mathrm{PCK_f}$ , e.g., a relative improvement of  $40.90\%$  on MPV dataset against baseline.
Table 2 presents the results for the disentangled control ability of XAGen compared to the baseline methods. It is worth noting that ENARF and EVA3D are not fully controllable, but we still report all the evaluation metrics for these two methods to show the controllability lower bound. Notably, the generated images of ENARF are blurry. Thus, our pose estimator cannot estimate precise jaw poses, which leads to an outlier on UBC jaw pose. In general, XAGen demonstrates state-of-the-art
Table 1: Quantitative comparisons with baselines in terms of appearance and overall control ability, with best results in **bold**. F.Ctl. indicates whether the approach generates fully controllable human body or not. *We implement AvatarGen by conditioning it on SMPL-X.
DeepFashion [36]	MPV [14]
F.Ctl.	FID↓	FIDf↓	FIDh↓	PCK↑	PCKf↑	PCKh↑	FID↓	FIDf↓	FIDh↓	PCK↑	PCKf↑	PCKh↑
ENARF [43]	✘	68.62	52.17	46.86	3.54	3.79	1.34	65.97	47.71	37.08	3.06	3.55	0.67
EVA3D [26]	✘	15.91	14.63	48.10	56.36	75.43	23.14	14.98	27.48	32.54	33.00	42.47	19.24
AG3D [16]	✘	10.93	14.79	-	-	-	-	-	-	-	-	-	-
AvatarGen [69]*	✓	9.53	13.96	27.68	60.12	73.38	46.50	10.06	13.08	19.75	38.32	45.26	30.75
XAGen (Ours)	✓	8.55	10.69	24.26	66.04	87.06	47.56	7.94	12.07	17.35	48.84	63.77	32.01
UBC [68]	SHHQ [18]
F.Ctl.	FID↓	FIDf↓	FIDh↓	PCK↑	PCKf↑	PCKh↑	FID↓	FIDf↓	FIDh↓	PCK↑	PCKf↑	PCKh↑
ENARF [43]	✘	36.39	34.27	32.72	6.90	7.44	6.37	79.29	50.19	46.97	4.43	4.62	2.71
EVA3D [26]	✘	12.61	36.87	45.66	36.31	55.31	8.38	11.99	20.04	39.83	31.24	37.60	18.38
AG3D [16]	✘	11.04	15.83	-	-	-	-	-	-	-	-	-	-
AvatarGen [69]*	✓	9.75	13.23	18.09	65.31	77.09	55.09	10.52	12.57	28.21	59.18	78.71	36.29
XAGen (Ours)	✓	8.80	9.82	16.72	69.18	84.18	55.17	5.88	10.06	19.23	65.14	91.44	38.53
performance for fine-grained controls, particularly in expression, jaw, and hand pose, improving upon baseline by  $38.29\%$ ,  $25.93\%$ , and  $33.87\%$  respectively on SHHQ dataset which contains diverse facial expressions and hand gestures. These results highlight the effectiveness of XAGen in enabling disentangled control over specific attributes of the generated human avatar images.
Table 2: Quantitative comparisons with baselines in terms of disentangled control ability measured by MSE. We report  $\mathrm{Jaw}\times 10^{-4}$  and others  $\times 10^{-2}$  for simplicity, with best results in bold. *We implement AvatarGen by conditioning it on SMPL-X.
DeepFashion [36]	MPV [14]
Exp↓	Shape↓	Jaw↓	Body↓	Hand↓	Exp↓	Shape↓	Jaw↓	Body↓	Hand↓
ENARF [43]	13.47	6.30	5.79	3.14	9.87	11.21	4.91	8.36	2.75	12.90
EVA3D [26]	6.03	2.87	5.11	1.78	3.68	9.97	4.14	13.83	1.80	4.65
AvatarGen [69]*	4.92	3.06	5.05	1.23	3.17	8.98	3.88	15.22	1.11	3.47
XAGen (Ours)	4.46	2.77	3.67	1.26	2.95	6.31	3.88	7.43	0.94	2.23
UBC [68]	SHHQ [18]
Exp↓	Shape↓	Jaw↓	Body↓	Hand↓	Exp↓	Shape↓	Jaw↓	Body↓	Hand↓
ENARF [43]	10.70	6.11	3.62	1.07	8.19	14.51	6.43	8.16	3.27	9.83
EVA3D [26]	7.00	2.98	5.36	1.00	2.78	7.43	4.15	9.26	1.93	5.15
AvatarGen [69]*	9.59	4.50	9.34	1.22	3.01	9.01	3.99	8.87	1.52	4.99
XAGen (Ours)	5.35	2.57	4.76	0.73	1.63	5.56	3.66	6.57	1.24	3.30
Qualitative comparisons. Figure 3 provides qualitative comparisons between XAGen and baselines. From the results, we observe that ENARF struggles to produce reasonable geometry or realistic images due to the limitations of low training resolution. While EVA3D and AvatarGen achieve higher quality, they still fail to synthesize high-fidelity appearance and geometry for the face and hands. In contrast, XAGen demonstrates superior performance with detailed geometries for face and hands regions, resulting in more visually appealing human avatar images. The improvement of XAGen against baseline models is also confirmed by the perceptual user study, which is summarized in Table 3. Notably, XAGen achieves the best perceptual preference scores for both image appearance  $(\geq 57.2\%)$  and geometry  $(\geq 48.3\%)$  on all the benchmark datasets.
Figure 4 showcases qualitative results for fine-grained control ability. We first observe that ENARF fails to generate a correct arm for the given body pose. Although EVA3D demonstrates a better pose condition ability, its shape conditioning ability is limited and the generated face suffers from unrealistic scaling. On the other hand, AvatarGen shows comparable results for pose and shape control. However, when it comes to expression, jaw pose, and hand pose controls, ours significantly
Figure 3: Comparisons against baselines in terms of appearance and 3D geometry. Our method produces photo-realistic human images with superior detailed geometries.
Table 3: We conduct a perceptual human study and report participants' preferences on images and geometries generated by our method and baselines. It is measured by preference rate  $(\%)$ , with best results in bold. RGB represents image, and Geo represents geometry. *We implement AvatarGen by conditioning it on SMPL-X.
DeepFashion [36]	MPV [14]	UBC [68]	SHHQ [18]
RGB↑	Geo↑	RGB↑	Geo↑	RGB↑	Geo↑	RGB↑	Geo↑
ENARF [43]	0.0	0.0	0.0	0.0	0.6	0.0	0.0	0.0
EVA3D [26]	17.3	35.6	15.0	17.2	7.8	34.4	11.3	15.5
AvatarGen [69]*	15.4	16.1	17.2	18.9	34.4	3.9	28.2	28.6
XAGen (Ours)	67.3	48.3	67.8	63.9	57.2	61.7	60.5	55.9
outperforms AvatarGen, e.g., AvatarGen produces distortion in mouth region and blurred fingers while XAGen demonstrates natural faces and correct hand poses.
To verify the design choices in our method, we conduct ablation studies on SHHQ dataset, which contains diverse appearances, i.e., various human body, face, and hand poses as well as clothes.
Representation. XAGen adopts a multi-scale and multi-part representation to improve the quality for face and hands regions. We study the necessity of this design by removing Tri-planes for face and hands. Table 4a provides the results, indicating that using only a single full-body Tri-plane (without any specific Tri-planes for face or hands) results in a significant degradation in appearance quality. Adding either face or hand Tri-plane can alleviate this issue and all the FID metrics drop slightly. The best results are achieved when both face and hand Tri-planes are enabled, demonstrating the importance of our multi-scale and multi-part representation.
Multi-part rendering. In our model, we render multiple parts independently in the forward process to disentangle the learning of body, face, and hands. Table 4b demonstrates that independent rendering for face is crucial, as it significantly improves both fidelity  $(\mathrm{FID}_{\mathrm{f}}:20.63$  vs. 10.06) and control ability (Exp: 6.58 vs. 5.56, Jaw: 7.26 vs. 6.57) for face. Similarly, without rendering for hand,  $\mathrm{FID}_{\mathrm{h}}$  increases from 18.85 to 25.94, and MSE increases from 3.28 to 4.55 (Table 4c). The effectiveness of multi-part rendering is further supported by the qualitative results shown in Figure 5. Without independent rendering, the geometry quality degrades. For example, the eyes and mouth are collapsed without face rendering, and the model also fails to synthesize geometric details for hand when hand rendering is disabled. These highlight the importance of multi-part rendering in facilitating the learning of 3D geometries for different body parts.
Discriminators. To study the effect of multi-part discriminators, we disable each of them during training. As shown in Table 4b, without face discriminator, the overall appearance quality deteriorates.
Figure 4: Qualitative comparisons in terms of disentangled control ability. Our method exhibits state-of-the-art control abilities for body pose, shape, expression, jaw pose, and hand pose.
Table 4: Ablations of our method on SHHQ dataset. We vary our representation, rendering method, and discriminators to investigate their effectiveness.
Repr.	FID↓	FIDf↓	FIDh↓
w/o both	11.50	12.57	20.97
w/ face	11.27	11.95	20.10
w/ hand	9.64	11.61	19.92
w/ both	5.88	10.06	19.23
Face	FID↓	FIDf↓	Exp↓	Jaw↓	Hand	FID↓	FIDh↓	Hand↓
w/o Rend	14.53	20.63	6.58	7.26	w/o Rend	14.28	26.66	4.51
w/o Disc	7.40	9.20	6.27	6.58	w/o Disc	7.78	16.74	4.46
w/ both	5.88	10.06	5.56	6.57	w/ both	5.88	19.23	3.33
(a) The effect of multi-scale and multi-part representations.
(b) The effect of face rendering and face discriminator.
(c) The effect of hand rendering and hand discriminator.
Despite the slight improvement in face appearance, there is a drop in the control ability, as evidenced by the increase in the MSE values for expression and jaw pose. A similar observation can be made for hand discriminator in Table 4c. Furthermore, the qualitative results shown in Figure 5 provide visual evidence of the impact of the face and hand discriminators on the 3D geometries. When they are removed, the geometries for face and hand collapse.
Text-guided avatar synthesis. Inspired by recent works [25, 69, 67] on text-guided avatar generation, we leverage a pretrained vision-language encoder CLIP [49] to guide the generation process using the given text prompt. The text-guided avatar generation process involves randomly sampling a latent code  $\mathbf{z}$  and a control parameter  $p_{\mathrm{b}}$  from the dataset, and optimizing  $\mathbf{z}$  by maximizing the CLIP similarities between the synthesized image and text prompt. As shown in Figure 6a, the generated human avatars exhibit the text-specified attributes, i.e., hair and clothes adhere to the given text prompt (e.g., brown hair and red T-shirt). The generated avatar can be re-targeted by novel SMPL-X parameters, allowing for additional control and customization of the synthesis.
Audio-driven animation. The ability of XAGen to generate fully animatable human avatars with fine-grained control (Figure 1) opens up possibilities for audio-driven animation. The 3D avatars can be driven by arbitrary SMPL-X motion sequences generated by recent works such as [66] given audio inputs. Specifically, we sample an audio stream and SMPL-X sequence from TalkSHOW [66] and use it to animate the generated avatars. As shown in Figure 6b, XAGen is able to synthesize temporally consistent video animations where the jaw poses of the avatars are synchronized with the audio stream (highlighted in red box). Additionally, the generated avatars are generalizable given novel body poses and hand gestures, allowing diverse and expressive animations.
Although XAGen is able to synthesize photo-realistic and fully animatable human avatars, there are still areas where improvements can be made: (1) XAGen relies on pre-estimated SMPL-X parameters, the inaccurate SMPL-X may introduce potential errors into our model, which can lead to artifacts
w/o Face Rend
w/o Hand Rend
w/o Face Disc
w/o Hand Disc
Figure 5: Qualitative results for the ablations on multi-part rendering and discriminators.
(a) Text-guided avatar synthesis.
(b) Audio-driven avatar animation.
Figure 6: Downstream applications of our method.
and degraded body images. Please refer to Sup. Mat. for the experimental analysis of this issue. We believe our method can benefit from a more accurate SMPL-X estimation method or corrective operations. (2) SMPL-X only represents naked body. Thus, methods built upon SMPL-X could struggle with modeling loose clothing, which is a long-standing challenge for 3D human modeling. We believe an advanced human body prior or independent clothing modeling approach is helpful to alleviate this issue. (3) Face and hand images in existing human body datasets lack diversity and sharpness, which affects the fidelity of our generation results, particularly for the novel hand poses that are out-of-distribution. A more diverse dataset with high-quality face and hand images could help tackle this problem. (4) XAGen utilizes inverse blend skinning to deform the points from canonical space to the observation space. However, this process could introduce errors, particularly when computing nearest neighbors for query points located in the connection or interaction regions. Thus, exploring more robust and accurate techniques, such as forward skinning [9], could open up new directions for future work.
This work introduces XAGen, a novel 3D avatar generation framework that offers expressive control over facial expression, shape, body pose, jaw pose, and hand pose. Through the use of multi-scale and multi-part representation, XAGen can model details for small-scale regions like faces and hands. By adopting multi-part rendering, XAGen disentangles the learning process and produces realistic details for appearance and geometry. With multi-part discriminators, our model is capable of synthesizing high-quality human avatars with disentangled fine-grained control ability. The capabilities of XAGen open up a range of possibilities for downstream applications, such as text-guided avatar synthesis and audio-driven animation.
[1] O. Alexander, M. Rogers, W. Lambeth, J.-Y. Chiang, W.-C. Ma, C.-C. Wang, and P. Debevec. The digital emily project: Achieving a photorealistic digital actor. IEEE Computer Graphics and Applications, 2010.
[2] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scope: shape completion and animation of people. In ACM SIGGRAPH, 2005.
[3] T. Bagautdinov, C. Wu, T. Simon, F. Prada, T. Shiratori, S.-E. Wei, W. Xu, Y. Sheikh, and J. Saragih. Driving-signal aware full-body avatars. ACM Trans. on Graphics, 2021.
[4] T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross. High-quality passive facial performance capture using anchor frames. In ACM SIGGRAPH, 2011.
[5] A. Bergman, P. Kellnhofer, W. Yifan, E. Chan, D. Lindell, and G. Wetzstein. Generative neural articulated radiance fields. In NeurIPS, 2022.
[6] C. Bregler, J. Malik, and K. Pullen. Twist based acquisition and tracking of animal and human kinematics. Int'l. J. Computer Vision, 2004.
[7] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
[8] M. Chen, J. Zhang, X. Xu, L. Liu, Y. Cai, J. Feng, and S. Yan. Geometry-guided progressive nerf for generalizable and efficient neural human rendering. In ECCV, 2022.
[9] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In CVPR, 2021.
[10] H. Chu, S. Ma, F. De la Torre, S. Fidler, and Y. Sheikh. Expressive telepresence via modular codec avatars. In ECCV, 2020.
[11] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. ACM Trans. on Graphics, 2015.
[12] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun. Performance capture from sparse multi-view video. In ACM SIGGRAPH, 2008.
[13] Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In CVPR, 2020.
[14] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin. Towards multi-posed guided virtual try-on network. In CVPR, 2019.
[15] J. Dong, Q. Fang, Y. Guo, S. Peng, Q. Shuai, X. Zhou, and H. Bao. Totalselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies. In NeurIPS, 2022.
[16] Z. Dong, X. Chen, J. Yang, M. J. Black, O. Hilliges, and A. Geiger. Ag3d: Learning to generate 3d avatars from 2d image collections. In ICCV, 2023.
[17] Y. Feng, V. Choutas, T. Bolkart, D. Tzionas, and M. J. Black. Collaborative regression of expressive bodies using moderation. In 3DV, 2021.
[18] J. Fu, S. Li, Y. Jiang, K.-Y. Lin, C. Qian, C. C. Loy, W. Wu, and Z. Liu. Stylegan-human: A data-centric odyssey of human generation. In ECCV, 2022.
[61] Y. Xiu, J. Yang, D. Tzionas, and M. J. Black. Icon: implicit clothed humans obtained from normals. In CVPR, 2022.
[62] H. Xu, E. G. Bazavan, A. Zanfir, W. T. Freeman, R. Sukthankar, and C. Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR, 2020.
[63] H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR, 2023.
[64] H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In CVPR, 2023.
[65] Z. Xu, J. Zhang, J. Liew, W. Zhang, S. Bai, J. Feng, and M. Z. Shou. Pv3d: A 3d generative model for portrait video generation. In ICLR, 2023.
[66] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black. Generating holistic 3d human motion from speech. In CVPR, 2023.
[67] K. Youwang, K. Ji-Yeon, and T.-H. Oh. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In ECCV, 2022.
[68] P. Zablotskaia, A. Siarohin, B. Zhao, and L. Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. In BMVC, 2019.
[69] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng. Avatargen: a 3d generative model for animatable human avatars. In ECCV Workshop, 2023.
[70] J. Zhang, H. Yan, Z. Xu, J. Feng, and J. H. Liew. Magicavatar: Multimodal avatar generation and animation. arXiv, 2023.
[71] X. Zhang, J. Zhang, C. Rohan, H. Xu, G. Song, Y. Yang, and J. Feng. Getavator: Generative textured meshes for animatable human avatars. In ICCV, 2023.