SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongcong Xu
Show Lab
zhongcongxu@u.nus.edu
Jianfeng Zhang
jianfengzhang@bytedance.com
Jun Hao Liew
junhao.liew@bytedance.com
Jiashi Feng
jshfeng@bytedance.com
Mike Zheng Shou *
mike.zheng.shou@gmail.com
Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available at https://showlab.github.io/xagen.
3D avatars present an opportunity to create experiences that are exceptionally authentic and immersive in telepresence [10], augmented reality (AR) [22], and virtual reality (VR) [50]. These applications [1, 52, 3, 35] require the capture of human expressiveness, including poses, gestures, expressions, and others, to enable photo-realistic generation [65, 70], animation [56], and interaction [33] in virtual environments.
Traditional methods [11, 60, 4, 20, 23] typically create virtual avatars based on template registration or expensive multi-camera light stages in well-controlled environments. Recent efforts [69, 43, 5, 26, 16] have explored the use of generative models to produce 3D human bodies and clothing based on input parameters, such as SMPL [38], without the need of 3D supervision. Despite these advancements, current approaches are limited in their ability to handle expressive attributes of the human body, such
Figure 1: XAGen can synthesize realistic 3D avatars with detailed geometry, while providing disentangled control over expressive attributes, i.e., facial expressions, jaw, body, and hand poses.
as facial expressions and hand poses, as they primarily focus on body pose and shape conditions. Yet, there exist scenarios where fine-grained control ability is strongly desired, e.g., performing social interactions with non-verbal body languages in Metaverse, or driving digital characters to talk with various expressions and gestures, etc. Due to the lack of comprehensive modeling of the full human body, existing approaches [43, 5, 26] fail to provide control ability beyond the sparse joints of major body skeleton, leading to simple and unnatural animation.
In this work, our objective is to enhance the fine-grained control capabilities of GAN-based human avatar generation model. To achieve this, we introduce the first eXpressive 3D human Avatar Generation model (XAGen) that can (1) synthesize high-quality 3D human avatars with diverse realistic appearances and detailed geometries; (2) provide independent control capabilities for fine-grained attributes, including body poses, hand poses, jaw poses, shapes, and facial expressions.
XAGen is built upon recent unconditional 3D-aware generation models for static images [7, 44]. One straightforward approach to implement fully animatable avatar generation is extending 3D GAN models to condition on expressive control signals, such as SMPL-X [47]. Though conceptually simple, such a direct modification of conditioning signal cannot guarantee promising appearance quality and control ability, particularly for two crucial yet challenging regions, i.e., the face and hands. This is because (1) Compared with body, face and hands contain similar or even more articulations. In addition, their scales are much smaller than arms, torso, and legs in a human body image, which hinders the gradient propagation from supervision. (2) Face and hands are entangled with the articulated human body and thus will be severely affected by large body pose deformation, leading to optimization difficulty when training solely on full-body image collections.
To address the above challenges, we decompose the learning process of body, face, and hands by adopting a multi-scale and multi-part 3D representation and rendering multiple parts independently using their respective observation viewpoints and control parameters. The rendered images are passed to multi-part discriminators, which provide multi-scale supervision during the training process.
With these careful designs, XAGen can synthesize photo-realistic 3D human avatars that can be animated effectively by manipulating the corresponding control parameters for expressions and poses, as depicted in Figure 1. We conduct extensive experiments on a variety of benchmarks [18, 68, 14, 36], demonstrating the superiority of XAGen over state-of-the-arts in terms of appearance, geometry, and controllability. Moreover, XAGen supports various downstream applications such as text-guided avatar creation and audio-driven animation, expanding its potential for practical scenarios.
Our contributions are three-fold: (1) To the best of our knowledge, XAGen is the first 3D GAN model for fully animatable human avatar generation. (2) We propose a novel framework that incorporates
multi-scale and multi-part 3D representation together with multi-part rendering technique to enhance the quality and control ability, particularly for the face and hands. (3) Experiments demonstrate XAGen surpasses state-of-the-art methods in terms of both quality and controllability, which enables various downstream applications, including text-guided avatar synthesis and audio-driven animation.
Generative models for avatar creation. Generative models [27, 28, 51] have demonstrated unprecedented capability for synthesizing high-resolution photo-realistic images. Building upon these generative models, follow-up works [7, 44, 55, 59, 63] have focused on extending 2D image generation to the 3D domain by incorporating neural radiance field [42] or differentiable rasterization [29]. Although enabling 3D-aware generation, these works fail to provide control ability to manipulate the synthesized portrait images. To address this limitation, recent research efforts [64, 26, 43, 69, 57, 16, 71] have explored animatable 3D avatar generation leveraging parametric models for face [32] and body [38]. These works employ inverse [31] or forward [9] skinning techniques to control the facial attributes or body poses of the generated canonical avatars [69, 71]. For human body avatars, additional challenges arise due to their articulation properties. Consequently, generative models for human avatars have explored effective 3D representation designs. Among them, ENARF [43] divides an efficient 3D representation [7] into multiple parts, with each part representing one bone. EVA3D [26] employs a similar multi-part design by developing a compositional neural radiance field. Despite enabling body control, such representation fails to generate the details of human faces or hands since these parts only occupy small regions in the human body images.
Our method differs in two aspects. First, existing works can either control face or body, whereas ours is the first 3D avatar generation model with simultaneous fine-grained control over the face, body, and hands. Second, we devise a multi-scale and multi-part 3D representation, allowing for generating human body with high fidelity even for small regions like face and hands.
Expressive 3D human modeling. Existing 3D human reconstruction approaches can be categorized into two main categories depending on whether explicit or implicit representations are used. Explicit representations mainly utilize the pre-defined mesh topology, such as statistical parametric models [38, 62, 45, 2] or personalized mesh registrations [19, 12], to model naked human bodies with various poses and shapes. To enhance the expressiveness, recent works have developed expressive statistical models capable of representing details beyond major human body [47, 46, 17] or introduced the surface deformation to capture fine-grained features [30, 58]. On the other hand, leveraging the remarkable advances in implicit neural representations [41, 42], another line of research has proposed to either rely purely on implicit representations [53] or combine it with statistical models [61, 48, 8] to reconstruct expressive 3D human bodies. The most recent work [15, 54] proposed to learn a single full-body avatar from multi-part portrait videos or 3D scans. In contrast, our approach focuses on developing 3D generative model for fully animatable human avatars, which is trainable on only unstructured 2D image collections.
In this section, we introduce XAGen, a 3D generative model for synthesizing photo-realistic human avatars with expressive and disentangled controllability over facial expression, shape, jaw pose, body pose, and hand pose. Figure 2 depicts the pipeline of our method.
Given a random noise  $\mathbf{z}$  sampled from Gaussian distribution, XAGen first synthesizes a human avatar with canonical body, face, and hand configurations. In this work, we use X-pose [34] and neutral shape, face, and hand as canonical configurations. We leverage Tri-plane [7] as the fundamental building block of 3D representation in our canonical generator. To increase the capability of 3D representation for the smaller-scale face and hands, we introduce multi-part and multi-scale designs into the canonical Tri-plane (Sec. 3.1). A mapping network first encodes  $\mathbf{z}$  and the camera viewpoint of body  $c_{\mathrm{b}}$  into latent code  $\mathbf{w}$ . The canonical generator then synthesizes three Tri-planes  $\mathcal{F}_k$  conditioned on  $\mathbf{w}$ , where  $k \in \{\mathrm{b}, \mathrm{f}, \mathrm{h}\}$  which stands for  $\{\mathrm{body}, \mathrm{face}, \mathrm{hand}\}$ .
Based on the generated canonical avatar, we deform it from canonical space to observation space under the guidance of control signal  $p_{\mathrm{b}}$  parameterized by an expressive statistical full body model, i.e., SMPL-X [47]. We adopt volumetric rendering [39] to synthesize the full body image. However, due
Figure 2: Pipeline of XAGen. Given a random noise  $\mathbf{z}$ , the canonical generator synthesizes the avatar in the format of canonical multi-part and multi-scale Tri-planes given the corresponding camera pose  $c_{\mathrm{b}}$ . We then deform the canonical avatar under the guidance of control parameters  $p_*$  to render multi-part images using respective camera poses  $c_*$  and upsample the images using a super-resolution module. Discriminators encode the output images, camera poses, and control parameters into real or fake probabilities to critique the rendered images. IS represents inverse skinning.
to the scale imbalance between the face/hands and body, rendering only the full body image cannot guarantee quality for these detailed regions. To address this issue, we propose a multi-part rendering technique (Sec. 3.2). Specifically, we employ part-aware deformation and rendering based on the control parameters  $(p_{\mathrm{f}}$  and  $p_{\mathrm{h}})$  and cameras  $(c_{\mathrm{f}}$  and  $c_{\mathrm{h}})$ . Accordingly, to ensure the plausibility and controllability of the generated avatars, we develop multi-part discriminators to critique the rendered images (Sec. 3.3).
3.1 Multi-scale and Multi-part Representation
XAGen is designed for expressive human avatars with an emphasis on the high-quality face and hands. However, the scale imbalance between face/hands and body may hamper the fidelity of the corresponding regions. To address this issue, we propose a simple yet effective multi-scale and multi-part representation for expressive human avatar generation. Our multi-scale representation builds upon the efficient 3D representation, i.e., Tri-plane [7], which stores the generated features on three orthogonal planes. Specifically, we design three Tri-planes for body, face, and hands, denoted as  $\mathcal{F}_{\mathrm{b}} \in \mathbb{R}^{W_{\mathrm{b}} \times W_{\mathrm{b}} \times 3C}$ ,  $\mathcal{F}_{\mathrm{f}} \in \mathbb{R}^{W_{\mathrm{f}} \times W_{\mathrm{f}} \times 3C}$ , and  $\mathcal{F}_{\mathrm{h}} \in \mathbb{R}^{W_{\mathrm{h}} \times W_{\mathrm{h}} \times 3C}$ , respectively. The size of the face and hand Tri-planes is set to half of the body Tri-plane, with  $W_{\mathrm{f}} = W_{\mathrm{h}} = W_{\mathrm{b}} / 2$ .
As depicted in Figure 2, our canonical generator first synthesizes a compact feature map  $\mathcal{F} \in \mathbb{R}^{W_{\mathrm{b}} \times W_{\mathrm{b}} \times 9C / 2}$ , where  $C$  represents the number of channels. We then separate and reshape  $\mathcal{F}$  into  $\mathcal{F}_k$ , where  $k \in \{\mathrm{b}, \mathrm{f}, \mathrm{h}\}$ , representing the canonical space of the generated human avatar. Furthermore, to save computation cost, we exploit the symmetry property of hands to represent both left and right hands using one single  $\mathcal{F}_{\mathrm{h}}$  through a horizontal flip operation (refer to Appendix for details).
3.2 Multi-part Rendering
Our method is trainable on unstructured 2D human images. Although this largely reduces the difficulty and cost to obtain data, the training is highly under-constrained due to the presence of diverse poses, faces, and clothes. To facilitate the training process and improve the appearance quality, we propose a multi-part rendering strategy. This strategy allows XAGen to learn each part based on the independent camera poses, which further enhances the geometry quality of the face and hands. Specifically, for each training image, we utilize a pretrained model [17] to estimate SMPL-X parameters  $\{p_{\mathrm{b}}, p_{\mathrm{f}}, p_{\mathrm{h}}\}$  and camera poses  $\{c_{\mathrm{b}}, c_{\mathrm{f}}, c_{\mathrm{h}}\}$  for body, face, and hands, respectively. In the rendering stage, we shoot rays using  $\{c_{\mathrm{b}}, c_{\mathrm{f}}, c_{\mathrm{h}}\}$  and sample points  $\{\mathbf{x}_{\mathrm{o}}^{\mathrm{b}}, \mathbf{x}_{\mathrm{o}}^{\mathrm{f}}, \mathbf{x}_{\mathrm{o}}^{\mathrm{h}}\}$  along the rays in the observation space. To compute the feature for each point, we employ inverse linear-blend skinning [31], which finds the transformation of each point from observation space to canonical space produced by the canonical generator. Based on the parameter  $p_k$ , where  $k \in \{\mathrm{b}, \mathrm{f}, \mathrm{h}\}$ , SMPL-X yields an expressive human body model ( $\mathbf{v}, \mathbf{w}$ ), where  $\mathbf{v} \in \mathbb{R}^{N \times 3}$  represents  $N$  vertices, and  $\mathbf{w} \in \mathbb{R}^{N \times J}$  represents the skinning
weights of each vertex with respect to joint  $J$ . For each point  $\mathbf{x}_{\mathbf{o}}^{k,i}$ , where  $i = 1 \cdots M_k$  and  $M_k$  is the number of sampled points, we find its nearest neighbour  $\mathbf{n}$  from vertices  $\mathbf{v}$ . We then compute the corresponding transformation from observation space to canonical space
$$
T ^ {k, i} = \left(\sum_ {j} \mathbf {w} _ {j} ^ {\mathbf {n}} \left[ \begin{array}{l l} R _ {j} & t _ {j} \\ \mathbf {0} & \mathbf {1} \end{array} \right] \left[ \begin{array}{l l} \mathbf {I} & \Delta^ {\mathbf {n}} \\ \mathbf {0} & \mathbf {1} \end{array} \right]\right) ^ {- 1}, \tag {1}
$$
where  $j = 1 \dots J$ ,  $R_{j}$  and  $t_j$  are derived from  $p_k$  with Rodrigues formula [6], and  $\Delta^{\mathbf{n}}$  represents the offset caused by pose and shape for vertex  $\mathbf{n}$ , which is calculated by SMPL-X. Based on this inverse transformation, we can calculate the coordinates for each point in canonical space  $\mathbf{x}_{\mathbf{c}}^{k,i}$  as
$$
\mathbf {x} _ {\mathbf {c}} ^ {k, i} = T ^ {k, i} \mathbf {x} _ {\mathbf {o}} ^ {k, i}, \tag {2}
$$
where we apply homogeneous coordinates for the calculation.
For the face and hands rendering, i.e.,  $k \in \{\mathrm{f},\mathrm{h}\}$ , we directly interpolate their corresponding Tri-plane  $\mathcal{F}_{\mathrm{f}}$  and  $\mathcal{F}_{\mathrm{h}}$  to compute the feature  $\mathbf{f}_{\mathbf{c}}^{\mathrm{f},i}$  and  $\mathbf{f}_{\mathbf{c}}^{\mathrm{h},i}$ . Regarding the body rendering, we first define three bounding boxes  $\mathbb{B}_{\mathrm{f}},\mathbb{B}_{\mathrm{lh}},\mathbb{B}_{\mathrm{rh}}$  for face, left and right hands in canonical body space. Then, we query canonical body points that are outside these bounding boxes from body Tri-plane  $\mathcal{F}_{\mathrm{b}}$ , while the canonical points inside these boxes from  $\mathcal{F}_{\mathrm{f}}$  and  $\mathcal{F}_{\mathrm{h}}$ . The query process for body point  $\mathbf{x}_{\mathbf{c}}^{\mathrm{b},i}$  is mathematically formulated as
$$
\mathbf {f} _ {\mathbf {c}} ^ {\mathrm {b}, i} = \left\{ \begin{array}{l l} Q \left(\mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i}, \mathcal {F} _ {\mathrm {f}}\right), & \text {i f} \mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i} \in \mathbb {B} _ {\mathrm {f}}, \\ Q \left(\mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i}, \mathcal {F} _ {\mathrm {h}}\right), & \text {i f} \mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i} \in \left\{\mathbb {B} _ {\mathrm {r h}}, \mathbb {B} _ {\mathrm {l h}} \right\}, \\ Q \left(\mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i}, \mathcal {F} _ {\mathrm {b}}\right), & \text {i f} \mathbf {x} _ {\mathbf {c}} ^ {\mathrm {b}, i} \notin \left\{\mathbb {B} _ {\mathrm {f}}, \mathbb {B} _ {\mathrm {l h}}, \mathbb {B} _ {\mathrm {r h}} \right\}, \end{array} \right. \tag {3}
$$
where  $Q$  denotes querying the feature for the given point from the corresponding Tri-planes.
Once the features  $\mathbf{f}_{\mathbf{c}}^{k,i}$  are obtained, they are encoded into color  $\mathbf{c}$  and geometry  $d$  via two lightweight multi-layer perceptrons (MLP), where  $\mathbf{c} = \mathrm{MLP}_{\mathbf{c}}(\mathbf{f}_{\mathbf{c}}^{k,i})$ . Inspired by prior works [44, 26, 69], we employ signed distance field (SDF) as a proxy to model geometry. Additionally, following [26, 69], we also query a base SDF  $d_{\mathbf{c}}$  in the canonical space, and predict delta SDF, such that  $d = d_{\mathbf{c}} + \mathrm{MLP}_d(\mathbf{f}_{\mathbf{c}}^{k,i},d_{\mathbf{c}})$ . We then convert the SDF value into density  $\sigma = \frac{1}{\alpha}\mathrm{Sigmoid}(\frac{-d}{\alpha})$  for volume rendering, where  $\alpha$  is a learnable parameter.
To handle the body features queried from multiple Tri-planes, we apply feature composition on RGB and density using a window function [37] for smoothness transition. Specifically, if point  $\mathbf{x}_{\mathbf{c},\mathbf{b}}^{k,i}$  is located in the overlapping region between the body and other parts (face, right hand, and left hand), their features are sampled from both Tri-planes and linearly blended together. More details on the feature composition can be found in the Appendix. Finally, volume rendering is applied to synthesize raw images for body, face, and hands, denoted as  $\{I_{\mathrm{b}}^{\mathrm{raw}}, I_{\mathrm{f}}^{\mathrm{raw}}, I_{\mathrm{h}}^{\mathrm{raw}}\}$ . These raw images are then upsampled into high-resolution images  $\{I_{\mathrm{b}}, I_{\mathrm{f}}, I_{\mathrm{h}}\}$  by a super-resolution module.
3.3 Multi-part Discriminators
Based on the images synthesized by XAGen generator, we design a discriminator module to critique the generation results. To ensure both the fine-grained fidelity of appearance and geometry as well as disentangled control over the full body, including face and hands, we introduce multi-part discriminators to encode images  $\{I_{\mathrm{b}}, I_{\mathrm{f}}, I_{\mathrm{h}}\}$  into real-fake scores for adversarial training. As depicted in Figure 2, these discriminators are conditioned on the respective camera poses to encode 3D priors, resulting in improved geometries as demonstrated in our experiments. To enhance the control ability of the face and hands, we further condition face discriminator on expression and shape parameters  $[p_{\mathrm{f}}^{\psi}, p_{\mathrm{f}}^{\beta}]$ , and condition hand discriminator on hand pose  $p_{\mathrm{h}}^{\theta}$ . We encode the camera pose and condition parameters into intermediate embeddings by two separate MLPs and pass them to the discriminators. The multi-part discriminator is formulated as
$$
s _ {k} = \mathcal {D} _ {k} \left(I _ {k}, \mathrm {M L P} _ {k} ^ {c} \left(c _ {k}\right) + \mathrm {M L P} _ {k} ^ {p} \left(p _ {k} ^ {\prime}\right)\right), \text {w h e r e} p _ {k} ^ {\prime} = \left\{ \begin{array}{l l} \emptyset , & \text {i f} k = \mathrm {b} \\ \left[ p _ {\mathrm {f}} ^ {\psi}, p _ {\mathrm {f}} ^ {\beta} \right], & \text {i f} k = \mathrm {f}. \\ p _ {\mathrm {h}} ^ {\theta}, & \text {i f} k = \mathrm {h} \end{array} \right. \tag {4}
$$
Here  $s_k$  denotes the probability of each image  $I_k$  being sampled from real data, and  $\mathcal{D}_k$  refers to the discriminator corresponding to the specific body part  $k$ . For body part, no conditioning parameters are used because we empirically find that the condition for body hinders the learning of appearance.
3.4 Training Losses
The non-saturating GAN loss [21] is computed for each discriminator, resulting in  $L_{\mathrm{b}}$ ,  $L_{\mathrm{f}}$ , and  $L_{\mathrm{h}}$ . We also regularize these discriminators using R1 regularization loss [40]  $L_{\mathrm{R1}}$ . To improve the plausibility and smoothness of geometry, we compute minimal surface loss  $L_{\mathrm{Minsurf}}$ , Eikonal loss  $L_{\mathrm{Eik}}$ , and human prior regularization loss  $L_{\mathrm{Prior}}$  as suggested in previous works [44, 69].
Due to the occlusion in the full body images, some training samples may not contain visible faces or hands. Thus, we balance the loss terms for both generator and discriminator based on the visibility of face  $\mathcal{M}_{\mathrm{f}}$  and hands  $\mathcal{M}_{\mathrm{h}}$ , which denote whether face and hands are detected or not. The overall loss term of XAGen is formulated as
$$
\begin{array}{l} L _ {\mathcal {G}} = L _ {\mathrm {b}} ^ {\mathcal {G}} + \lambda_ {\mathrm {f}} \mathcal {M} _ {\mathrm {f}} \odot L _ {\mathrm {f}} ^ {\mathcal {G}} + \lambda_ {\mathrm {h}} \mathcal {M} _ {\mathrm {h}} ^ {\mathcal {G}} \odot L _ {\mathrm {h}} + \lambda_ {\text {M i n s u r f}} L _ {\text {M i n s u r f}} + \lambda_ {\text {E i k}} L _ {\text {E i k}} + \lambda_ {\text {P r i o r}} L _ {\text {P r i o r}}, \\ \begin{array}{l} L _ {\mathrm {b}} + \lambda_ {\mathrm {f}} \mathcal {M} _ {\mathrm {f}} \odot L _ {\mathrm {f}} + \lambda_ {\mathrm {h}} \mathcal {M} _ {\mathrm {h}} \odot L _ {\mathrm {h}} + \lambda_ {\text {M i n s u r f}} L _ {\text {M i n s u r f}} + \lambda_ {\text {E i k}} L _ {\text {E i k}} + \lambda_ {\text {P r i o r}} L _ {\text {P r i o r}}, \\ L _ {\mathcal {D}} = L _ {\mathrm {b}} ^ {\mathcal {D}} + L _ {\mathrm {R 1}} ^ {\mathrm {b}} + \lambda_ {\mathrm {f}} \mathcal {M} _ {\mathrm {f}} \odot \left(L _ {\mathrm {f}} ^ {\mathcal {D}} + L _ {\mathrm {R 1}} ^ {\mathrm {f}}\right) + \lambda_ {\mathrm {h}} \mathcal {M} _ {\mathrm {h}} \odot \left(L _ {\mathrm {h}} ^ {\mathcal {D}} + L _ {\mathrm {R 1}} ^ {\mathrm {h}}\right), \end{array} \tag {5} \\ \end{array}
$$
where  $\odot$  means instance-wise multiplication, and  $\lambda_{*}$  are the weighting factors for each term.
We evaluate the performance of XAGen on four datasets, i.e., DeepFashion [36], MPV [68], UBC [14], and SHHQ [18]. These datasets contain diverse full body images of clothed individuals. For each image in the dataset, we process it to obtain aligned body, face and hand crops, and their corresponding camera poses and SMPL-X parameters. Please refer to Appendix for more details.