SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zechuan Zhang $^{1}$ , Li Sun $^{1}$ , Zongxin Yang $^{1}$ , Ling Chen $^{2}$ , Yi Yang $^{1\dagger}$
ReLER, CCAI, Zhejiang University
$^{2}$  AAII, University of Technology Sydney
Reconstructing 3D clothed avatars from single images is a challenging task, especially when encountering complex poses and loose clothing. Current methods exhibit limitations in performance, largely attributable to their dependence on insufficient 2D image features and inconsistent query methods. Owing to this, we present Global-correlated 3D-decoupling Transformer for clothed Avatar reconstruction (GTA), a novel transformer-based architecture that reconstructs clothed human avatars from monocular images. Our approach leverages transformer architectures by utilizing a Vision Transformer model as an encoder for capturing global-correlated image features. Subsequently, our innovative 3D-decoupling decoder employs cross-attention to decouple tri-plane features, using learnable embeddings as queries for cross-plane generation. To effectively enhance feature fusion with the tri-plane 3D feature and human body prior, we propose a hybrid prior fusion strategy combining spatial and prior-enhanced queries, leveraging the benefits of spatial localization and human body prior knowledge. Comprehensive experiments on CAPE and THuman2.0 datasets illustrate that our method outperforms state-of-the-art approaches in both geometry and texture reconstruction, exhibiting high robustness to challenging poses and loose clothing, and producing higher-resolution textures. Codes are available at https://github.com/River-Zhang/GTA.
As virtual worlds and metaverse technology gain popularity, the demand for advanced techniques to reconstruct 3D clothed human avatars from single images is rapidly increasing. These techniques [1, 2, 3, 4, 5, 6, 7, 8, 9] are employed across various areas, such as AR/VR, social telepresence, virtual try-on, or the movie industry. However, in-the-wild images often present challenges, such as loose clothing and complex poses, which are not typically found in training data. As a result, there is a pressing need for models that can effectively generalize to these scenarios and reconstruct accurate, animatable, and high-resolution 3D human avatars.
In light of the significant progress made in 3D clothed human avatar reconstruction, existing models still face two main limitations: (i) Overreliance on 2D image features. Sole dependence on 2D CNN-based features compromises the accuracy of 3D object reconstructions due to the lack of global correlation. Despite the integration of 3D features from human body priors in methods like [3, 2, 7, 10], their inconsistent performance with loose clothing and challenging poses (See Fig. 2) indicates insufficient integration. Additionally, optimization-based methods [4, 11, 12, 13] can be complex and prone to errors, reducing reliability. (ii) Inconsistent query methods. Current strategies for querying features differ and have drawbacks. The pixel-aligned method [1, 6] directly
†: the corresponding author.
Monocular Image
Texture
Figure 1: Given a monocular image as input, GTA reconstructs the full 3D geometry and texture of the subject portrayed, allowing for various applications such as virtual try-on and animation.
Virtual Try-on
projects query points on feature maps but lacks guidance from human body prior, while the prior-guided strategy [7] integrates features on a human body prior but may lead to loss of detailed information on the image and result in 3D avatar reconstructions with an increased level of fuzziness.
Considering the limitations discussed above, we propose that 2D feature maps are insufficient for 3D reconstruction tasks, while global-correlated 3D feature representations offer a more effective solution. Traditional 3D representations are space-intensive and inefficient, necessitating alternatives such as the memory-conserving tri-plane representation [14]. However, generating global-correlated 3D representations from monocular images remains challenging due to difficulties in obtaining orthogonal plane feature maps. Our approach employs learnable embeddings and cross-attention mechanisms to effectively model intricate cross-plane relationships, enabling robust and precise 3D feature extraction. Furthermore, it is important to develop a strategy that synergizes various query methods while maintaining simplicity and efficiency. By combining existing strategies for 3D features, our method leverages localized spatial features and prior knowledge of human body structure, resulting in a balanced feature extraction process that improves reconstruction performance.
In response to the identified challenges, we present GTA (Global-correlated 3D-decoupling Transformer for clothed Avatar reconstruction), employing a novel global-correlated 3D-decoupling transformer and a hybrid prior fusion strategy for comprehensive 3D geometry and texture reconstruction. Our vision transformer-based encoder extracts global-correlated features from the input image, while our unique 3D-decoupling decoder disentangles tri-plane 3D features using learnable embeddings as queries. This integration of global-correlated encoding and 3D-decoupling decoding effectively captures the 3D avatar structure from a single image. To further enhance feature fusion, our hybrid prior fusion strategy combines spatial and prior-enhanced queries, leveraging the benefits of spatial localization and human body prior knowledge. This efficient and accurate integration strategy achieves state-of-the-art performance in single-view human avatar reconstruction.
Our proposed model, trained on THuman2.0 [15], outperforms state-of-the-art(SOTA) methods in geometry and texture reconstruction. We achieve a significant reduction in Chamfer distance on CAPE-FP [16] test dataset, below  $0.8\mathrm{cm}$  for the first time, and demonstrate superior side-view normal performance, illustrating our method's efficacy in reconstructing accurate 3D clothed human avatars. Our model excels in handling complex poses and loose clothing, and attains state-of-the-art texture reconstruction with higher PSNR scores. Moreover, it can be extended to animation and virtual try-on applications, showcasing its wide-ranging real-world potential. Our main contributions include:
- We introduce a novel global-correlated 3D-decoupling transformer that effectively disentangles tri-plane features, thereby substantially enhancing the reconstruction of clothed avatars from 2D images. To the best of our knowledge, our approach is the pioneering application of transformers in 3D feature decoupling for monocular human avatar reconstruction tasks.
- We put forward an innovative hybrid prior fusion strategy for feature query, combining spatial query's localization capabilities with prior-enhanced query's ability to incorporate knowledge of the human body prior, ultimately leading to improved geometry and texture reconstruction performance.
- Our proposed model achieves state-of-the-art performance in both clothed human geometry and texture reconstruction, outperforming previous models and exhibiting enhanced side-view normal performance.
Figure 2: GTA vs. SOTA. SOTA methods (red) are vulnerable to challenging poses and loose clothing, leading to artifacts such as non-human shapes (PIFu [1], PaMIR [3]), incomplete clothing reconstruction (ICON [2]), and erroneous stitching (ECON [4]). GTA deals with these challenges and produces high-quality results (blue).
Monocular Human Reconstruction has been an active area of research for many years. This task is inherently ill-posed due to the lack of 3D information, requiring additional assumptions or prior knowledge to recover the full 3D structure. Previous research has proposed effective parametric human prior models [17, 18, 19, 20, 21], which employs statistical methods to reduce the variations in human body shape and pose to a compact set of parameters. By leveraging this model, subsequent research has proposed novel methods to estimate or regress the model parameters from a single RGB image [22, 23, 24, 25, 26]. However, the human prior models can only capture a minimally clothed body without complex details like garments, adornments, or hairstyles. To address this limitation, some researchers add offsets on the top of prior body vertices to simulate outfits [27, 28, 29, 30, 31]. While these methods can effectively represent clothing close to the body surface and use blending weights of surface vertices to drive the clothing, they are not suitable for geometry topology far from the human body, such as robes and dresses.
In order to overcome the constraints imposed on reconstruction by clothing shape and type, researchers have explored various alternative representations for the human body, including voxels [32, 33], visual hulls [34], double depth maps [11, 12, 13, 4], and UV maps translation [35]. Among these diverse methods, implicit function-based methods [36] have shown the most remarkable performance. Saito et al. introduced PIFu [1], which firstly incorporates implicit functions into the problem of human body reconstruction. The method leverages a CNN-based neural network to extract features from 2D images and uses implicit functions to express the spatial geometry field, such as signed distance functions (SDF) [37] and occupancy fields [38]. While implicit function-based methods [1, 39, 6] can accurately reconstruct the complex topology of clothed human body surfaces, they may generate non-anatomical shapes for out-of-distribution poses due to the lack of regularization.
To improve pose robustness, recent research [3, 2, 4, 5, 7] has utilized the prior knowledge to guide implicit function representation. These methods have shown promising results in enhancing the quality and accuracy of reconstruction geometry, particularly for challenging poses. However, these methods, like previous ones, still rely on 2D features extracted from CNN-based networks, even though some of them incorporate 3D features obtained from human body prior. In the reconstruction process, the feature obtained by 2D projection may result in incomplete reconstructions from other viewpoints and diminish overall reconstruction accuracy. Our method extracts global-correlated 3D-aware feature to efficiently represent the clothed human avatar.
Transformers in Vision. The transformer architecture, initially proposed by Vaswani et al. [40], has achieved immense success across domains like NLP, speech recognition, and multimodal applications. Inspired by this, many studies have attempted to adapt the transformer architecture to the field of computer vision. Among these explorations, the Vision Transformer (ViT) proposed by Dosovitskiy et al. [41] has shown impressive performance in 2D visual tasks. Meanwhile, transformer's ability to model global and long-range correlation is also suitable for 3D vision tasks. Therefore, we leverage a ViT-based 3D transformer with cross-plane attention to efficiently extract global-correlated 3D features for better human reconstruction.
Generative 3D-aware Feature. Recent studies [14] have proposed the tri-plane 3D feature representation method, which efficiently extracts features from objects in three orthogonal orientations. Tri-plane representation has been demonstrated high efficacy in generating 3D objects [42], particu
Figure 3: GTA Overview. GTA has two key modules: (1) the global-correlated 3D-decoupling transformer and (2) the hybrid prior fusion strategy. The former module extracts a latent  $\pmb{h}$  from the input image and integrates it with learnable embeddings  $\pmb{z}$  via 3D-decoupling decoders, producing disentangled tri-plane features. The latter module employs spatial (red) and prior-enhanced (green) queries to merge features from the two tri-planes, thus enabling geometry and texture reconstruction.
larly 3D human bodies [43, 44, 45]. Thus, the tri-plane representation method holds potential for application in human body reconstruction. Nevertheless, the challenge of establishing a reasonable relationship between the input monocular image and the three planes of features remains an unresolved problem. In our method, we introduce learnable embeddings to represent features of spatial planes that are not directly visible, and utilize cross-attention mechanisms to establish relationships between the input image and other planes.
We introduce an implicit function-based framework for reconstructing 3D clothed human models from a single image (See Fig. 3). Our model employs a global-correlated 3D-decoupling transformer to disentangle tri-plane features and harnesses a hybrid prior fusion strategy for reconstructing the full 3D geometry and texture of the clothed avatar. In the following sections, we will discuss the preliminaries of GTA in Sec. 3.1, the global-correlated 3D-decoupling transformer in Sec. 3.2, and the hybrid prior fusion strategy in Sec. 3.3.
SMPL. The Skinned Multi-Person Linear (SMPL) model [18] is a widely-used parametric human body model. The SMPL model utilizes shape parameters  $\beta \in \mathbb{R}^{10}$  and pose parameters  $\theta \in \mathbb{R}^{3\times K}$  to parameterize the deformation of the human body mesh  $\mathcal{M}$ :
$$
\mathcal {M} (\boldsymbol {\beta}, \boldsymbol {\theta}): \boldsymbol {\beta} \times \boldsymbol {\theta} \mapsto \mathbb {R} ^ {3 \times N} \tag {1}
$$
where  $K = 24$  joints and  $N = 6890$  vertices. Shape parameter  $\beta$  describes the body's overall size and proportions and pose parameter  $\theta$  defines the positions and orientations of the joints relative to their default positions. SMPL enables effective representation and manipulation of human body shape and pose in various applications.
Implicit Function. Implicit function is a powerful tool for modeling complex geometries with neural networks. Our implicit function maps an input point to a scalar value that represents the spatial field including occupancy field and color field. The occupancy field takes a point in space as input and outputs a binary value indicating whether the point is inside or outside the human surface. Our reconstructed human surface can be represented as  $S_{\mathcal{I}\mathcal{F}}$ :
$$
\mathcal {S} _ {\mathcal {I F}} = \left\{\boldsymbol {x} \in \mathbb {R} ^ {3} \mid \mathcal {I F} (\boldsymbol {x}) = (o, c) \right\} \tag {2}
$$
where occupancy  $o = 0.5$ , color  $\pmb{c} \in \mathbb{R}^3$ , and  $\mathcal{IF}$  represents the implicit function.
3.2 Global-correlated 3D-decoupling Transformer
Directly extracting 3D information from a single 2D image is not feasible, this is largely due to the fact that 3D features include information from planes orthogonal with the image plane (also noted as  $xy$ -plane, principle-plane). To effectively decouple cross-plane features from a monocular image input, it is crucial to have additional guiding information. This is where the idea of learnable embeddings and the use of a cross-attention mechanism prove valuable. Inspired by this, we devise a novel transformer-based architecture including a global-correlated encoder and a 3D-decoupling decoder to disentangle 3D features from single input images.
Global-correlated Encoder. In our method, we employ a vision transformer to encode the input image and capture global correlations in the image, resulting in high-dimensional global-correlated image features. Our encoder module processes the input image by dividing it into non-overlapping  $n \times n$  patches and subsequently mapping them to image features through transformer blocks. This procedure generates a latent  $h$  for a image  $I$ .
3D-decoupling Decoder. To decode 3D tri-plane features from the encoder output, we propose to use two types of decoders: the principle-plane decoder and the cross-plane decoder. Our principle-plane decoder generates  $xy$ -plane features, which share the same plane as the input image. This decoder effectively reverses the encoding operation, leveraging a self-attention mechanism on the encoder output and converting the image features into a principal feature map  $F_{xy} \in \mathbb{R}^{H \times W \times C}$ .
In order to generate plane features orthogonal with the principle plane while preserving global correlation with the principle plane, we employ the cross-plane decoder to decode  $yz$  and  $xz$  plane features from input image features. To guide the decoder in decoding features from different planes, we introduce a learnable embedding  $z$  that supplies additional information for decoupling new planes. The learnable embedding  $z$  is first processed through self-attention encoding. It is then used as a query in a multi-head cross-attention mechanism with the output image latent  $h$  from the encoder stack. The image features are converted into keys and values for the cross-attention mechanism,
$$
\operatorname {C r o s s A t t n} (z, h) = \operatorname {S o f t m a x} \left(\frac {\left(W ^ {Q} \operatorname {S e l f A t t n} (z)\right) \left(W ^ {K} h\right) ^ {T}}{\sqrt {d}}\right) \left(W ^ {V} h\right) \tag {3}
$$
where  $W^{Q}, W^{K}$ , and  $W^{V}$  are learnable parameters and  $d$  is the scaling coefficient. Following the original transformer architecture [40], our model employs residual connections [46] and layer normalization [47] after each sub-layer. The entire decoder consists of multiple identical layers, and we use two such decoders to produce feature maps  $F_{yz} \in \mathbb{R}^{H \times W \times C}$  and  $F_{xz} \in \mathbb{R}^{H \times W \times C}$ .
Principle-plane Refinement. In accordance with the approach demonstrated in [6], higher-resolution feature maps play a crucial role in producing detailed geometry and sharper textures. Hence, we use both the original image and the principle-plane feature map, to produce a higher-resolution feature map. The original image is initially down-convoluted to match the principle-plane size and then concatenated along the channel dimension. Subsequently, they are fed into a streamlined Hourglass network and a super-resolution module for refinement. This process generates a higher-resolution feature map  $F_{xy}^{refine} \in \mathbb{R}^{2H \times 2W \times C}$ .
$$
F _ {x y} ^ {\text {r e f i n e}} = \operatorname {S u p e r R e s} (\operatorname {H o u r g l a s s} (\operatorname {D o w n C o n v} (I) \odot F _ {x y})) \tag {4}
$$
where  $\odot$  means concatenation operation. The resulting  $xy$  plane (principle-plane) exhibits a higher resolution than the  $yz$  and  $xz$  planes and incorporates more information from the original image, leading to higher fidelity reconstruction.
Tri-plane Division. After obtaining each plane, we evenly divide the plane features along the channel dimension into two groups, creating two tri-planes. For one group, we perform a spatial query to acquire features for query points, while for the other group, we utilize a prior-enhanced query to integrate the human body prior. Please refer to the next section for a detailed introduction to our novel hybrid prior fusion strategy.
Figure 4: Qualitative 3D human reconstruction for real images showcasing diverse poses and clothing variations. For each example, we show the input image along with two views of the reconstructed geometry and front view of the reconstructed texture. Our approach is robust to challenging poses and loose clothing, and contains detailed geometry and texture. See SupMat. for more results.
3.3 Hybrid Prior Fusion Strategy
In previous works, two primary methods have been utilized for acquiring query point features, each with significant limitations as previously discussed in Sec. 1. To address this, we propose a hybrid prior fusion strategy that combines the strengths of both spatial query and prior-enhanced query.
Spatial Query. Following [14], we extend the pixel-aligned query into 3D space, denoted as spatial query. This method projects query points onto the  $xy$ ,  $yz$ , and  $xz$  planes of a tri-plane group, producing localized features that capture important details for reconstruction. We combine the  $F_{yz}$  and  $F_{xz}$  features by summation and concatenate the result with  $F_{xy}^{refine}$  to generate the spatial query feature  $F_{SQ}(\pmb{x})$ :
$$
F ^ {S Q} (\boldsymbol {x}) = F _ {x y} ^ {S Q} (\boldsymbol {x}) \text {©} \left(F _ {y z} ^ {S Q} (\boldsymbol {x}) + F _ {x z} ^ {S Q} (\boldsymbol {x})\right) \tag {5}
$$
where  $F_{xy}^{SQ}(\pmb{x}), F_{yz}^{SQ}(\pmb{x}), F_{xz}^{SQ}(\pmb{x})$  are extracted from  $F_{xy}^{refine}, F_{yz}, F_{xz}$ , and  $\odot$  is concatenation.
Prior-enhanced Query. For the other tri-plane, we project the human body prior [18, 19] mesh vertices onto the three planes similar to the spatial query above to obtain the feature  $F^{PQ}(\pmb{v})$ ,  $\pmb{v} \in \mathcal{M}$ , where  $\mathcal{M}$  is the body prior mesh. For each query point  $\pmb{x}$ , we find the nearest triangular face  $t_{\pmb{x}} = [\pmb{v}_0, \pmb{v}_1, \pmb{v}_2] \in \mathbb{R}^{3 \times 3}$  and use barycentric interpolation to integrate features for  $\pmb{x}$  (See Fig. 3), denoted as  $F_{PQ}(\pmb{x})$ :
$$
F ^ {P Q} (\boldsymbol {x}) = u F ^ {P Q} \left(\boldsymbol {v} _ {0}\right) + v F ^ {P Q} \left(\boldsymbol {v} _ {1}\right) + w F ^ {P Q} \left(\boldsymbol {v} _ {2}\right) \tag {6}
$$
where  $[u, v, w]$  represents the barycentric coordinates of the query point  $x$  projected onto triangle  $t_x$ .
Hybrid Prior Fusion Strategy. Spatial query projects the query points directly onto tri-plane features, providing detailed information but lacking prior knowledge. On the other hand, the prior-enhanced query merges body prior information but may causes an increased level of fuzziness. Therefore, we concatenate these two query features to capitalize on each method's strengths and compensate for their weaknesses. Furthermore, we also include the signed distance between query point and human prior mesh  $SDF_{Prior}(\pmb{x})$  and pixel-aligned normal feature  $F_{\mathcal{N}}(\pmb{x})$  as input to the implicit function for predicting occupancy and color. Consequently, the reconstructed human surface  $S_{\mathcal{I}\mathcal{F}}$  can be represented as:
Table 1: Quantitative comparison on geometry against other methods. *: obtained from [2, 4].
Method	Training Data	CAPE-NFP [16]	CAPE-FP [16]	THuman2.0 [15]
Chamfer ↓	P2S↓	Normals↓	Chamfer ↓	P2S↓	Normals↓	Chamfer ↓	P2S↓	Normals↓
PIFu [1]	THuman2.0 [15]	2.458	2.117	0.094	1.786	1.639	0.071	1.586	1.530	0.088
PIFu* [1]	Renderpeople [48]	2.973	2.940	0.111	2.100	2.093	0.091	-	-	-
PIFuHD* [6]	Renderpeople [48]	3.767	3.591	0.123	2.302	2.335	0.090	-	-	-
PaMIR [3]	THuman2.0 [15]	1.603	1.429	0.068	1.502	1.291	0.064	1.276	1.247	0.080
PaMIR* [3]	Renderpeople [48]	1.413	1.321	0.063	1.225	1.206	0.055	-	-	-
ICON [2]	THuman2.0 [15]	1.096	1.085	0.046	0.969	0.987	0.041	1.249	1.368	0.076
ICON* [2]	Renderpeople [48]	1.070	1.013	0.059	1.202	1.170	0.055	-	-	-
ECON [4]	THuman2.0 [15]	0.942	0.933	0.035	0.904	0.894	0.033	2.120	1.807	0.074
ECON* [4]	THuman2.0 [15]	0.926	0.917	0.037	-	-	-	-	-	-
Ours	THuman2.0 [15]	0.911	0.917	0.042	0.763	0.763	0.035	0.814	0.862	0.055
$$
\mathcal {S} _ {\mathcal {I F}} = \left\{\boldsymbol {x} \in \mathbb {R} ^ {3} \mid \mathcal {I F} \left(F ^ {S Q} (\boldsymbol {x}), F ^ {P Q} (\boldsymbol {x}), \mathcal {S D F} _ {\text {P r i o r}} (\boldsymbol {x}), F _ {\mathcal {N}} (\boldsymbol {x})\right) = (o, c) \right\} \tag {7}
$$
where occupancy  $o = 0.5$ , color  $c \in \mathbb{R}^3$ , and  $\mathcal{L}\mathcal{F}$  represents the implicit function.
Training Objectives. For each 3D scan, we consider two sets of points as training data, denoted as  $G_{o}$  and  $G_{c}$ .  $G_{c}$  is sampled uniformly with a slight perturbation along the normals of the mesh surface, whereas  $G_{o}$  is sampled according to the same strategy as in PIFu [1], where points are sampled near the mesh surface and throughout the entire space.
For the points in  $G_{o}$ , we employ the following loss function:
$$
\mathcal {L} _ {o} = \frac {1}{| G _ {o} |} \sum_ {\boldsymbol {x} \in G _ {o}} B C E \left(\hat {o} _ {\boldsymbol {x}} - o _ {\boldsymbol {x}}\right) \tag {8}
$$
where  $\hat{o}_{\pmb{x}}$  denotes the model's predicted occupancy, while  $o_{x}$  signifies the ground truth occupancy. For the sampled points in  $G_{c}$ , we apply the following loss function: