SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {c} = \frac {1}{| G _ {c} |} \sum_ {\boldsymbol {x} \in G _ {c}} | \hat {\boldsymbol {c}} _ {\boldsymbol {x}} - \boldsymbol {c} _ {\boldsymbol {x}} | \tag {9}
$$
where  $\hat{c}_x$  represents the predicted color at location  $x$  by the model, while  $c_x$  indicates the true color of the mesh at  $x$ . The overall loss function is expressed by:
$$
\mathcal {L} _ {G T A} = \mathcal {L} _ {o} + \mathcal {L} _ {c} \tag {10}
$$
Strategy for Point Sampling. In the context of each training subject, our approach involves obtaining 2048 points for occupancy, denoted as  $G_{o}$ , and 2048 points for color, symbolized as  $G_{c}$ . The method for occupancy point sampling is aligned with the strategy illustrated in [1]. Color points are sampled uniformly, with a minor Gaussian disturbance, expressed as  $\mathcal{N}(0,\sigma)$ , wherein our experiment  $\sigma$  is set at  $0.1\mathrm{cm}$ . This disturbance occurs along the normals of the mesh surface. We obtain labels for the ground truth geometry, which specify whether a point is inside or outside the surface, through the application of Kaolin [49] to ascertain if a point lies within the ground truth mesh. The source of the ground truth color labels is the UV texture map of the 3D meshes.
Model Structure. To generate global-correlated latent features, we utilize a Vision Transformer (ViT) [41] model of depth 6, functioning as our global-correlated encoder, and generating an output of size  $1024 \times 256$ . Our 3D-decoupling decoder incorporates both cross-plane and principal-plane decoders, each with a depth of three. The cross-plane decoder is initialized with learnable embeddings that experience a Gaussian perturbation to align cohesively with the encoder's output shape. The configuration of the cross-plane decoder corresponds with the structure described in [40], while the principal-plane decoder emulates the global-correlated encoder. Each decoder outputs a feature map  $F \in \mathbb{R}^{128 \times 128 \times 64}$ . During refinement, a 2-stack hourglass and a transpose convolution module are integrated to generate a higher resolution principal-plane feature map  $F_{xy}^{refine} \in \mathbb{R}^{128 \times 128 \times 64}$ . Following the feature acquisition through our hybrid prior fusion method, two identical Multilayer
Perceptrons (MLPs) are employed for separate predictions of occupancy and color, each with layer sizes of [512, 1024, 512, 256, 128, 1]. In the inference phase, we utilize Rembg [50] for background subtraction in in-the-wild images. The Marching Cubes algorithm [51] is employed for generating 3D meshes, while off-the-shelf models from ICON [2] are leveraged for the production of normal maps. This normal map is further processed through a 2-stack hourglass to achieve a size of  $128 \times 128 \times 6$ . Besides the front/back normal maps are also used as input into the encoder with the image. The model, implemented in PyTorch Lightning [52], is trained for 10 epochs with a learning rate of 1e-4 and a batch size of 4, over a span of 2 days on a single NVIDIA GeForce RTX 3090 GPU.
Datasets. Our model was trained on the THuman2.0 [15], featuring 526 high-quality human scans, with 505 designated for training and 21 for evaluation. Testing was primarily conducted on the CAPE [16] and THuman2.0, with the former divided into "CAPE-FP" and "CAPE-NFP" subsets to examine model generalization on different pose types. Further dataset and implementation details are available in SupMat.
Metrics. We employ Chamfer and point-to-surface (P2S) distances, capturing significant geometric errors, to evaluate the accuracy of the
Table 2: Normal Evaluation of Different Views on THuman2.0 [15]. These views are obtained by positioning a virtual camera at the front, left, back, right, above, and below the reconstructed human.
Method	Normals of Different Views	Average
Front	Left	Back	Right	Above	Below
PIFu [1]	0.053	0.143	0.046	0.109	0.067	0.066	0.081
PaMIR [3]	0.051	0.124	0.068	0.077	0.051	0.054	0.071
ICON [2]	0.074	0.091	0.0647	0.076	0.044	0.044	0.066
ECON [4]	0.043	0.123	0.045	0.083	0.050	0.041	0.064
Ours	0.048	0.069	0.044	0.061	0.035	0.040	0.050
reconstructed meshes. We assess the quality of local details and the efficacy of 3D features via the L2 error between normal images of reconstructed and ground-truth meshes from six views. Finally, the quality of texture prediction is measured using Peak Signal-to-Noise Ratio (PSNR), comparing images rendered from both reconstructed and ground-truth surfaces across different views.
Evaluation of Geometry. We compare our GTA model with body-agnostic methods like PIFu [1], PIFuHD [6], and body-aware methods such as PaMIR [3], ICON [2], and ECON [4]. Our evaluation is thorough, involving training and testing these models ourselves and incorporating testing results from [4, 2] for a comprehensive comparison. As depicted in Tab. 1, GTA excels in terms of Chamfer and P2S distances on images with out-of-distribution (OOD) poses and diverse clothing. Notably, GTA is the first to reduce the Chamfer distance to less than  $0.8\mathrm{cm}$  on CAPE-FP. On par with ECON for normals on CAPE, GTA sets a new state-of-the-art on THuman2.0. Fig. 8a visually underlines our model's superior performance on the THuman2.0 benchmarks.
Quantitative Evaluation of Side-face Reconstruction. In our novel quantitative evaluation of side-face reconstruction, we spotlight the advantages of 3D features in crafting plausible side-faces and accurate thickness. Using six virtual camera angles, we render normal images of the reconstructed human and compute the normal difference for each face on the THuman2.0 [15] dataset. As shown in Tab. 2, GTA surpasses other models in 5 out of 6 views, matching ECON only in the front view, thereby underscoring our model's prowess in capturing inherent 3D structures within an image. For visual results, refer to Fig. 2 and SupMat.
Evaluation of Texture. In evaluating texture reconstruction, we compare GTA with color-predicting models like PIFu [1], ARCH [8], ARCH++ [9], PHORHUM [5], and S3F [7]. By rendering textured meshes from multiple angles and calculating the PSNR with respect to ground truth images, we find GTA outperforms other models on THuman2.0. As Fig. 5 demonstrates, GTA exceeds the state-of-the-art S3F [7] by  $22\%$  in PSNR. Notably, our model provides superior textures on the front side and accurately predicts invisible regions, as seen in Fig. 8b, highlighting the effectiveness of 3D features.
Figure 5: Quantitative Evaluation of Texture on THuman2.0 [15].
Ablation Details. To ensure a fair and unbiased experimental setup, we employed an approach wherein the implicit functions of each ablation experiment were augmented with front and back
Table 3: Ablation study of several of our designs on CAPE [16] dataset.
(a) Different networks.
Method	Chamfer↓	P2S↓	Normals↓
use convolution	0.991	0.968	0.055
w/o cross-atten	0.937	0.922	0.051
w/o refine	0.890	0.882	0.053
Ours	0.861	0.866	0.045
(b) 2D features vs. 3D features.
Method	Chamfer↓	P2S↓	Normals↓
2D+SQ	1.054	1.052	0.054
2D+PQ	1.133	1.116	0.053
2D+hybrid	1.008	0.967	0.051
Ours	0.861	0.866	0.045
(c) Different query methods.
Method	Chamfer↓	P2S↓	Normals↓
3D+SQ	0.987	0.965	0.049
3D+PQ	1.059	0.987	0.048
Ours	0.861	0.866	0.045
Input
use convolution
w/o cross-attn
w/o refine
2D+SQ
2D+PQ
2D+hybrid
3D+SQ
Figure 7: Application of our model in animation and virtual try-on.
Figure 6: Texture change of different ablation settings.
Animation
normal features and signed distances from the human prior body. For the ablation experiments pertaining to different network architectures, we replaced the corresponding components of our network structure with three identical UNet [53] and Vision Transformer [41] decoders solely based on self-attention, respectively. Moreover, for the ablations on 2D features, we generated a 2D feature map of size  $256 \times 256$  with UNet. Additionally, for the ablations on the hybrid prior fusion strategy, we employed a single tri-plane for the spacial query and the prior-enhanced query, respectively.
A. Different Networks for 3D Feature Decoupling. We evaluate alternative architectures by modifying our global-correlated encoder and 3D-decoupling decoder, confirming the strength of our proposed transformer. We experiment with UNet [53], also used in [7], as a convolution filter representative, and employ three separate UNets to compose the tri-plane. We also test a transformer encoder-decoder with only self-attention and without the refinement module or additional learnable embeddings. Results (see Tab. 3 and Fig. 6) suggest that a purely convolution-based network struggles to decouple 3D features due to limited correlation and receptive field constraints. While the refinement module shows minor geometric improvements, it is crucial for texture reconstruction. Additional learnable embeddings and cross-attention blocks notably enhance geometry results.
B. 2D Features vs. 3D Features. We analyze the effectiveness of 3D features by conducting an ablation study using solely 2D feature maps for reconstruction. Utilizing UNet [53] to generate high-dimensional 2D features, we compare spatial query (SQ), prior-enhanced query (PQ), and our hybrid prior fusion strategy. Results highlight the inferiority of 2D features in producing accurate reconstructions (Tab. 3), emphasizing the importance of 3D features.
C. Hybrid Prior Fusion Strategy vs. Others. We evaluate our hybrid prior fusion strategy against individual use of spatial query (SQ) and prior-enhanced query (PQ). Results show that spatial query surpasses prior-enhanced query in both geometry and texture quality due to its provision of localized features for detailed reconstruction. However, combining both methods optimizes geometry performance while preserving texture quality, demonstrating the effectiveness of our hybrid strategy.
Reconstruction of Images in-the-wild. The GTA model demonstrates significant prowess in reconstructing 3D human meshes from unconstrained, real-world images (refer to Fig. 4 and SupMat.), addressing the complexities posed by varied poses and clothing styles. This capability of recon
structuring high-fidelity 3D models from in-the-wild images paves the way for extensive applications, notably in virtual and augmented reality.
Animation and Virtual Try-On. We present a robust approach for generating novel poses of 3D clothed human meshes, catering to applications in animation and virtual try-on (See Fig. 7). We extend the S3F [7] model by employing estimated body shape and pose parameters to derive tri-plane features, facilitating realistic deformations. For a single-image clothed human reconstruction, our method excels by only needing the target pose, thus overcoming the limitations of previous deep learning methods. Additionally, our model supports virtual try-on by enabling feature replacement across body parts of different parametric bodies. By selectively interchanging these features, we can simulate changes in clothing on the target image. Our method, therefore, provides a versatile solution for both animation and virtual try-on applications, merging the strengths of previous methods while alleviating their weaknesses. More technical details and results are available in SupMat.
(a) Qualitative comparison of geometry.
(b) Qualitative comparison of texture.
Figure 8: Comparison of geometry and texture on THuman2.0 [15] benchmark.
In conclusion, we present the GTA model, a cutting-edge approach for reconstructing 3D clothed human from single images. Our global-correlated 3D-decoupling transformer effectively extracts latent representations from input images and integrates them with learnable embeddings through 3D-decoupling decoders, generating disentangled tri-plane features. Moreover, our hybrid prior fusion strategy integrates the benefits of spatial query's localization capabilities with the priorenhanced query's ability to incorporate knowledge of the human body prior, ultimately leading to improved geometry and texture reconstruction performance. We demonstrate that our proposed model outperforms state-of-the-art methods in geometric and texture reconstruction, exhibiting resilience against challenging poses and loose clothing, enabling a wide range of applications.
Acknowledgements. This work was supported in part by the Fundamental Research Funds for the Central Universities (No. 226-2023-00048) and the Natural Science Foundation of Zhejiang Province (DT23F020008).
[1] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[2] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13296-13306, June 2022.
[3] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence, 44(6):3170-3184, 2021.
[4] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[5] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1506-1515, 2022.
[6] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020.
[41] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[42] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854, 2022.
[43] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 597-614. Springer, 2022.
[44] Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. Advances in Neural Information Processing Systems, 35:1990-19916, 2022.
[45] Suyi Jiang, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. arXiv preprint arXiv:2212.05321, 2022.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[47] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[48] Renderpeople. https://renderpeople.com/.
[49] Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozant-sev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. https://github.com/NVIDIAGameWorks/kaolin, 2022.
[50] Daniel Gatis. Rembg: A tool to remove images background. https://github.com/danielgatis/rembg, 2022.
[51] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
[52] William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019.
[53] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.