Global-correlated 3D-decoupling Transformer for Clothed Avatar Reconstruction
Authors: Zechuan Zhang
Li Sun
Zongxin Yang
Ling Chen
Yi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D human avatar reconstruction
vision transformer
parametric body model
tri-plane representation
TL;DR: We propose a novel model for clothed avatar reconstruction from monocular input image.

Abstract: Reconstructing 3D clothed human avatars from single images is a challenging task, especially when encountering complex poses and loose clothing. Current methods exhibit limitations in performance, largely attributable to their dependence on insufficient 2D image features and inconsistent query methods. Owing to this, we present the Global-correlated 3D-decoupling Transformer for clothed Avatar reconstruction (GTA), a novel transformer-based architecture that reconstructs clothed human avatars from monocular images. Our approach leverages transformer architectures by utilizing a Vision Transformer model as an encoder for capturing global-correlated image features. Subsequently, our innovative 3D-decoupling decoder employs cross-attention to decouple tri-plane features, using learnable embeddings as queries for cross-plane generation. To effectively enhance feature fusion with the tri-plane 3D feature and human body prior, we propose a hybrid prior fusion strategy combining spatial and prior-enhanced queries, leveraging the benefits of spatial localization and human body prior knowledge. Comprehensive experiments on CAPE and THuman2.0 datasets illustrate that our method outperforms state-of-the-art approaches in both geometry and texture reconstruction, exhibiting high robustness to challenging poses and loose clothing, and producing higher-resolution textures.  Codes are available at https://github.com/River-Zhang/GTA.

Global-correlated 3D-decoupling Transformer for Clothed Avatar Reconstruction
Zechuan Zhang' Li Sun', Zongxin Yang', Ling Chen?, Yi Yang" if
ReLER, CCAI, Zhejiang University
2 AAII, University of Technology Sydney
Abstract
Reconstructing 3D clothed avatars from single images is challenging task, pecially when encountering complex poses and loose clothing. Current methods exhibit limitations in performance, largely attributable to their dependence on insufficient 2D image features and inconsistent query methods. Owing to this, we present Global-correlated 3D dD-decoupling Transformer for clothed A vatar recon- struction (GTA), a novel transformer- based architecture that reconstructs clothed human avatars from monocular images. Our approach leverages transformer archi- tectures by utilizing Vision Transformer model as an encoder for capturing global- correlated image features. Subsequently, employs cross-attention to decouple tri-plane features, using learnable embeddings as queries for cross-plane generation. To effectively enhance feature fusion with the tri-plane 3D feature and human body prior, we propose hybrid prior fusion strategy combining spatial and prior-enhanced queries, leveraging the benefits of spatial localization and human body prior knowledge. Comprehensive experiments on CAPE and THuman2.0 0 datasets illustrate that our method outperforms state-of of- the-art approaches in both geometry and texture reconstruction, exhibiting high robustness to challenging poses and loose clothing, and producing higher-resolue resolution textures. Codes are available at
innovative 3DDdecoupling decoder
1 Introduction
As virtual worlds and metaverse technology gain popularity, the demand for advanced techniques to reconstruct 3D clothed human avatars from single images is rapidly increasing. These techniques [1, 2,3, 6, 7, 8, 9] are employed across various areas, such as AR/VR, social telepresence, virtual try-on, the movie industry. However, in-the-wild images often present challenges, such loose clothing and complex poses, which are not typically found in training data. As a result, there is pressing need for models that can effectively generalize these scenarios and reconstruct accurate, animatable, and high-resolution 3D human avatars.
In light of the significant progress made in 3D clothed human avatar reconstruction, existing models still face two main limitations: (i) Overreliance on 2D image features. Sole dependence on 2D CNN-based features compromises the accuracy of 3D) object reconstructions due the lack of global correlation. Despite the integration of 3D features from human body priors in methods like [3, 2, 10], their inconsistent performance with loose clothing and challenging poses (See Fig. 2) indicates insufficient integration. Additionally, optimization-base
methods [4, 11, 12, 13] can be complex and prone errors, reducing reliability. (ii) Inconsistent query methods. Current strategies for querying features differ and have drawbacks. The pixel-aligned method [1, 6] directly
:: the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Monocular Image
Geometry
Texture
Virtual Try-on
Animation
Figure I:
Given monocular image as input, GTA reconstructs the full 3D geometry and texture of the subject portrayed, allowing for various applications such as virtual try-on and animation.
projects query points on feature maps but lacks guidance from human body prior, while the prior- guided strategy [7] integrates features on a human body prior but may lead loss of detailed information on the image and result in 3D avatar reconstructions with an increased level of fuzziness.
Considering the limitations discussed above, we propose that 2D feature maps insufficient for 3D reconstruction tasks, while global-correlated 3D feature representations offer a more effective solution. Traditional 3D representations are space-intensive and inefficient, necessitating alternatives such as the memory--cnssrvingg tri-plane representation [14]. However, generating global-correlated 3D representations from monocular images remains challenging due to difficulties obtaining orthogonal plane feature maps. Our approach employs learnable embeddings and cross cross-attention mechanisms to effectively model intricate cross-plamn plose relationships, enabling robust and precise 3D feature extraction. Furthermore, it is important to develop a strategy that synergizes various query methods while maintaining simplicity and efficiency. By combining existing strategies for 3D features, our method leverages localized spatial features and prior knowledge of human body structure, resulting in a balanced feature extraction process that improves reconstruction performance.
In response to the identified challenges, we present GTA (Global-correlated 3D-decoupling Transformer for clothed Avtarr reconstruction), employing novel global-correlated 3D-decoupling transformer and a hybrid prior fusion strategy for comprehensive 3D geometry and texture recon- struction. Our vision wansformer-basee encoder extracts global-correlated features from the input image, while our unique DDddecuugling decoder disentangles tri-plane 3D features using learnable embeddings as queries. This integration of global-correlaed encoding and 3D-decoupling decoding effectively captures the 3D avatar structure from a single image. To further enhance feature fusion, our hybrid prior fusion strategy combines spatial and prior-enhanced queries, leveraging the benefits of spatial localization and human body prior knowledge. This efficient and accurate integration strategy achieves state-of-the-art performance single-view human avatar reconstruction.
methods in geometry and texture reconstruction. We achieve significant reduction in Chamfer distance on CAPE-FP [16] test dataset, below 0.8cm for the first time, and demonstrate superior side-view normal performance, illustrating our method's 's efficacy in reconstructing accurate 3D clothed human avatars. Our model excels in handling complex poses and loose clothing, and attains statc-of-the- texture reconstruction with higher PSNR scores. Moreover, can extended to animation and virtual try-on applications, showcasing its wide-ranging real-world potential. Our main contributions include:
Our proposed model, trained on THuman2.0 [15], outperforms state-of-thheaar(
We introduce novel global-correlttdd 3D-decoupling transformer that effectively disentan- gles tri-plane features, thereby substantially enhancing the reconstruction of clothed avatars from 2D images. To the best of our knowledge, our approach is the pioneering application transformers in 3D feature decoupling for monocular human avatar reconstruction tasks.
We put forward an innovative hybrid prior fusion strategy for feature query, combining spatial query's 's localization capabilities with prior-enhanced query's ability incorporate knowledge of the human body prior, ultimately leading to improved geometry and texture reconstruction performance.
Our proposed model achieves offte-of-the-ar performance in both clothed human geometry and texture reconstruction, outperforming previous models and exhibiting enhanced side- view normal performance.

---

RCON
Figure GTA vs. SOTA. SOTA methods (red) are vulnerable to challenging poses and loose clothing, leading to artifacts such as non-human shapes (PIFu [1], PaMIR [3]), incomplete clothing reconstruction (ICON [2]), and erroneous stitching (ECON [4]). GTA deals with these challenges and produces high-quality results (blue).
2 Related Work
Monocular Human Reconstruction has been an active area of research for many years. This task inherently ill-posed due to the lack of 3D information, requiring additional assumptions or prior knowledge to recover the full 3D structure. Previous research has proposed effective parametric human prior models [17, 18, 19, 20, 21J. which employs statistical methods to reduce the variations in human body shape and pose compact set of parameters. By leveraging this model, subsequent research has proposed novel methods to estimate or regress the model parameters from a single RGB image [22, 23, 24, 25, 26]. However, the human prior models only capture minimally clothed body without complex details like garments, adornments, or hairstyles. To address this limitation, some researchers add offsets the top of prior body vertices to simulate outfits [27, 28, 29, 30, 31]. While these methods can effectively represent clothing close to the body surface and use blending weights of surface vertices drive the clothing, they not suitable for geometry topology far from the human body, such as robes and dresses.
In order to overcome the constraints imposed on reconstruction by clothing shape and type, researchers have explored various alternative representations for the human body, including voxels [32, 33], visual hulls [34], double depth maps [11, 12, 13, 4], and UV maps translation [35]. Among these diverse methods, implicit function- methods [36] have shown the most remarkable performance. Saito al. introduced PIFu [1], which firstly incorporates implicit functions into the problem of human neural network to extract features from 2D images and uses implicit functions to express the spatial geometry field, such signed distance functions (SDF) [37] and occupancy fields [38]. While implicit function-
body reconstruction. The method leverages a
methods [1. 39, 6] can accurately reconstruct the complex topology of clothed human body surfaces, they may generate non-anatomical shapes for out-of-distribution poses due to the lack of regularization.
To improve pose robustmess, recent research [3, 2, 4, 5, 7] has utilized the prior knowledge to guide implicit function representation. These methods have shown promising results enhancing the quality and accuracy of reconstruction geometry, particularly for challenging poses. However, these methods, like previous ones, still rely on 2D features extracted from CNN-based networks, even though some of them incorporate 3D features obtained from human body prior. In the reconstruction process, the feature obtained by 2D projection may result in incomplete reconstructions from other viewpoints and diminish overall reconstruction accuracy. Our method extracts global-correlated 3D-aware feature to efficiently represent the clothed human avatar.
Transformers in Vision. The transformer architecture, initially proposed by Vaswani et al. [40], has achieved immense success across domains like NLP, speech recognition, and multimodal applications. Inspired by this, many studies have attempted to adapt the transformer architecture to the field of computer vision. Among these explorations, the Vision Transformer (ViT) proposed by Dosovitskiy al. [41] has shown impressive performance in 2D visual tasks. Meanwhile, transformer's ability to model global and long-range correlation is also suitable for 3D vision tasks. Therefore, we leverage ViT-based 3D transformer with cross- cross-plane plane attention to efficiently extract global-correlatdd 3D features for better human reconstruction.
Generative 3D-aware Feature. Recent studies [14] have proposed the tri-plane 3D feature repre- sentation method, which efficiently extracts features from objects in three orthogonal orientations. Tri-plane representation has been demonstrated high efficacy in generating 3D objects [42], particu-
3

---

A
3D-Decopling Decoders
Refinement & Division
Croc-glane
X--C
Encoder
Principleppane
NLLP
Siery
enfioel by spggfoe
xtrplexe faauue
Input Imagy
Croseglane Decoder
Dviba
Voyllne fanker
Global-Correlated 3D-Decoupling Transformer
Hybrid Prior Fusion Strategy
Figure GTA Overview. GTA has two key modules: (1) the global-correattdd 3D-decoupling transformer and (2) the hybrid prior fusion strategy. The former module extracts a latent h from the input image and integrates with learnable embeddings z via 3D-decoupling decoders, producing disentangled tri- tri-plane features. The latter module employs spatial (red) and prior-enhanced (green) queries merge features from the two tri-planes, thus enabling geometry and texture reconstruction.
larly 3D human bodies [43, 44, 45]. Thus, the tri-plane representation method holds potential for application in human body reconstruction. Nevertheless, the challenge of establishing a reasonable relationship between the input monocular image and the three planes of features remains unre- solved problem. our method, we introduce learnable embeddings to represent features of spatial planes that are not directly visible, and utilize cross-attention mechanisms to establish relationships between the input image and other planes.
Method
We introduce an implicit function-based framework for reconstructing 3D clothed human models from a single image (See Fig. 3). Our model employs a global-correlated 3D-decoupling transformer to disentangle tri-plane features and harnesses a hybrid prior fusion strategy for reconstructing the full 3D geometry and texture of the clothed avatar. In the following sections, we will discuss the preliminaries of GTA in Sec. 3.1, the global-correlate 3D-decoupling transformer in Sec. 3.2, and the hybrid prior fusion strategy in Sec. 3.3.
3.1 Preliminary
SMPL. The Skinned Multi-Person Linear (SMPL) model [18] is widely-used parametric human body model. The SMPL model utilizes shape parameters /3 e R100 and pose parameters e R3* parameterize the deformation of the human body mesh M:
M 0) B 3 3
where K = 24 joints and N = 6890 vertices. Shape parameter /3r describes the body's overall size and proportions and pose parameter 0 defines the positions and orientations of the joints relative to their default positions. SMPL enables effective representation and manipulation of human body shape and pose in various applications.
Implicit Function. Implicit function is powerful tool for modeling complex geometries with neural networks. Our implicit function maps input point to scalar value that represents the spatial field including occupancy field and color field. The occupancy field takes a point in space as input and outputs a binary value indicating whether the point is inside outside the human surface. Our reconstructed human surface can be represented as Szz:
S {x
3 IF(x) (o, c)}
(2)
where occupancy = 0.5, color 33, and IJF represents the implicit function.

---

3.2 Global-correlated 3D-decoupling Transformer
Directly extracting 3D information from a single 2D image not feasible, this is largely due to the fact that 3D features include information from planes orthogonal with the image plane (also noted as ry-plane, principle-plane To effectively decouple cross-plane features from a monocular image input, it is crucial to have additional guiding information. This is where the idea of learnable embeddings and the use cross-attention mechanism prove valuable. Inspired by this, we devise architecture including coral-correlated encoder and DD-decuupling decoder to disentangle 3D features from single input images.
novel transformer-basee
Global-correlate Encoder. In our method, we employ vision transformer to encode the input image and capture global correlations in the image, resulting in high-dimensional global-correlated image features. Our encoder module processes the input image by dividing it into non-overlapping n patches and subsequently mapping them image features through transformer blocks. This procedure generates a latent h for image I.
3D 3D-decoupling Decoder. To decode 3D tri-plane features from the encoder output, we propose to use two types of decoders: the principle-plane decoder and the cross-plane decoder. Our principle-plane decoder generates sy-plane features, which share the same plane as the input image. This decoder effectively reverses the encoding operation, leveraging self-attention mechanism on the encoder output and converting the image features into a principal feature map Fay E ##xxCC
In order to generate plane features orthogonal with the principle plane while preserving global correlation with the principle plane, employ the cross-plane decoder decode y2 and 22 plane features from input image features. To guide the decoder in decoding features from different planes, we introduce a learnable embedding that supplies additional information for decoupling new planes. The learnable embedding z is first processed through self-attention encoding. It is then used as query in multi-head cross-attention mechanism with the output image latent h. from the encoder stack. The image features are converted into keys and values for the cross-attention mechanism,
CrossAttn(z,hh h) = Softmax (W (Weel (WW)))
(3)
vd
where Wº, WKK and WV are learnable parameters and d is the scaling coefficient. Following the original transformer architecture [40], our model employs residual connections [46] and layer normalization [47] after each sub-layer. The entire decoder consists of multiple identical layers, and we use two such decoders to produce feature maps Fys E x xC and Fas E " XC
Principle-plane Refinement. In accordance with the approach demonstrated in [6], higher-resolution feature maps play a crucial role in producing detailed geometry and sharper textures. Hence, we use both the original image and the principle-plane feature map, produce higher-resolution feature map. The original image is initially down-convoluted to match the manciple-plane size and then concatenated along the channel dimension. Subsequently, they are fed into a streamlined Hourglass network and a super-resolution module for refinement. This process generates a higher-resolution feature map pref fine E R22Hx22 X2Hx2Wxx
2y
prefine
SuperRes (oppereee(oorrdssssDDvvvoCoooooL
where C means concatenation operation. The resulting xy plane (principle-plane) exhibits a higher resolution than the yz and planes and incorporates more information from the original image, leading to higher fidelity reconstruction.
Tri-plane Division. After obtaining each plane, we evenly divide the plane features along the channel dimension into two groups, creating two tri-planes. For one group, we perform spatial query to acquire features for query points, while for the other group, we utilize a prior-enhanced query to integrate the human body prior. Please refer to the next section for detailed introduction to our novel hybrid prior fusion strategy.
5

---

Figure Qualitative 3D human reconstruction for real images showcasing diverse poses and clothing variations. For each example, we show the input image along with two views of the reconstructed geometry and front view of the reconstructed texture. Our approach is robust to challenging poses and loose clothing, and contains detailed geometry and texture. See SupMat. for more results.
3.3 Hybrid Prior Fusion Strategy
In previous works, two primary methods have been utilized for acquiring query point features, each with significant limitations as previously discussed in Sec. 1. To address this, we propose hybrid prior fusion strategy that combines the strengths of both spatial query and prior-enhanced query.
Spatial Query. Following [14], we extend the pixel-aligned query into 3D space, denoted as spatial query. This method projects query points onto the zy, yz, and planes of tri-plane group, producing localized features that capture important details for reconstruction. We combine the Fy= and Fas features by summation and concatenate the result with grefine to generate the spatial query feature F (x):
zy
psQ(2) = 125Q (xr (FFS (a) + FSQ (a:)
(5)
where (:50(x) FSS FSQ (z) are extracted from Fzy
Frefine, Fyzs Frs- and O is concatenation.
Prior-enhanced Query. For the other tri-plane, we project the human body prior [18, 19] mesh vertices onto the three planes similar to the spatial query above to obtain the feature FPQ(v), v E M, where M is the body prior mesh. For each query point æ, we find the nearest triangular face te [Vo U1 U2] E 3x33 and use barycentric interpolation to integrate features for x (See Fig. 3), denoted as FPQ(2))
PPQ uF + uFFQ(()) (v1) + WFPQ (v2)
(6)
where [u, v, w| represents the barycentric coordinates of the query point projected onto triangle tx-
Hybrid Prior Fusion Strategy. Spatial query projects the query points directly onto tri-plane features, providing detailed information but lacking prior knowledge. On the other hand, the prior-enhanced query merges body prior information but may causes an increased level of fuzziness. Therefore, we concatenate these two query features to capitalize on each method's 's strengths and compensate for their weaknesses. Furthermore, we also include the signed distance between query point and human prior mesh SDF Prior (æ: and pixel-aligned normal feature Fx(x:) as input to the implicit function for predicting occupancy and color. Consequently, the reconstructed human surface Sry can be represented as:

---

Table 1: Quantitative comparison on geometry against other methods. obtained from [2, 4].
CAPE-NFP [16]
CAPE-FP [16]
THuman2.0 [15]
Method
Training
Chamfer
P2S]
Normals,
Chamfer
P25.]
Normals
Chamfer 1 P2S)
Normals
THuman2.0 [15]
1.586
0.088
PIPa" [1]
Renderpcople [48]
2,973
2,940
0.111
2.100
2.093
0.091
PIFaHDD
Renderpeople [48]
3.767
3.591
0.123
2.302
2.335
0.090
PaMIR [3
THummm2.00 [15]
1.603
1.499
0,068
1.502
1.291
0.064
1,276
1.477
0.080
PaMIR
Renderpeople [48]
1.413
1.321
0.063
1.225
1.206
0.055
THummn2.00 [15]
1.249
0.076
ICON' 2
Renderpeople [48]
1.070
1.013
0,059
1.202
1.170
0.055
BCON
THuman2.0 [15]
0.035
0.033
0.074
ECON
[4]
THuman2.0 2 [15]
0.926
0.917
0.037
THuman2.0 [15]
0,042
0.035
0.055
SI far R.3
IF(F (z)), (P)) P (x)), Fxr(:)) = (o.
where occupancy = 0.5, color с E R3, and IF represents the implicit function.
Training Objectives. For each 3D scan, consider two sets of points training data, denoted as Go and Ge. Ge is sampled uniformly with a slight perturbation along the normals of the mesh surface, whereas Go is sampled according the same strategy in PIFu [1], where points sampled near the mesh surface and throughout the entire space.
For the points in Go. we employ the following loss function:
1
Σ BCF 0z)
(8)
BG..
where å denotes the model's predicted occupancy, while o~ signifies the ground truth occupancy. For the sampled points in Ge, we apply the following loss function:
Le
Σ
Carl
(9)
Ge
BEG.
where &. represents the predicted color at location x by the model, while ea indicates the true color of the mesh æ. The overall loss function is expressed by:
LGTA = L.+66 Le
(10)
Experiments
Strategy for Point Sampling. In the context of each training subject, our approach involves obtaining 2048 points for occupancy, denoted as Go, and 2048 points for color, symbolized as G.. The method for occupancy point sampling is aligned with the strategy illustrated in [1]. Color points are sampled uniformly, with minor Gaussian disturbance, expressed as /N0,, a), wherein our experiment o is set at 0.1 em. This disturbance occurs along the normals of the mesh surface. We obtain labels for the ground truth geometry, which specify whether point is inside outside the surface, through the application of Kaolin [49] to ascertain a point lies within the ground truth mesh. The source the ground truth color labels is the UV texture map of the 3D meshes.
Model Structure. To generate coobel-crreeltted latent features. we utilize a Vision Transformer (ViT) [41] model of depth 6, functioning as our global corbe-correl encoder, and generating an output of size 1024 256. Our 3D-dccoupling decoder incorporates both cross-plane and principal-plane decoders, each with depth of three. The cross- cross-plane decoder is initialized with learnable embeddings that experience a Gaussian perturbation to align cohesively with the encoder's output shape. The configuration of the cross cross-plane decoder corresponds with the structure described in [40], while the principal-plan
decoder emulates the global-correlated encoder. Each decoder outputs a feature map F E IR 1228x 128x61 During refinement, 2-stack hourglass and transpose convolution module are integrated to generate a higher resolution principal-plane feature map Frefine E 128 128x664 Following the feature acquisition through our hybrid prior fusion method, two identical Multilayer
22
7

---

Perceptrons (MLPs) are employed for separate predictions of occupancy and color, each with layer sizes of [512, 1024, 512, 256, 128, 1]. In the inference phase, we utilize Rembg [50] for background subtraction in in-the-wild images. The Marching Cubes algorithm [51] is employed for generating 3D meshes, while off-the-shelff models from ICON [2] are leveraged for the production of normal maps. This normal map is further processed through a 2-stack hourglass to achieve size of 128 x 128 x 6. Besides the front/back normal maps are also used as input into the encoder with the image. The model, implemented in PyTorch Lightning [52]. is trained for 10 epochs with a learning rate of le-4 and a batch size of 4, over a span of 2 days on a single NVIDIA GeForce RTX 3090 GPU.
Datasets. Our model was trained on the THu- man2.0 [15], featuring 526 high-quality human scans, with 505 designated for training and 21 for evaluation. Testing was primarily conducted on the CAPE [16] and THuman2.0, with the former divided into "CAPE-FP" and "CAPE- NFP" subsets to examine model generalization on different pose types. Further dataset and im- plementation details are available in SupMat.
Table 2: Normal Evaluation of Different Views on THuman2.0 [15]. These views are obtained by positioning a virtual camera at the front, left, back, right, above, and below the reconstructed human.
Normals of Different Views
Method
Average
Front
Left
Back
Right
Above
Below
PIFu [1] 0.053 0.143
0.046
0.067
0.066
0.081
PaMIR [3] 0.051
0.124
0.068
0.077
0.051
0.054
0.071
ICON [2]
0.091
0.076
0.044
0.044
Metrics. We employ Chamfer and point-to- surface (P2S) distances, capturing significant geometric errors, to evaluate the accuracy of the reconstructed meshes. We assess the quality of local details and the efficacy of 3D features via the 1.2 error between normal images of reconstructed and ground-truth meshes from six views. Finally, the quality of texture prediction is measured using Peak Signal-to-Noise Ratio (PSNR), comparing images rendered from both reconstructed and ground-truth surfaces across different views.
ECON [4]
0.043
0.123
0.045
0.083
0.050
0.041
0.064
Ours 0.048 0.069
0.044
0,061
0.035
0.040
0.050
4.1 Evaluation
Evaluation of Geometry. We compare our GTA model with body-agnostic methods like PIFu [1], PIFuHD [6]. and body-aware methods such as PaMIR [3]. ICON [2], and ECON [4]. Our evaluation is thorough, involving training and testing these models ourselves and incorporating testing results from [4. 2] for a comprehensive comparison. As depicted in Tab. 1, GTA excels in terms of Chamfer and P2S distances on images with outto--distribution (OOD) poses and diverse clothing. Notably, GTA is the first to reduce the Chamfer distance to less than 0.8 cm CAPE-FP. On par with ECON for normals on CAPE, GTA sets a new oftt-----heaan on THuman2.0. Fig. 8a visually underlines model's superior performance the THuman2.0 benchmarks.
Quantitative Evaluation of Side-face Reconstruction. In our novel quantitative evaluation of side-face reconstruction, we spotlight the advantages of 3D features in crafting plausible side-faces and accurate thickness. Using six virtual camera angles, we render normal images of the reconstructed human and compute the normal difference for each face on the THuman2.0 [15] dataset. As shown in Tab. 2. GTA surpasses other models in 5 out of 6 views, matching ECON only in the front view, thereby underscoring our model's prowess in capturing inherent 3D structures within an image. For visual results, refer to Fig. and SupMat.
Evaluation of Texture. In evaluating texture reconstruction, we compare GTA with color-predicting models like PIFu [1], ARCH [8], ARCH++ [9], PHORHUM [5], and S3F [7]. By rendering textured meshes from multiple angles and calculating the PSNR with respect to ground truth images, we find GTA outperforms other models on THuman2.0, As Fig. 5 demon- strates, GTA exceeds the state-of-the-an S3F [7] by 22% in PSNR. Notably, our model provides superior textures on the front side and accurately predicts invisible regions, as seen in Fig. 8b, highlighting the effectiveness of 3D features.
PTFu ARCH a ARCH444
PHORHUM S3P Ouars
20
1845
14
PSNR
10:6
10.37
11,20
5
Figure 5: Quantitative Evaluation of Texture on THuman2.0 [15].
4.2 Ablation Study
Ablation Details. To ensure fair and unbiased experimental setup, we employed an approach wherein the implicit functions of each ablation experiment were augmented with front and back
8

---

Table 3: Ablation study of several of our designs on CAPE [16] dataset.
Different networks.
2D features vs. 3D features.
(c) Different query methods.
Method
Chamfer P2S Normals
Method
Chamfer, P2S Normals
Method Chamfer; P2SL, Normals
convolution
0.991
0.968
0.055
2D+SQ
1,054
1.052
0.054
3D+SQ
0.965
0.049
w/o cross-aten
0.937
0.922
0.051
2D+PQ
1.133
1.116
0.053
3D+PQ
1.059
0.987
0.048
w/o refine
0,890
0.882
0.053
2D+hybrid
1,008
0.967
0.051
Ours
0.861
0.866
0.045
Ours
0.861
0.866
0.045
Ours
0.861
0.866
0.045
Input
convvllution
wio cross-
refine
2D+SQ
2D+0
DD+bybrid
3D+SO
Figure 6: Texture change of different ablation settings.
Input
Animation
Virtual Try-on
Figure 7: Application of our model in animation and virtual try-on.
normal features and signed distances from the human prior body. For the ablation experiments pertaining to different network architectures, we replaced the corresponding components of our network structure with three identical UNet [53] and Vision Transformer [41] decoders solely based on self-attention, respectively. Moreover, for the ablations on 2D features, we generated 2D feature map of size 256 256 with UNet. Additionally, for the ablations on the hybrid prior fusion strategy, we employed a single tri-plane for the spacial query and the prior-enhanced query, respectively.
A. Different Networks for 3D Feature Decoupling. We evaluate alternative architectures by modifying our global-correated encoder and 3D-dccuupling decoder, confirming the strength of proposed transformer. We experiment with UNet [53], also used in [7]. convolution filter representative, and employ three separate UNets to compose the tri-plane. We also test a transformer decoder with only self-attention and without the refinement module additional learnale embeddings. Results (see Tab. 3 and Fig. 6) suggest that a purely convolution- based network struggles to decouple 3D features due to limited correlation and receptive field constraints. While the refinement module shows minor geometric improvements, it is crucial for texture reconstruction. Additional learnable embeddings and cross-attention blocks notably enhance geometry results.
B. 2D Features vs. Features. We analyze the effectiveness of features by conducting an ablation study using solely 2D feature maps for reconstruction. Utilizing UNet [53] to generate high-dimensional 2D features, we compare spatial query (SQ), prior-enhanced query (PQ), and our hybrid prior fusion strategy. Results highlight the inferiority of 2D features in producing accurate reconstructions (Tab. 3), emphasizing the importance of 3D features.
C. Hybrid Prior Fusion Strategy vs. Others. We evaluate our hybrid prior fusion strategy against individual use of spatial query (SQ) and prior-enhanced query (PQ). Results show that spatial query surpasses prior-enhanced query in both geometry and texture quality due to its provision of localized features for detailed reconstruction. However, combining both methods optimizes geometry performance while preserving texture quality, demonstrating the effectiveness of our hybrid strategy.
4.3 Applications
Reconstruction of Images in-the-wild. The GTA model demonstrates significant prowess in recon- structing 3D human meshes from unconstrained, real-world images (refer to Fig. and SupMat.), addressing the complexities posed by varied poses and clothing styles. This capability of recon-

---

structing high-fidelity 3D models from in- in-the-wild images paves the way for extensive applications,
notably in virtual and augmented reality.
We present a robust approach for generating novel poses of 3D clothed hu- man meshes, catering to applications in animation and virtual try-on (See Fig. 7). We extend the S3F [7] model by employing estimated body shape and pose parameters to derive tri tri-plane features, facilitating realistic deforma- tions. For a single-imm -image clothed human reconstruction, our method excels by only needing the target pose, thus overcoming the limitations of previous deep learning meth- ods. Additionally, our model supports virtual try-on by enabling feature replacement across body parts of differ- ent parametric bodies. By selectively interchanging these features, can simulate changes in clothing the tar- get image. Our method, therefore, provides a versatile solution for both animation and virtual try-on applications, merging the strengths previous methods while alleviat- ing their weaknesses. More technical details and results are available in SupMat.
Animation and Virtual Try-On.
INPUT
PIFa
ICON
ECON
GEAnw
(a) Qualitative comparison of geometry.
INPUT
GEA
(b) Qualitative comparison of texture.
Figure 8: Comparison of geometry and
Conclusion
texture on THuman2.0 [15] benchmark.
In conclusion, we present the GTA model, a cutting-edge approach for reconstructing 3D clothed human from single images. Our global-correlttd 3D-decoupling transformer effectively extracts latent representations from input images and integrates them with learnable embeddings through 3D-decoupling decoders, generating disentangled tri-plane features. Moreover, our hybrid prior fusion strategy integrates the benefits of spatial query's localization capabilities with the prior- enhanced query's 's ability to incorporate knowledge of the human body prior, ultimately leading to improved geometry and texture reconstruction performance. We demonstrate that our proposed model outperforms state-of-the-an methods in geometric and texture reconstruction, exhibiting resilience against challenging poses and loose clothing, enabling a wide range of applications.
Acknowledgements. This work was supported in part by the Fundamental Research Funds for the Central Universities (No. 226-2023-00048) and the Natural Science Foundation of Zhejiang Province (DT23F020008).
References
[1] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao
Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[2] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13296-13306, June 2022.
[3] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric
implicit representation for image-based human reconstruction. IEEE trunsactions on puttern analysis and machine intelligence, 44(6):3170
2021.
[4] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[5] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1506-1515, 2022.
[6] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel- aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020.
10

---

[7] Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu. Structured 3d features for reconstructing relightable and animatable avatars. arXiv preprint arXiv: wtw.222/2..88000 2022.
[8] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruction of clothed humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093-3102, 2020.
[9] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation- ready clothed human reconstruction revisited. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11046-11056, 2021.
[10] Yukang Cao, Kai Han, and Swan-Yee K. Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[11] Valentin Gabeur,
Franco, Xavier Martin, Cordelia Schmid, and Gregory Ro- gez. Moulding humans: Non-paranene pan--arramticc 3d human shape estimation from single images. In Proceedings of the IEEE/CVF international conference computer vision, pages 2232-2241, 2019.
[12] David Smith, Matthew Loper, Xiaochen Hu, Paris Mavroidis, and Javier Romero. Facsimile: Fast and accurate scans from an image in less than a second. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5330-5339, 2019.
[13] Yongqin Xian, Julian Chibane, Bharat Lal Bhatnagar, Bernt Schiele, Zeynep Akata, and Gerard Pons-Moll. Any-shot gin: Generalizing implicit networks for reconstructing novel classes. In 2022 International Conference 3D Vision (3DV), pages 526-535 IEEE, 2022.
[14] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
[15] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Func- tion4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR2021), June 2021.
[16] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to Dress 3D People in Generative Clothing, In Computer Vision and Pattern Recognition (CVPR), June 2020.
[17] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPHI 2005 Papers, pages 408-416. ACM, 2005.
[18] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[19] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975-10985, 2019.
[20] Ahmed AA Osman, Timo Bolkart, and Michael J Black. Star: Sparse trained articulated human body regressor. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part VI 16, pages 598 613. Springer, 2020.
[21] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchiseseu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6184-6193, 2020.
[22] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Computer Vision ECCV 2016, Lecture Notes in Computer Science. Springer International Publishing, October 2016.
[23) Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Computer Vision and Pattern Regognition (CVPR), 2018.
11

---

[24]
Muhammed Kocabas, Chun-Hat P. Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proc. International Conference on Computer Vision (ICCV), pages 11127-11137, October 2021.
[25] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEF/CVF International Conference on Computer Vision,
pages 11446-1 11 1456, 2021.
[26] Yao Feng, Vasileios Choutas, Timo Bolkart, Dímitrios Tzionas, and Michael J Black, Collabo- rative regression of expressive bodies using moderation. In 2021 International Conference on 3D Vision (3DV), pages 792-804. IEEE, 2021.
[27] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1175-1186,
2019.
[28]
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In 2018 International Conference on 3D Vision (3DV), pages 98-109. IEEE, 2018.
[29]
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Thcobalt, and Gerard Pons-Moll. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3387-8397, 2018
[30] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2293-2303, 2019.
[31] Hao Zhu, Xinxin Zuo, Sen Wang. Xun Cao, and Ruigang Yang. Detailed human shape estimation from single image by hierarchical mesh deformation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4491-4500, 2019.
[32] Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. Bodynet: Volumetric inference of 3d human body shapes. In Proceedings of the European conference on computer vision (ECCV), pages 20-36, 2018.
[33] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7739-7749. 2019.
[34] Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima. Siclope: Silhouette-based clothed people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4480-4490, 2019.
[35] Albert Pumarola, Jordi Sanchez-Riera, Gary Choi, Alberto Sanfeliu, and Francese Moreno- Noguer. 3dpeople: Modeling the geometry of dressed humans. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2242-2251, 2019.
[36] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939- 5948, 2019.
[37] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[38] Lars Mescheder, Michael Oechsle. Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CV conference on computer vision and pattern recognition, pages 4460-4470, 2019.
[39] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto. Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. Advances in Neural Informa- tion Processing Systems, 33:9276-9999 977769992,,222 2020.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
12

---

[41] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[42] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854, 2022.
[43] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 597-614. Springer, 2022.
[44] Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. Advances in Neural Information Processing Systems, 11119900-19116 2022.
[45] Suyi Jiang, Haoran Jiang. Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. arXiv preprint arXiv:2212.05321/ 2022.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[47] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:1607.064500 2016.
[48]
Renderpeoppe. https: https://renddrpeoplle com/
[49] Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozant- sev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. https: /tpps:///ithub..
wtm.//VDAAAaaaooorks/////// kkaolinn,k 2022.
[50] Daniel Gatis. Rembg: A tool to remove images background. https://githe com/
Sanielgatis/rembg 2022.
[51] William Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 2164):163-169, 1987.
[52] William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019.
[53] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asiisted Intervention-MICCAl 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
13


Reviews:
[{"review_id": "uFduV0QO4P", "reviewer": "Reviewer_Lf3d", "summary": "The proposed system extracts global features with a 3D transformer backbone for tri-plane feature-based 3D human reconstruction. Unlike existing methods that mainly rely on 2D CNN pixel-aligned features, this paper is the first one to use Vision Transformers for decoupling the 3D tri-plane features for refined reconstruction. Qualitative and quantitative experiments on the 3D human reconstruction dataset demonstrate the state-of-the-art performance of the proposed method.", "questions": "I mainly have some concerns regarding more rigorous and fair comparisons with other methods as written in the weaknesses section. Please make them more clear in the rebuttal.", "limitations": "Limitations are properly discussed in the supplementary material.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presentation is good, I had a pleasant time reading the paper.\n\n1. The idea is well-motivated. The proposed method tries to solve the issues raised by mainly relying on pixel-aligned features and bad query methods of existing methods and introduces a tri-plane feature-based reconstruction method with a transformer-based network to mitigate these limitations.\n\n1. Experiments demonstrate that the proposed method outperforms existing state-of-the-art methods both qualitatively and quantitatively.\n", "weaknesses": "1. This method uses a Vision Transformer-based feature extractor, which I expect has a higher model capacity than the CNNs used in previous methods. Is the performance improvement brought by the global-correlated nature of the transformer or due to the larger model capacity? Plus, a larger encoder may introduce higher inference time and also make the comparison with other methods unfair. It would be nice if the authors could add some inference time statistics of different methods to make the comparison more comprehensive.\n\n1. How did you obtain the SMPL prior at the test time on Thuman 2.0? Is the SMPL obtained by an HMR method or using the ground-truth SMPL in Thuman 2.0? As the ground-truth SMPL is not available at test time, using the ground-truth SMPL would make the comparison unfair, especially for the methods in Figure 5. Please add some discussions on the SMPL condition. If the authors are using the ground-truth SMPL as the condition, it would be interesting to investigate the robustness of the proposed components (e.g., the querying and fusion strategy) when the SMPL condition is obtained with off-the-shelf HMR methods.\n\n1. In Table 1, what is the difference between ECON and ECON* as they both trained on THuman2.0? The authors mention that * results are obtained from [2,4], but I cannot find PIFuHD* results trained on THuman2.0 in [2,4].\n\n1. Normals metric (0.050) on Thuman2.0 in Table 1 seems to be calculated by averaging 6 view normals as it matches the results reported in Table 2. However, I think the original numbers in [2,4] are calculated by averaging normals of 4 views (without the above and below views). Does that indicate that there is some evaluation discrepancy between the evaluation metrics of this paper and previous ones?\n\n1. As ICON/ECON released their code and models trained on Renderpeople, their performances on Thuman2.0 and CAPE-FP should be better provided. \n\n1. In Figure 3 in the supplementary material, PiFUHD results should also be shown. And from Figure 3, I found the results of the proposed method are of lower spatial resolution and produce less detailed geometry than ECON. These low-resolution results do not match the results in other figures. The authors should give some explanations.\n\n1. Minors\n\n- The texts in Figure 3 are too small. Consider enlarging the figure for better readability.\n- Figure 2 should compare different methods with the same input. Pairwise comparison gives the impression that the authors are cherry-picking the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive remarks on our paper's presentation and the recognition of our method's effectiveness in addressing existing limitations. Thank you very much for your valuable comments!\n\nWe will address your inquiries and concerns point by point in the following responses.\n\n- **Answer to weakness 1**: Please refer to global response \"Comparison of number of parameters\" and \"Comparison of inference time\".\n\n- **Answer to weakness 2**: \n\n  1. All our experiments are carried out under equitable conditions. \n  2. For our geometry testing with open-source models, as delineated in Table 1, we utilized the ground-truth SMPL/SMPL-X, consistent with the methodologies employed by ICON [2] and ECON [4]. \n  3. In the texture performance evaluations with non-public models, such as S3F [7] in Figure 5, we abstained from using ground-truth SMPL and instead leveraged PyMAF [25] to derive the SMPL prior. \n\n  **Explanation:** \n\n  1. **Many current leading-edge works, like ECON [4], utilize ground-truth SMPL for geometry comparisons, and we align with this approach in our geometry assessments (Table 1).** This choice helps counteract errors stemming from the HMR method, ensuring a more accurate evaluation of reconstruction quality across diverse model frameworks. \n  2. For Figure 5 experiments, with most models not being open-sourced, we used data from S3F's texture comparison (Table 3 in S3F [7]). S3F confirms not using ground-truth human priors in testing (Section 4.2 in [7]), and their use of GHUM instead of SMPL precludes using the same HMR method. **We selected PyMAF [25] for SMPL estimation from input images to ensure fairness in testing.**\n  3. We appreciate your astute observation. In subsequent revisions, we will elucidate our experimental settings more clearly and emphasize the fairness of our evaluations.\n\n- **Answer to weakness 3**:  \n  1. The \"ECON\" outcomes were acquired by evaluating on our hardware using the checkpoint provided by the ECON authors as they did not release training code. Conversely, the \"ECON*\" results are directly extracted from their original paper (see Table 1 in [4]).\n  2. We deeply apologize for the unintended mistake, attributed to LaTeX formatting. We highly appreciate your meticulousness and will rectify this in the revised manuscript. **For clarity, consult [10] results (chamfer: 2.008, P2S: 1.965), where PIFuHD is assessed on THuman2.0. This oversight doesn't lessen GTA model's merits**.\n\n- **Answer to weakness 4**: \n  1. **We followed the established settings of previous works for widely used datasets.** The normal metrics in Table 1 for CAPE-NFP and CAPE-FP were computed by averaging normals from 4 views, aligning with methodologies in [2, 4].\n  2. **For our newly adopted THuman2.0 test set, we introduced a new standard.** Based on empirical observations, we found that incorporating normals from 6 views provides a more comprehensive model evaluation. \n  3. We appreciate your keen observation. Your feedback has highlighted a potential ambiguity in our manuscript regarding the normal testing methodology. To maintain uniformity and preclude any potential misinterpretations, we will standardize the normal results to 4 views in our revised manuscript.\n\n- **Answer to weakness 5**: We employed the ECON [4] model trained on THuman2.0 (not Renderpeople) due to unavailable training code (Table 1, \"ECON\"). For ICON [2], we used their released model and evaluated it on the specified datasets.\n\n  **Explanation:**  Our decision to initially omit the test data for the provided ICON model was driven by two primary considerations:\n\n  1. The released ICON model underperformed our THuman-trained version and their paper's reported results, so we omitted it.\n\n  2. We refrained from using the proprietary Renderpeople dataset due to its commercial and non-public nature, aligning with our commitment to promoting academic research. We appreciate your suggestion and will consider incorporating this result into the paper while still showcasing our model's superiority.\n\n|           | Chamfer | P2S   | Normals |\n| :-------- | :------ | :---- | :------ |\n| CAPE-NFP  | 1.207   | 1.256 | 0.071   |\n| CAPE-FP   | 1.165   | 1.216 | 0.073   |\n| THuman2.0 | 1.395   | 1.527 | 0.115   |\n\n\n\n- **Answer to weakness 6**: We add PIFuHD results in the attached PDF. Thank you for your keen observation. Because similar comments have been raised by other reviewers, we consolidate our response in the global author response \"Answer for Unsatisfactory results in Figure 3 of SupMat\".\n\n- **Answer to weakness 7**: Thank you for your valuable feedback. We sincerely appreciate your insights, and we will certainly make the necessary revisions based on your suggestions."}, {"review_id": "qYxfUb7Z0y", "reviewer": "Reviewer_rJsL", "summary": "The authors propose to reconstruct a clothed 3D human avatar from a single 2D image by introducing a global-correlated 3D-decoupling transformer to disentangle tri-plane features. A hybrid prior fusion strategy in the feature query phase is introduced to combine the spatial query’s localization capabilities with the prior-enhanced query’s ability to incorporate knowledge of the human body prior. The experiments on CAPE and THuman2.0 datasets demonstrate the effectiveness of the proposed method which outperforms state-of-the-art approaches in both geometry and texture reconstruction.", "questions": "* In Table 1, I am wondering why are the scores of ECON on Thuman2.0 dataset much worse than CAPE-NFP and CAPE-FP. \n* Please explain the difference between the visualized results in Figure 3 (supplementary material) and that in Figure 4 (main paper), as mentioned in 'Weaknesses'.\n* What is the method to predict the normal map in this work?", "limitations": "The inaccurate estimating SMPL(-x) model would affect the reconstruction. And the performance may degrade if the subject is with extremely loose clothing that considerably deviates from the human body.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The idea to introduce a global-correlated 3D-decoupling transformer to disentangle tri-plane features is novel.\n* The comparison experiments and ablation experiments are comprehensive and sound.", "weaknesses": "* In Figure 3 of the supplementary material, the result of GTA contains unnatural clothes wrinkles, while ECON performs better. It seems that the result in Figure 3 of the supplementary are worse than the results in Figure 4 of the main paper, as the former contains fewer geometric details.\n* In Table 1, the proposed method is slightly better than ECON in Chamfer and slightly worse in Normals. The explanation about this comparison doesn’t make much sense to me, and the quantitative improvement is minor.\n* The paper lacks an adequate explanation of the benefits of introducing 3D features via tri-plane or the query strategy. \n\nOne minor issue: \n* The predicted normal and SDF are only explained in Equation 7, and they should also be included in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for your recognition of the novelty in our approach of introducing a global-correlated 3D-decoupling transformer. Your acknowledgment of our comprehensive and sound experimental comparisons is highly appreciated. \n\n \n\nWe will address your inquiries and concerns point by point in the following responses. \n\n- **Answer to weakness 1 and question 2**:  Please refer to global response \"Answer for Unsatisfactory results in Figure 3 of SupMat\".\n\n- **Answer to weakness 2**: \n  1. **The predominant factor contributing to GTA's surpassing performance over ECON [4] in terms of Chamfer Distance and P2S metrics, lies in our consideration of features of the orthogonal planes during reconstruction.** Firstly, both of these metrics are employed to assess the large geometric differences. As exemplified by the results presented in Figure 3 of the SupMat, ECON tends to exhibit larger errors in the orthogonal planes compared to GTA. These orthogonal plane errors manifest in three-dimensional space as overall surface shifts, significantly influencing chamfer distance and P2S metrics.\n  2. **ECON's superior performance on Normals can be attributed to its utilization of explicit integration to predict the front and back depth maps.** The resulting depth maps harbor a multitude of high-resolution details, thus yielding better Normals in front and back normal maps. However, ECON's Normals in the orthogonal planes are generally worse than GTA's (refer to Section A.4, lines 404 - 407 in the SupMat), consequently leading to variances in overall Normals metrics depending on the dataset. In simpler datasets such as CAPE [16], characterized by minimal occlusion, differences in Normals of orthogonal planes are marginal. Thus, the Normals metric of ECON is slightly better. Conversely, in more intricate datasets like THuman2.0 [15], characterized by complexity and substantial occlusions, the disparities in the normal maps are pronounced, resulting in GTA exhibiting better overall Normals.\n  3. **As an implicit function-based monocular human reconstruction method, our quantitative improvements are significant.** While our quantitative metrics exhibit relatively minor enhancements compared to ECON on CAPE-NFP, it's important to note that ECON is an explicit optimization-based approach that demands substantial computational resources. As demonstrated in the global author response, GTA's inference time is notably superior to that of ECON and comparable to other implicit function methods. Moreover, GTA outperforms ECON in the evaluation results on the CAPE-FP [16] and THuman2.0 [15] datasets.\nThank you for your feedback. We will incorporate more relevant discussions in the revised version of this paper.\n\n\n\n- **Answer to weakness 3**:  **While briefly addressing these in Introduction, lines 27-49, page constraints led us to emphasize testing results.** We value this chance to provide a more thorough explanation, enhancing transparency and understanding of our methodology.\n\n  1. The tri-plane representation is pivotal in our method. 2D approaches struggle with complex 3D details, especially in challenges like occlusion. **Encompassing xy, yz, and xz planes, the memory-efficient tri-plane approach (see intro, lines 38-41) provides a comprehensive view of 3D structure from a 2D image.** \n\n  2. **Our model gains spatial comprehension by extracting features from distinct planes** (refer to lines 118-122). Crucial for precise 3D reconstruction, it's especially valuable in complex scenarios where depth matters. Tri-plane technique utilizes varied spatial data, improving clothed avatar reconstruction, especially in challenging poses.\n\n     Thank you for your feedback. We will incorporate more detailed explanations in our future revisions.\n\n- **Answer to question 1**: **The primary underlying factor for this disparity is the variations in data distribution between these datasets.**\n  1. In comparison to CAPE [16], the THuman2.0 [15] dataset features a higher density of mesh points, more complex attire and poses, as well as more occlusion. Consequently, during the stitching process, the proportions occupied by front and back depth maps are diminished, leading to a higher proportion of the regions being represented via SMPL-X [19] or IFnet+.  This, in turn, contributes to an overall augmented error due to the complexity of the THuman2.0 dataset.\n  2. Meanwhile, the intricacy of THuman2.0 also results in offsets between the SMPL-X predictions and the front and back depth surface predicted by ECON [4]. This discrepancy makes it more susceptible to stitching errors, as illustrated in Figure 8.B of the ECON paper. This scenario further contributes to the deterioration of ECON's quantitative metrics on the THuman2.0 dataset.\n\n- **Answer to question 3**:  In our work, we employ the pretrained body-guided normal prediction network from [2] to generate the normal map (refer to Section A, lines 23-25 in the supplementary material). \n\n  **Explain:** **Since the emergence of [6], the integration of normal maps has gained traction within the realm of 3D human reconstruction.** Normal maps can be efficiently generated through available, model-free techniques [6] or guided by SMPL-based approaches, exemplified in [2, 4]. In our study, we chose to employ the pre-trained model from [2] for normal prediction due to its exceptional accuracy and stability. **Building upon the use of normal maps, our model introduces a global-correlated 3D-decoupling transformer to disentangle tri-plane features and a hybrid prior fusion strategy, leading to state-of-the-art results.**\n\n"}, {"review_id": "lScpZYSnCh", "reviewer": "Reviewer_MMKZ", "summary": "This paper presents a single image human reconstruction method. Different from previous pixel-aligned 2D features, authors propose to extend the 2D features to 3D features using triplane. Moreover, SMPL prior is introduced to enhance the extracted features. Qualitative and quantitative experiments are thoroughly performed to show the effectiveness of the proposed method.", "questions": "N/A", "limitations": "No. Authors should discuss more on the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n- The 3D-decoupling decoder is well-designed and proved to be effective. Especially, the side views have noticeable advantage compared with methods using pixel-aligned methods.\n- Taking the 3D human prior into consideration is reasonable. Though, I might have some concerns with it (see weaknesses).\n- The experiments are thorough and compelling. SOTA performance is achieved on standard benchmarks.", "weaknesses": "- The 3D human prior rely on accurate SMPL estimation, which is not quite reliable especially for hard poses. Therefore, a discussion on how would the model perform if the input SMPL is not accurate should be included.\n- I have no complaints on the geometry part. But the texture reconstruction results are not satisfactory. Compared with NeRF-based methods (e.g. SHERF: Generalizable Human NeRF from a Single Image), the rendering quality is inferior. I understand that this is caused by the technical choices. But I think this deserves some discussion. This can also make a potential future direction.\n- Qualitative ablation studies on more complex poses are expected. From qualitative comparison with other methods, geometry qualities of arms are better. My guess is that the SMPL prior might be the main contributor here. Therefore, I think more qualitative ablation studies is in need here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on the clarity of our paper and the effectiveness of our 3D-decoupling decoder. Your recognition of our thorough and compelling experimental results is highly valued. We are grateful for your comments.\n\n\n\nWe will address your inquiries and concerns point by point in the following responses. \n\n- **Answer to weakness 1**: We appreciate your insightful comment, and we fully agree that accurate SMPL estimation is crucial for achieving precise 3D human reconstruction. Due to the constraint of page limitations, **we have included discussions on limitations and broader impact in the supplementary material (please refer to Sections C and D).** \n  1. **We adopt the SMPL parameter optimization process from [2] to mitigate inaccuracies arising from HMR estimates** (as indicated in Section A, lines 8-10 of the Supplementary Material). Our goal is to attain the utmost accuracy in reconstruction outcomes. Yet, as illustrated in Figure 1 of the supplementary material, substantial inaccuracies in HMR-derived SMPL estimations can result in erroneous reconstruction outcomes.\n  2. **It's crucial to emphasize that this limitation is not unique to our model; rather, it's a common challenge encountered by numerous cutting-edge approaches in the field.** The issue of handling inaccuracies stemming from SMPL estimation while leveraging SMPL priors for reconstruction and bolstering model robustness is a vital research avenue. This area will be a central focus of our forthcoming research efforts.\n\n\n\n- **Answer to weakness 2**: \n\n  1. **Our work follows the previous research of PIFu, ICON, and similar methods, with a focus on enhancing the accuracy of reconstructed geometry.** Thus, we employ mesh as our 3D reconstruction. \n\n  2. **Despite this, we have maintained vigilant attention to the advancements in the field of NeRF.** We concur that leveraging NeRF may yield promising advancements in texture reconstruction, positioning it as a compelling avenue for future research. However, NeRF currently exhibits several drawbacks in comparison to the 3D mesh representation we employ, such as storage, transmission, and real-time rendering challenges.\n\n  3. **We posit that the primary constraint hindering the application of NeRF in the digital human domain is its inability to accurately reconstruct geometry.** This limitation restricts the utility of NeRF-based approaches in applications such as animation and interactive environments. Enhanced establishment of the relationship between NeRF and spatial geometry structures could potentially lead to significant advancements for NeRF-based methods.\n\n     Thank you for your suggestions. We will incorporate relevant discussions in the revised version of this paper.\n\n\n\n- **Answer to weakness 3**:  **The utilization of human body priors in studies related to digital humans is a prevailing consensus in current research.** \n  1. Among the methods compared, including PaMIR [3], ICON [2], ECON [4], and our proposed approach, all utilize SMPL(-X) as a human body prior. In contrast to early methods like PIFu [1] and PIFuHD [6] that lack such a human body prior, incorporating the human body prior provides a robust constraint for reconstruction results, preventing the generation of meshes that deviate from the anatomical structure when dealing with complex poses. \n  2. Currently, methods that do not utilize human prior exhibit a significant performance lag compared to those leveraging human priors. Referencing Figure 3 in the supplementary material or Figure 1 in our global response PDF can illustrate this point. \n\n\n\n- **Answer about limitations**:  Due to space constraints, **we have included discussions on Limitations and Broader Impact in the supplementary material, specifically in Sections C and D.** \n  1. **Our model has two main limitations.** Firstly, if the HMR model estimates an inaccurate SMPL pose, it may lead to mesh errors. Secondly, extremely loose clothing in input images can challenge complete garment reconstruction. Refer to Figure 1 in the supplementary material for visual examples.\n  2. **It is essential to highlight that these limitations are not exclusive to our model; they are challenges faced by many state-of-the-art (SOTA) approaches in the field.** Our focus will be on enhancing robustness, ensuring accurate mesh reconstruction despite pose errors, and improving the reconstruction of loose clothing."}, {"review_id": "5jgr6DDZm2", "reviewer": "Reviewer_7DFs", "summary": "This paper proposes a new method for the task of single-view 3D human reconstruction. The main idea is to extract 3D tri-plane features from the input image using a transformer-based architecture, instead of using CNN to extract 2D pixel-aligned features as done in previous works. The experiments demonstrate that the proposed method enables more accurate and detailed reconstruction compared to SoTA.", "questions": "1. Most previous methods predict frontal and back normal maps to help 3D human reconstruction. This paper does not have this additional step, which simplifies the framework and is good. However, it would be helpful if the authors could comment on if the proposed method is compatible with normal map input. And could a normal map further improve the result?\n\n2. Since the method can reconstruct loose clothing and the authors have demonstrated animation, I am wondering what the animation of loose clothing (e.g. long dresses) look like.", "limitations": "There is no discussion about the limitation of the proposed method.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Method:** The method is technically sound. Using tri-plane representation and using a transformer to generate side planes is an interesting design choice and seems to be effective.\n\n**Experiment:** The results are convincing. The authors compare the method with SOTA methods following standard protocol and report a noticeable improvement over SOTA.  The ablation study is very well designed and clearly demonstrates the effectiveness of each individual component.\n\n**Presentation:** The paper is well-structured and written. I find it easy to follow and understand.", "weaknesses": "I am overall positive about this paper because the idea of using transformers to extract tri-plane features from 2D images is well-motivated and technically sound, and the idea has been very well executed and validated. \n\nI don’t find any critical flaws in the paper. The only weakness is that the back side texture is still blurry but this is a common problem for all existing works.\n\nHowever, since the proposed method is an adaption of well-established methods (tri-plane, transformer) to a specific task, the originality and the potential impact are limited IMO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of the technical soundness of our method and the effectiveness of our design choices. Your acknowledgment of our convincing results and well-structured presentation is highly appreciated.\n\n\nWe will address your inquiries and concerns point by point in the following responses. \n\n- **Answer to question 1**: \n  1. We sincerely appreciate your recognition of our work. In our research, **we do utilize normal maps within our framework, as mentioned in Section 3.3 (lines 213-215) and Supplementary Material Section A (lines 23-25).** \n  2. **Since the emergence of [6], the integration of normal maps has gained traction within the realm of 3D human reconstruction.** Normal maps can be efficiently generated through available, model-free techniques [6] or guided by SMPL-based approaches, exemplified in [2, 4]. In our study, we chose to employ the pre-trained model from [2] for normal prediction due to its exceptional accuracy and stability. **Building upon the use of normal maps, our model introduces a global-correlated 3D-decoupling transformer to disentangle tri-plane features and a hybrid prior fusion strategy, leading to state-of-the-art results.**\n\n\n\n- **Answer to question 2**: \n\n  1. We sincerely appreciate your interest in our work and their query regarding the animation of loose clothing. In response, **we provide animation results of loose clothing in the first row of Figure 5 in the supplementary material. Additionally, we have included results in Figure 2 of the PDF attachment in our \"global response\" in the rebuttal.**\n\n  **Explanation:** \n\n  1. **The strength of our animation method lies in its simplicity and efficiency, as it eliminates the need for additional models.** By directly animating the mesh within the same framework, our approach achieves natural and smooth animated results that are competitive with or even surpass models requiring separate training for animation.\n  2. As elaborated in Supplementary Material Section A (lines 45-56), animation relies on the SMPL model, utilizing barycentric interpolation for feature acquisition. For loose clothing, the garment vertices follow the nearest SMPL surfaces' movement, occasionally causing separation in long dresses due to leg movement. **Although this issue is prevalent across this field, resolving it remains a challenge.** We look forward to advancing this field by exploring innovative techniques and dedicated strategies for animating loose clothing.\n\n\n\n- **Answer about limitations**:  Due to space constraints, **we have included discussions on Limitations and Broader Impact in the supplementary material, specifically in Sections C and D.** \n  1. **Our model has two main limitations.** Firstly, if the HMR model estimates an inaccurate SMPL pose, it may lead to mesh errors. Secondly, extremely loose clothing in input images can challenge complete garment reconstruction. Refer to Figure 1 in the supplementary material for visual examples.\n  2. **It is essential to highlight that these limitations are not exclusive to our model; they are challenges faced by many state-of-the-art (SOTA) approaches in the field.** Our focus will be on enhancing robustness, ensuring accurate mesh reconstruction despite pose errors, and improving the reconstruction of loose clothing.\n\n\n\n- **Answer for comment \"Since the proposed method is an adaption of well-established methods (tri-plane, transformer) to a specific task, the originality and the potential impact are limited IMO\"**:\n  1. **Our adoption of these techniques was driven by the imperative to accurately reconstruct 3D human meshes from 2D images.** Notably, our approach distinguishes itself from prior state-of-the-art (SOTA) methods, representing a novel direction within this field. Empirical validation substantiates our model's attainment of SOTA performance.\n  2. **Mere aggregation of these techniques proved insufficient in achieving exceptional reconstruction outcomes.** Our focus on the 3D human body reconstruction task led to tailored enhancements of these techniques. Innovations such as the global-correlated 3D decoupling transformer and hybrid prior fusion strategy were meticulously integrated to meet our objectives, culminating in the observed SOTA performance.\n  3. **The evaluations of Reviewers Lf3d and rJsL unanimously underscore the novelty inherent in our approach, addressing the limitations that have persisted in the realm of 2D feature-based methods**. Their recognition underscores our method's significance in overcoming the constraints associated with traditional reliance on 2D feature methodologies."}, {"review_id": "ckZ4xBsz3I", "reviewer": "Reviewer_6dkA", "summary": "This paper aims at reconstructing 3D clothed human models from single images. Current methods rely heavily on 2D image features extracted from the input image, while ignoring information lying in the planes orthogonal to the input image plane. To address this limitation, this paper proposes a new method that utilizes attention mechanisms to construct tri-plane features in order to capture more 3D information. In addition, the authors also introduce a new feature query strategy that combines spatial query and manifold query. Results show that the proposed method outperforms existing baselines, showing high robustness to challenging poses and loose clothing.", "questions": "1. Since the principle plane is refined after 3D-decoupling decoding, why not refine other planes (i.e., yz and xz planes) as well?\n\n2. The authors keep emphasizing \"global correlation\" in the paper, but I cannot get the exact meaning of this term after reading the paper. Could the authors provide more detailed explanation or some examples to help reader understand what \"global correlation\" means?", "limitations": "The authors addressed the limitations and potential social impact in the supplemental document.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Current methods typically use 2D convolutional networks to extract pixel-aligned features on the xy-plane, and rely on the networks themselves to infer 3D information along z-axis. In contrast, this paper explicitly considers the other two planes, i.e., the yz-plane and the xz-plane, and extracts features on these two planes as well. I think the insight behind the proposed method is valuable and can inspire future research.", "weaknesses": "1. My biggest concern is the inconsistent results. In Figure 1, the authors demonstrate a nice reconstruction result where geometric details are fairly recovered. However, in Figure 3 of the Supp.Mat., the reconstruction results of the proposed method are really coarse and many geometric details are missing. In addition, this figure shows that the proposed method performs significantly worse than ECON in terms of the front-view normal quality, although they should perform similarly according to Table 2 in the main paper. \n\n2. Although the proposed method outperforms SOTA quantitatively, it comes at a cost of a much heavier network architecture. According to the method description, the proposed method uses multiple networks, including a vision transformer, several attention-based networks, an Hourglass network, an MLP as well as learnable embeddings. Compared to the networks used in PIFu/PaMIR/ICON/ECON, these networks are more complex and much heavier, containing a larger number of learnable parameters. Therefore, the performance gains may be due to \"bigger\" networks. Unfortunately, I do not see any discussion of method complexity in the paper. \n \n\n3. Also, large networks may be prone to overfitting. However, it seems that the authors can train such large networks using a relatively small amount of data, i.e., only about 500 scans. It is not clear how the authors achieved this. \n\n4. I think Figure 4 shoud show texture from more view points. The current version only shows the prediction results in the front view, which can be directly queried from the input image and cannot prove the texture estimation performance of the proposed method. \n\n5. The learnable embeddings do not make sense to me. If I understand correctly, the learnable embeddings remain fixed after network training and are the same for different input images. However, for different images, the side view varies dramatically. Therefore, I don't think there exist unified embeddings that fit all input scenarios. I would like to see the visualization of learnable embeddings if possible. \n\n6. Although I agree that solely relying on xy-plane is insufficient to reconstruct 3D models, I am suspicious whether the proposed method really solves this issue or not. According to Figure 2 in the Supp.Mat., the yz-plane and xz-plane feature maps do not contain much valid and distinguishable information compared to xy-plane. In other words, I guess that the learned network still relies heavily on xy-plane to reconstruct 3D models, and the robustness to challenging poses is mainly resulted from the introduction of prior-enhanced queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your recognition of the valuable insight behind our method and its potential to inspire future research.  \nWe will address your inquiries and concerns point by point in the following responses. \n\n\n- **Answer to weakness 1**: Thank you for your keen observation. Because similar comments have been raised by other reviewers, we consolidate our response in the global author response \"Answer for Unsatisfactory results in Figure 3 of SupMat\".\n\n- **Answer to weakness 2**: Our results suggest that the performance improvement primarily arises from the intrinsic properties of the global-correlated 3D-decoupling transformer and the intricately crafted framework, rather than solely the parameter scale：\n  1. **Our transformer-based feature extractor possesses fewer parameters than the UNet-based counterpart but performs better.** \n  2. **Our model's time efficiency is comparable to other state-of-the-art (SOTA) methods that utilize implicit functions, while significantly outperforming ECON.** \n  3. **Please refer to global response for details about these comparisons.** In our forthcoming revisions, we will provide a clearer exposition of this issue in the supplementary material.\n\n- **Answer to weakness 3**: \n  1. **The training dataset we utilized, derived from THuman2.0, is substantial, which helps mitigate the risk of overfitting.** While it is true that we only selected about 500 scans from THuman2.0 for training, each scan was rendered in 36 views under various environmental lighting conditions. Consequently, the total number of data points used during training reached 18,180 (505 × 36), which is a considerably large dataset.\n  2. **Using a smaller dataset is also feasible in this field, but it can lead to longer training times.** For example, the S3F model [7], trained for five days on eight V100 GPUs with a batch size of 8, is larger than our model. Notably, our model, trained in two days on an NVIDIA GeForce RTX 3090 GPU with a batch size of 4, achieved competitive results. **S3F achieved SOTA performance using only 245 Renderpeople scans**, rendered five times each, as mentioned in [7] section 4.1 (Table 2, \"Only synthetic data\").\n  3. **The current state of this field lacks comprehensive research on the impact of training data size and the potential for overfitting.** We acknowledge and align with the reviewer's attention to this aspect. In the future, we intend to conduct further exploratory experiments to delve into these aspects more extensively.\n\n\n\n- **Answer to weakness 4**: \n  1. **We provide texture results from back side view and various view points in Figure 8 of the main text, Figure 4 in the supplementary material, and the SupMat's accompanying video.** \n  2. **We have meticulously computed PSNR for the outcomes rendered from multiple viewpoints.**\n  3. We greatly appreciate your suggestions and we are actively considering adjustments to the layout to incorporate texture results from different angles within the paper.\n\n- **Answer to weakness 5**: \n  1. **Despite the learnable embedding remaining fixed as query, the input image still serves as both the value and key inputs to the decoder.** This mechanism continues to decode features associated with different planes that maintain correlations with the image, as opposed to features unrelated to the image (refer to section 3.2 lines 166-172).\n  2. **The renowned object detection model DETR [a] also employs fixed learnable embeddings as queries in its decoder to detect objects in various input images.** This further illustrates the applicability of fixed learnable embeddings across different input images.\n  3. **In our ablation experiments, the reconstruction quality noticeably deteriorates when the learnable embedding is not employed** (as referenced in Section 4.2, lines 275-277, and \"w/o cross-atten\" in Table 3(a)). This compellingly underscores the significant role played by the incorporation of learnable embeddings.\n\n\n\n- **Answer to weakness 6**: \n  1. The features from the principal plane are crucial for reconstruction, yet previous methods have not considered incorporating features from the other two planes.\n  2. We address the challenges posed by heavy reliance on pixel-aligned features by introducing a novel tri-plane feature-based reconstruction approach. This concept, endorsed by reviewers Lf3d, rJsL, and 7DFs, is well-motivated, innovative, and efficacious.\n  3. The ablation study demonstrates the superiority of utilizing triplane features over relying solely on xy-plane features (2D+hybrid), as evidenced in Table 3(b).\n\n- **Answer to question 1**:\n  1. Refining yz and xz planes with the input image would merge decoupled features with the xy plane features, contradicting our goal of orthogonal plane feature separation.\n  2. If we were to independently design networks to refine the XZ and YZ planes without incorporating the input image, we believe that this approach would yield results similar to the impact achieved by introducing additional layers within the decoder.  However, this strategy would result in an excessively bulky model, thereby escalating training costs.\n\n\n\n- **Answer to question 2**: **The concept of \"global correlation\" serves as a means to highlight the disparities between transformer and convolution architectures**. Convolutions possess localized receptive fields, whereas transformers utilize attention mechanisms to establish connections across the entirety of input data, fostering global feature understanding (refer to lines 41-45 and lines 275-278). This attribute aids in disentangling the intricate 3D features, thus contributing to the enhanced reconstruction process.\n\n  Thank you for your question. We will provide clearer explanations in the article through further revisions.\n\n\n**Reference**：\n\n[a]. Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]//European conference on computer vision. Cham: Springer International Publishing, 2020: 213-229.\n"}]

Author rebuttal:
- **Comparison of inference time:**  
  1. **Table 1(b) in the attached PDF displays comparable time efficiency of our implicit function-based model with PIFu, Pamir, and ICON.** In contrast, ECON, relying on explicit methods, demands more time due to d-BiNI and Poisson inefficiencies. CAPE-NFP dataset with 256^3 resolution is used for testing, ground truth SMPL/SMPL-X provided.
  2. The omission of inference time statistics in our initial submission was due to the following reasons:
     - **Previous research in 3D human mesh reconstruction has primarily emphasized result quality rather than inference time.** Notably, many state-of-the-art methods (PIFu [1], PIFuHD [6], Pamir [3], ICON [2], ECON [4]) have not conducted time cost comparisons or prioritized efficiency improvements.
     - **The superior time performance (compared to ECON) is likely due to the inherent efficiency of implicit function based methods.** It is not the primary contribution of our work, and due to page limitations, we did not include the inference time statistics.



- **Comparison of number of parameters:** 
  1. **In Table 1(a) of the attached PDF, it's evident that our model outperforms the UNet-based approach with fewer parameters, as indicated by better reconstruction results** (refer to Table 3(a) and Supplementary material lines 38-41). This highlights the advantages of our transformer-based design in extracting 3D features. 
  2. **Our decision to initially forgo a direct comparison of model parameter sizes was influenced by the prevailing trend in the field, where seminal works (e.g., ICON [2], Pamir [3], S3F [7]) seldom engage in such comparisons**.
  3. However, we concur with your perspective on the importance of model parameter size comparisons. Such an approach would clarify the factors contributing to performance improvements, thereby fostering more robust development within the domain of 3D human reconstruction.



- **Answer for Unsatisfactory results in Figure 3 of SupMat:** 

  1. **The unsatisfactory results of GTA in Figure 3 of the SupMat are a direct consequence of our utilization of a lower Marching Cubes resolution for mesh generation from the implicit function network.** Specifically, in Figure 3 of SupMat, we employed a resolution of 128 for marching cubes. In contrast, for inference results such as Figure 4 of the main paper, we adopted a resolution of 512 for marching cubes. This parameter discrepancy is accountable for the incongruity between the results presented in these two figures. While all implicit function-based methods are constrained by the marching cubes resolution and can only reconstruct results with low spatial resolutions, the results of ECON[4], which don't rely on marching cubes, appear much better.
  2. **In Figure 1 of our rebuttal response PDF, we have supplemented revised normal map results and showcased the impact of marching cubes resolution on GTA and ECON[4].** In subfigure a, we provide rendered normal maps under the setting of quantitative testing (resolution of 256), along with PIFuHD[6]'s test results. In subfigure b, we illustrate various resolutions' effects on GTA's reconstruction results, elucidating the causes of detail loss. In subfigure c, we demonstrate the influence of different resolutions on ECON's results, explaining why ECON's normal maps retain more detail.

  **Explanation:** 

  1. **The reason we employed a resolution of 128 for marching cubes in Figure 3 of the SupMat is that the primary focus of this image is not on geometric details but rather on the main contribution of our paper, which is the enhancement in orthogonal plane geometry reconstruction achieved by GTA.** Within this image, we present a compilation of over 400 normal maps, each possessing a relatively compact scale. Therefore, we do not place strong emphasis on the geometric details of the normal maps. Moreover, taking into consideration that employing a resolution of 512 for marching cubes yields meshes that are approximately 20 times larger in terms of storage occupancy compared to the case of a resolution of 128, we opted for the adoption of a lower marching cubes resolution.
  2. **The more detailed normal maps observed in ECON's results in Figure 3 can be attributed to its utilization of explicit reconstruction methodologies, which doesn't necessarily involve the application of marching cubes during the reconstruction process.** Among the comparative methods, namely PIFu [1], PIFuHD [6], PaMIR [3], and ICON [2], as well as our proposed GTA , the final mesh reconstructions are all generated using marching cubes. However, as detailed in Section 3.3 of ECON[4]'s paper, the method employed by ECON to reconstruct the mesh involves stitching the predicted front and back depth maps with SMPL-X[19] and optionally IFnet+ (referred to as $ECON_{EX}$ and $ECON_{IF}$, respectively). IFnet+ is an implicit function network that employs marching cubes in its predictions and is influenced by the marching cubes resolution. However,  IFnet+ primarily affects regions where both front and back depth maps are unavailable or occluded parts, constituting a small portion of the reconstruction mesh during stitching process. The majority of the mesh is determined by the depth maps with a resolution of 512x512. As a result, the reconstruction results of ECON in the figure still preserve a multitude of high-resolution details.
  3. We sincerely appreciate the reviewer's keen observation and concerns. **We acknowledge that Figure 3 in the supplementary material may potentially lead to misunderstandings. To address this, in the upcoming revision, we will ensure uniformity by setting the marching cube resolution to 256 for all results**. While ECON [4]'s performance is relatively unaffected by marching cube resolution, our reconstructed results at this resolution are competitive with ECON's, particularly showcasing improved performance in side-view reconstructions.

Meta-review: All reviewers have consistent voting for acceptance. The AC takes a carefully look at all review comments and responses and think this is a work with very high completeness, the algorithm design makes sense and the experimental results verify the effectiveness very well. Thus, it can be accepted.
Decision: Accept (poster)