Garment4D: Garment Reconstruction from Point Cloud Sequences
Authors: Fangzhou Hong
Liang Pan
Zhongang Cai
Ziwei Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Garment Reconstruction
Dynamic Garment Reconstruction
Point Cloud
Point Cloud Sequence
GCN
Transformer
Dressed Human Modeling

Abstract: Learning to reconstruct 3D garments is important for dressing 3D human bodies of different shapes in different poses. Previous works typically rely on 2D images as input, which however suffer from the scale and pose ambiguities. To circumvent the problems caused by 2D images, we propose a principled framework, Garment4D, that uses 3D point cloud sequences of dressed humans for garment reconstruction. Garment4D has three dedicated steps: sequential garments registration, canonical garment estimation, and posed garment reconstruction. The main challenges are two-fold: 1) effective 3D feature learning for fine details, and 2) capture of garment dynamics caused by the interaction between garments and the human body, especially for loose garments like skirts. To unravel these problems, we introduce a novel Proposal-Guided Hierarchical Feature Network and Iterative Graph Convolution Network, which integrate both high-level semantic features and low-level geometric features for fine details reconstruction. Furthermore, we propose a Temporal Transformer for smooth garment motions capture. Unlike non-parametric methods, the reconstructed garment meshes by our method are separable from the human body and have strong interpretability, which is desirable for downstream tasks. As the first attempt at this task, high-quality reconstruction results are qualitatively and quantitatively illustrated through extensive experiments. Codes are available at https://github.com/hongfz16/Garment4D.

Garment4D: Garment Reconstruction from Point Cloud Sequences
Fangzhou Hong', Liang Pan', Zhongang Cai 'S-Lab, Nanyang Technological University Sense' Time Research {f (fangzhou001, liang.pan pan, ziwei. liu) @ntu. edu.sg caizhongang@sensetiic com
Ziwei Liu Shannhaii Al Laboratory
Abstract
Learning to reconstruct 3D garments is important for dressing 3D human bodies different shapes in different poses. Previous works typically rely on 2D images input, which however suffer from the scale and pose ambiguities. To circumvent the problems caused by 2D images, we propose principled framework. Garment4D, that uses 3D point cloud sequences of dressed humans for garment reconstruction. Garment4D has three dedicated steps: sequential garments registration, canonical garment estimation, and posed garment reconstruction. The main challenges are two-fold: 1) effective 3D feature learning for fine details, and 2) capture of garment dynamics caused by the interaction between garments and the human body, especially for loose garments like skirts. To unravel these problems, we introduce a novel Proposal-Guise Graph Convolution Network, which integrate both high-level semantic features and lowelevel geometric features for fine details reconstruction. Furthermore, we propose a Temporal Transformer for smooth garment motions capture. Unlike non-parametric methods, the reconstructed garment meshes by our method are separable from the human body and have strong interpretability, which desirable for downstream tasks. As the first attempt at this task, high-quality reconstruction results are qualitatively and quantitatively illustrated through extensive experiments. Codes are available at https: / /tte:////iiii github. com feffzf/ent//GGmDDDD
Hierarchical Feature Network and Iterative
Introduction
Garment reconstruction is crucial technique in many applications, e.g. virtual try-on [1], VR/ AR [2] and visual effects [3]. Extensive efforts [4-15] have been put into reconstructing the human body and garments as a whole with the help implicit representation or volumetric representation. However, it is desirable to have controllable garment model in many applications. In this work, we focus on the parametric modeling of 3D garment, which has advantages the following two ways. separate garments from human body. Secondly, the topology of the reconstructed meshes can controlled, and therefore allows downstream tasks that require high interpretability.
Firstly,
Different from previous parametric methods for garment reconstruction [7, 8, 16, 17] that take 2D RGB images as input, we choose to approach the problem from the perspective of 3D, specifically point cloud sequences (shown in Fig. 1), for the following three reasons. Firstly, 3D inputs eliminate scale and pose ambiguities that are difficult avoid when using 2D images. Secondly, exploiting temporal information is important for garment dynamics capturing, at which there are few attempts. Thirdly, recent development in 3D sensors (e.g. LiDAR) has reduced the cost difficulties in obtaining point clouds, which makes it easier to leverage 3D point clouds for research problems and commercial applications. However, garment reconstruction from point cloud sequences has not been properly explored by previous works. We contribute early attempt this meaningful task in this work. Our advantages against other garment reconstruction methods are listed in Tab.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(A)
(b)
(C)
Method Type
Template-Free
SMPL+D
Garment4D
(b)
Input Modality
image
image
ped seq.
No Ambiguity
X
Separable
X
Interpretability
X
/
(c)
Loose Garments
Figure 1 & Table 1: Comparison With Popular Garment Reconstruction Methods. Two major types of garment reconstruction methods, Le. (a) template-free frree and (b) SMPL+DDbasee methods, compared (c) Garment4D) in terms of input modality, scale/ pose ambiguity, ability separate garments from body, mesh interpretability and ahility to deal with loose garments. Some modified version of SMPL+D [16] could separate garments from body. 'ped seq." stands for point cloud sequence.
The proposed garment reconstruction framework, Garment4D, consists of three major parts: a) sequential garments registration, b) canonical garment estimation and c) posed garment reconstruction. In the registration part, have several sequences of garment meshes which have different mesh topology across the sequences but share same topology inside one sequence. For each type of garments (e.g. T-shirt, trousers, skirt), we use an optimization-base from each sequence to a template mesh. Then, inside each sequence, a barycentric interpolation method is used to re-mesh other frames unify their topology with the template mesh. Thereafter, as the first step of garment reconstruction, following the practice of previous parametric methods [16, 17]. we estimate the canonical garment mesh for each sequence by a semantic- aware garment PCA coefficients encoder, which takes the dressed human point cloud sequences as input.
method register one frame
The posed garment reconstruction part is where the challenges come in for reasons. Firstly, due to unordered and unstructured nature of point clouds, it is non-trivial to learn low-level geometric features directly. Secondly, challenging to capture garment dynamics caused by interactions between the human body and garments. Especially, it is difficult to model the non-rigid deformation of loose garments (e.g. skirts), which depends on both current human pose and previous human motions. To address the challenges, we first apply the Interpolated Linear Blend Skinning (LBS) to the estimated canonical garments as proposals. Unlike [7, 8, 16] that depends on SMPL+D model to skin the garment, the proposed Interpolated LBS has the ability to perform skinning on loose garments without artifacts. Moreover, our method does not require learning, which is different from [17]. After obtaining the proposals, we present the Proposal-Guice the Iterative Graph Convolution Network (GCN) for efficient geometric feature learning. Meanwhile, a Temporal Transformer is utilized for temporal fusion to capture smooth garment dynamics.
Hierarchical Feature Network along with
Our contributions can be summarised as:
I) We propose practical pipeline for the reconstruction various garments (including challenging loose garments) that are driven by large human movements, which does not rely any particular human body parametric model or extra annotations other than human and garment meshes.
2) Many novel learning modules, such as Proposal- -uiiedd Hierarchical Feature Network, Iterative GCN, and Temporal Transformer are proposed, which effectively learn spatio-temporal features from 3D point cloud sequences for garment reconstruction.
3) We establish comprehensive dataset consisting 2, 300 3D point cloud sequences by adapt- ing CLOTH3D [18], which serves as the first benchmark dataset for evaluating posed garment reconstruction based on point cloud sequences.
In addition, this work presents an early attempt at exploiting temporal information for capturing garment motions. To the best of our knowledge, we the first explore garment reconstruction from point cloud sequences.
2 Our Approach
We introduce our approach, Garment4D, in this section. Firstly, our garment model is formulated in Subsection 2. Afterwards, we claborate the three major steps of Garment4D: 2.2 sequential

---

Bediiaation-Based Registration
Borycestric Interpolation
Garment Registration
weights
weights
weights
Re-meli
Source Meshes
Target Mesh
PCA
Semantic
PCA Decompostion
Seg.
S35
pred
Imput Point Cloud
Sequences
Displacement
Shinning
LRS
Weight
GCN
b) Garment Reconstruction
Interpolation
Figure 2: Garment4D Pipeline Overview. The upper part illustrates the garment registration. Source meshes registered target meshes and re-meshed using barycentric interpolation. The lower part shows the garment reconstruction pipeline. The stage predict PCA coefficients estimate canonical garment. The second stage predicts displacements guided by the interpolated IBS proposals.
garments registration, 2.3 canonical garment estimation, and 2.4 posed garment reconstruction. Training loss functions are introduced in Subsection 2.5, An overview of the proposed Garment4D is shown Fig, 2.
2.1 3D Garment Model
We design the garment model by the following three parts:
Canonical Garment Model. Formally, the canonical garment T E R-Nx3 can be formulated as T(a) G Co where G represents the mean shape of current type of garment, is the PCA components that form garment shape sub-spaces, represents the PCA coefficients of the corresponding garment.
One critical problem in skinning step is to get a set of skinning weights of garment vertices that can provide reasonable proposals of posed garments. Previous methods [7, 8, 16] approximate the garment skinning weights by using the weights of the closest human mesh vertex. However, these methods can result in mesh artifacts, and hence limit the quality of reconstructed garments for the following two facts. Firstly, the resolutions of our garment meshes is higher than that of human meshes. Secondly, loose garments like skirts do not always share same topology with human body. To avoid artifacts, we propose to interpolate the skinning weights from K nearest neighbors of human mesh vertices Tū. Laplacian smoothing is also performed further ensure the smoothness of the skinning weights.
Interpolated LBS.
Displacement Prediction. lated results are mostly not satisfactory. To this end, we further refine the results of interpolated LBS by predicting displacements D. In general, the whole garment model can be formulated
For those challenging loose garments like skirts and dresses, the interpo-
M(,, 0, K, Wh, D) (T)()), J, 0, Wy(a, The Wh. K)) + D,
(a, Th, Wh, K) w (T(a), The Wh, (T(a), Th, K),
(2)
where 0 E R|/ represents the axis-angle of each joint, WA. E Mx|J] W E NRM are the skinning weights of human and garment mesh vertices. J is the joint locations of human body, W Mx|J|K is the garment skinning weights of K nearest body vertices, I,o MR the inverse distance weights used for interpolation.
Garment Registration
In order to perform PCA decomposition the same type of garments, we unify the mesh topology by registering the raw meshes to template mesh visualization of the registration process illustrated in Fig. 2 (a).

---

Canonical- Posed Garment Estimation
PN
Sample
MLP
Sumple
MLF
Max
MLP
PCA
Interpoaated 1BS
b) Posed Garment Reconstruction
MLP
Max
MLP
Max
MLP
Max
GCN
Displacement
Harrrchical Garmou Finture Pooling
MLP
Max
MaN
MLP
Max
Lincar
Temporal Trangforeer
Saleed Duo Product
Hivarchicut Boddy Surface Encoding
Attariin
Update
times
Figure 3: Details of Garment Reconstruction Network. a) The canonical garment estimation takes point clouds as input. Semantic segmentation is performed first to get the garment point clouds, upon which down- sampling and feature extraction further applied to predict the PCA coefficients. b) Using the interpolated LBS, proposals posed garments are obtained, guided by which, a Hierarchical Feature Network along with Iterative GCN and Temporal Transformer are used to iteratively predict per-vertex displacements.
Considering that souree meshes have large shape variance and diverse sleeve/ hem length, we use boundary-awarg optimization method to align the source meshes to the target mesh. The registration loss function Lreg can be formulated as
Creg \oCoD(R T) AcC)L( Xn E
where LRD(R', ( is the Chamfer Distance (CD) between source and target mesh vertices, CEL (R') normalizes Edge Length of the source mesh, C RR)) maintains the Normal Consistency between neighbouring faces, (R' ; and target meshes (e.g. cuffs and neckline), and X corresponding weights for each loss term. The LED term makes the optimization process aware of the boundaries of the meshes, which essential for aligning garments with large shape variance.
minimizes the CD between corresponding Boundaries of source
X x3 is then used to calculate barycentric interpolation weights against the vertices of template mesh T', which is further used re-mesh garment meshes in other frames. Specifically, for each vertex x of T, the nearest face found. We project to and then calculate the barycentric coordinate (w1, W2 W3y) of the projected point x' on f f.Using the barycentric coordinates, is straight- to re-mesh all the other frames of garment meshes to the topology of the template mesh T.
For each sequence, the aligned garment mesh
is
2.3 Canonical Garment Estimation
To estimate canonical garments T(x),, we predict the PCA coefficients a by using the segmented 3D points of garments from input point cloud sequences P E RT xpx3, where T is the sequence length. As shown Fig. (a), we first perform point-wise semantic segmentation to select the 3D points of garments for each frame, which results in sequence of garment point clouds Pa R7 xp, x3 Subsequently, Py used to predict the PCA coefficients a for each sequence, which is further used to reconstruct target canonical garments by T(a) G + Ca.
2.4 Posed Garment Reconstruction
Proposal-Guides Hierarchical Feature Network. As introduced in Subsection 2.1, the skinned garments Mp = (T J, 0, W,) of the proposed garment model can provide reasonable initial proposals for posed garments. However, due to complex human motions and garment flexibility, especially for intense movements and loose skirts, the initial proposals of garment poses can be far different from real poses. To address these problems, we propose a novel Proposal-Guided Hierarchical Feature Network, consisting of two major parts: 1) Hierarchical Garment Feature Pooling, and 2) Hierarchical Body Surface Encoder. Displacement D predicted refine the initial garment poses. Detailed architectures of these two modules demonstrated in Fig. (b).

---

The proposed Hierarchical Garment Feature Pooling can effectively capture local geometric details from garments. Hierarchical Body Surface Encoder facilitates predicting correct gurment poses by considering interactions between garment and human body. By estimating proper contacts between garments and skin, our body surface encoder highly alleviates the interpenetration problems.
Hierarchical Garment Feature Pooling aims at pooling accurate geometric features and rich semantic information for each vertex of mesh proposals. Specifically, for each vertex of Mp. several query balls with different radii are constructed to sample points from the segmented point clouds of garments. To make use of the rich semantic information extracted when predicting the PCA coefficients a, different levels of down-sampled garment point clouds and the corresponding features from the previous stage are used for the query process with different radii and sample numbers. Then for each vertex, the pooled garment features are further encoded to aggregate low-level geometric features and high-level semantic information.
Hierarchical Body Surface Encoder makes each vertex of the garment mesh aware of its neigh- bouring human body surface, which is important for correct interaction encoding between garment and human body, especially under the circumstances of large body movements. For each vertex of Mn. neighbouring human mesh vertices sampled using different radií of query balls. Both coordinates and normal directions of sampled human mesh vertices are further encoded.
Iterative GCN Displacement. Concatenating the per-vertex features obtained from the above feature pooling and surface encoder, along with the coordinates of the vertex, we have aggregated necessary information for the following Iterative Graph Convolution Network (GCN) to predict the displacement D. Loose garments skinned by the interpolated LBS are normally not in accordance with the real situation. Therefore, we choose to iteratively predict displacements and gradually approach the ground truth. As shown in Fig. 3 (b), for the j-th iteration, using the previous accumulated displacement predictions E'=l D, and the proposal Mp. we can calculate current mesh prediction by Myj Mp + D D;. Mpj is then used for the Proposal-Guider Hierarchical Feature Pooling of this iteration. The last GCN layer of each iteration outputs current displacement Dj. Inspired by [19], the GCN layer could be formulated as (1+ = ReLU (*) w + B()), where Á = A++ I is the bidirectional mesh graph with self-connections, = diag E, Asg). w and B!!) are the learn-able weights and bias.
Temporal Transformer. To support better temporal information extraction, a Transformer is inte- grated into the Iterative GCN. For the j-th iteration, we take the features extracted by the (j - 1)-th iteration, perform temporal fusion by Transformer and concatenate them to the input of the j-th iteration. Specifically, we denote the output feature of the second last layer of the (j - 1)-th iteration as Fj-1 E R x
Query, key and value are obtained by applying MLP to Fj Fj-1, which could be denoted as Q, K, V MLP(Fy-1), where Q, K, V E RR KM Then the value is updated as V softmax(QK) VTV. V is then concatenated to the pooled features of the j-th iteration.
2.5 Loss Functions
Loss functions corresponding to two parts of network are introduced in this section. For the canonical garment estimation, the loss function consists of five terms, which could be formulated as
M As
L. =>iCcellogits, labelg:) + >2
agel 22 -ggl2-
Σ i=1
1
Tí(a') -Ti(aat)
M
(4)
(Th, + >, (T(a'). ( L)
The first term Lcg is the cross entropy term for semantic segmentation. The second and third term use L2 loss to supervise our predicted PCA coefficients a' and mesh vertices T(a'), respectively. The fourth term is the interpenetration loss term, which could be formulated
M Σ
(T,, T(a'))
1 M
ReLU (Nna (Ti(a') - Vini)).
(5)
1=1
5

---

(a)
(d)
Figure 4: Visualization of the Garment Reconstruction Process. (a) Input point cloud sequence; (b) Canon- ical garment estimation; (c) Results interpolated LBS; (d) Final reconstruction results; (e) Ground truth.
where Vn and Nhe are the coordinate and normal vector of the nearest human mesh vertex of the i-th garment mesh vertex (a')). The fifth term is the Laplacian Regularization term,
Ms LI.((((a)) T(oegr, L) = Σ M (LLT('')2 - -ICT(a))))))
(6)
where L is the cotangent weighted Laplacian matrix of the garment mesh. For the j-th iteration of the posed garment reconstruction, the loss function includes four terms, which are defined as
M
ppj
A6
M'
Matl Xyhrn(Mn, M³) + >sLLR(M' Mgt: L) + AaLro(M") (7)
M
The first term is the L2 loss term between predicted and ground truth garment mesh vertices M and Mgt. The second term penalizes interpenetration between posed garment prediction M and posed human body M.. The third term performs cotangent weighted Laplacian regularization. The last term adds Temporal Constraints to the deformations of the predicted garments, which could be formulated
CTO(M') Σ - M 1)(2
Adding up weighted loss of each iteration, we get the loss the posed garment reconstruction Cp E, WLyp.
Experiments
3.1 Datasets and Evaluation Protocols
Datasets. We establish point cloud sequence- garment reconstruction dataset by adapting CLOTH3D [18] for our experiments. CLOTH3D is a large scale synthetic dataset with rich garment shapes and styles and abundant human pose sequences. We sample point sets from 3D human models to produce the point cloud sequence inputs. We select three types of garments, i.e. Skirts, T-Shirt, Trousers, for the experiments. For skirts, 453 sequences with 109629 frames of point clouds are used for training and testing. For T-shirts, 394 sequences with 95540 frames are used. For trousers, 1465 sequences and 357086 frames used. We split the sequences to training and testing sets at the ratio of 8 2.
'CLOTH3D downloaded from
evc. uab ss//attaset//Bddesriipppp

---

Figure Qualitative Reconstruction Results. Reconstructed sequences skirts. T-shirt trousers are shown in different colors. High reconstruction quality achieved even with large body movements.
Class
PN+
Ours
Class
MGN
Int. LBS
Ours
Class
MGN
Ours
Skirt
33.71
30,23
Skirt
71.66
79.52
49.39
Skirt
12.59
3.273
T-Shirt
41.05
28.39
T-Shirt
40.54
44.55
37.83
T-Shirt
7.173
4.749
Trousers
20.80
18.16
Trousers
29.12
31.05
26.48
Trousers
4.135
2.520
Table 2: L2 Error of Canoni-
Table 3: 1.2 Error of Posed Garment Recon- struction. MGN" is adapted from [16]. Int.
Table 4: Accelcration Error of Posed Garment Reconstruction. [m/s²]
cal Garment Estimation. PN+ refers to [21]. [mm]
LBS the results interpolated LBS. [mm]
In addition to the synthetic dataset, we also perform experiments on a real human scan dataset CAPE [20]. CAPE is a large-scale real clothed human scan dataset containing 15 subjects and 150k 3D scans. Since CAPE only releases the scanned clothed human sequences with no separable garment meshes, it is not practical train network CAPE from scratch. Therefore, directly inference on CAPE using the network pretrained on CLOTH3D.
Evaluation Metrics. Three metrics are utilized for evaluation on CLOTH3D. We first evaluate the per- vertex L2 error of canonical garments, M, = Σ..; 1
ME (a') - T (ur Secondly, we evaluate the per-vertex L2 error of posed garments, which is defined as M2 = =,, ME [M, f The last one is the acceleration error, which is used to evaluate the smoothness of the predicted sequences and could be formulated as M3 = EE, ME AA Agt ful2. where A stands for acceleration vector of the i-th vertex of f-th frame.
Because the group truth garment meshes of CAPE different methods by using the 'one-way' Chamfer Distance from the vertices of the reconstructed meshes the input clouds nearest distance from point to points P.
provided, we evaluate the performance of
is formulated as [77] Σrer dp(r), where (p(r) stands for the
Since it is the first work tackling garment reconstruction from point cloud sequences, there is readily available prior work for direct comparison. Therefore, for canonical garment estimation, we adapt the PointNet++ [21] structure to predict the PCA coefficients directly from input point clouds. For posed garments reconstruction, we adapt Multi-Garment Net [16] (MGN) on point cloud inputs for comparison.
Comparison Methods.
3.2 Qualitative Results
Step-by-Step Visualization. As shown Fig. 4, we perform a step-by-step visualization of the whole reconstruction process. The estimated canonical skirt recovered rough shape and length of
CAPE is downloaded from sttps://cape... tue mue.mp..de/
7

---

the skirt from the point cloud sequences. As expected, the interpolation LBS produces reasonable proposals with smooth surfaces and no artifacts. There are two things that make the LBS results look unnatural. One is the interpenetration between garments and huamn body. The other is the lifted skirt floating the air, not touching the leg, which should be the cause of the lift. These two flaws are inevitable because the skirt is not homotopic with human body. These defects would be eliminated finally. The final step of displacement prediction meets the expectations of three aspects. Firstly, the hierarchical garment feature pooling module captures the geometric features of cloth, which helps the precise shape recovery of posed garments. Secondly, the human body surface encoding module makes the network aware of the body surface positions, which prevents the interpenetration of garments and human body and helps in capturing the deformations caused by body movements, e.g. knees lifting up the skirt. Thirdly, the temporal fusion module encourages smooth deformation between consecutive frames.
Reconstruction for Different Garment Types. More visualization of the reconstruction results are illustrated in Fig. 5, which shows Garment4D's ability of handling both loose and tight garments driven by large body movements. For garments that are homotopic to human body, i.e. T-shirts and trousers, Garment4D can not only capture correct shape of them, but also the dynamics caused by body movement. Taking the bottom left trousers reconstruction as example, when the right leg is stretched out, the back side of trousers stick to the back of calf. The front part of trousers naturally form the dynamics being pulled backwards. Moreover, it can be observed that when the leg or arm bends, more of the ankle or forearm are exposed in order to maintain the length of trousers legs or sleeves. For the reconstruction of skirts which are not homotopic to human body, both long and short skirts can be faithfully recovered. The short skirt tighter and we can observe the natural deformation when the two legs are stretched to sides. For the looser long skirt, clear contacts and deformations can be observed when knees and calves lift. And when the legs are retracted to the neutral position, skirts would naturally fall back but would not stick leg surfaces.
3.3 Quantitative Results CLOTH3D
Canonical Garment Estimation. Tab. 2 reports the per-vertex L2 error of canonical garment estimation of the adapted PointNet++ [21] and our method. By explicitly making the network aware of the semantics method out-peroorms the vanilla PointNet++ in all three garment types.
Posed Garment Reconstruction. The per-vertex L2 errors of posed garment reconstruction are shown in Tab. 3. Because T-shirt and trousers are homotopic human body, the interpolated LBS step could give decent reconstruction results. In contrast, for skirts, The results of interpolated LBS of skirts is far from the real situation and therefore have high L2 errors, which supports the above qualitative analysis. The displacement prediction effectively captures the dynamics of skirts and improves 30.13 mm top of interpolated LBS. Moreover, Garment4D outperforms the adapted Malti-Garment Net [16] in all three garment types. Especially, Garment4D surpasses the adapted MGN in skirt by 22.27 mm, which further shows the advantage of Garmen44 over SMPL+DDbasedd methods at reconstructing loose garments like skirts.
Temporal Smoothness. As reported in Tab. 4, our method also outperforms the adapted MGN in terms of reconstruction smoothness. Even though the same temporal constraints term is included in the loss functions of both methods, the adapted MGN fails to properly model the motion of garments and aggregate temporal information. Particularly for loose garments like skirts, Garment4D outperforms the adapted MGN by 9.317 m/s².
$7.81
56.49
8.848
51.6
49.39
3.273
Class
MGN*
Ours
T-Shirt 0.430
0.366
Trousers
0.922
0.455
Troader
Weo LBS
wi Thandfaaeeer
Fall Modal
(a) Per-Vertex L2 Em. [mme]
(b) Accel. Err. [mvi]]
Table 5: 'One-way' Chamfer Dis- tance on CAPE [mm]. MGC' is adapted from [16].
Figure 6: Results Ablation Study, (a) shows the comparison of Per-vertes L2 error of posed garment reconstruction. (b) reports the acceleration error.
8

---

3.4 Quantitative Results on CAPE
As reported in Tab. 5, our method outperforms the adapted MGN on both classes in terms of the 'one-way' Chamfer Distance. It shows the effectiveness and advantages of Garment4D on real world data. Moreover, we directly inference on CAPE with the model trained on CLOTH3D without fine uningg which further demonstrated the generalizability of Garment4I).
Proud Garmnntt Kecontrrccio
66:22
$1,45
(9)
100.4
103.7
$1.6
190
=
K 256
Figure 7: Comparison of LBS results when nearest neighbour K is set to and 256. Severe artifacts appear when which is circled out.
Figure 8: Trade-off between number of neighbors K, the reconstruction performance and computational cost.
3.5 Ablation Study
Hierarchical Body Surface Encoder. As shown in Fig. 6, without the hierarchical body surface encoder (H.B.S Encoder), the L2 error increases by 8.42 mm, which is line with the expectations. Without knowing body positions, it is hard for networks to infer garment-body interactions.
Temporal Transformer. As illustrated in Fig. 6, we remove the temporal transformer, the L2 error increases by 2.21 mm and the acceleration error increases by 5.575 m/s². The results indicate that the temporal transformer not only helps the single frame reconstruction by aggregating information from neighbouring frames, but also increases the smoothness of the reconstruction sequences.
Interpolated LBS. As circled out in the left side of Fig. 7, if the skinning weights of garment vertices are copied from the nearest body vertex, which is equivalent of SMPL+D (e.g. MGN[16), the severe cliff-like artifacts appear between the legs. The quality of the LBS proposals would effect the final reconstruction results, which is shown in Fig. 6. K 1, the per-vertex L2 error of the final reconstruction results would increase by 7.1 mm..
.Caneeical Larrunnte Recomiractie
Puad Garment Becasdractoon
CCnooiical Receremetion
Gorments Becondmmctinn
$3.0
1.2 Ernu [ans]
48.52
51.99
535
6.13
18.35
20,8
16.06
56.18
29.22
28.9
29.07
31.34
MAX
303
91.22
31.22
3335
35.5
37.96
8%
185
205
34%
40%
20%
60%
78%
100%
90%
80%
70%
69%
90%
40%
18%
Degree Tncompleteness
Seymentation Error
Figure
Reconstruction Performance on Point Clouds with different levels of incompleteness.
Figure Reconstruction Performance given Differ- ent Segmentation Errors.
3.6 Further Analysis
Number of Neighbors K of Interpolated LBS. As shown in Fig. 8, the performance peaks
K = 256, which explains our parameter choice. The nearest neighbors and weights interpolation are implemented for GPU parallel computation. Therefore, the computation overhead is minor with the increase of K.
Performance Incomplete Point Clouds. The real-scanned point clouds can be incomplete, which makes an important setting to test on. To control the degree of incompleteness (i.e. the percentage of missing points), randomly crop out holes from the input point clouds and performs evaluation the skirt test set of CLOTH3D. As shown in Fig. 9, decent reconstruction results can be obtained at up 50% of incompleteness, which shows the robustness of our methods on imperfect point clouds.
Robustness Reconstruction Results over Segmentation Errors. The segmentation quality of input point clouds can affect the reconstruction results. To investigate such effects, construct

---

different degrees of segmentation errors and evaluate on the skirt test set of CLOTH3D. As shown in Fig. 10, decrease of reconstruction quality can be observed with the decrease of segmentation accuracy. However, even with the accuracy of 50%, our method can still produce decent reconstruction results. Therefore, Garment4D is robust to segmentation errors.
Related Work
Template-Free Clothed Human Modeling. Most previous template-free methods reconstruct clothed human use volumetric representation or implicit function. [4, 5, 9-13] reconstruct clothed human from images using volumetric representation. Due to large memory footprint of such represen- tation, these methods usually have trouble in recovering fine details of surfaces. [6, 22-25] propose efficient clothed human reconstruction pipeline with the help of implicit function. The above two types of methods both reconstruct human and cloth as whole, which means it is not possible to directly take higher level of control of the garments, e.g. re-pose, re-targeting.
Template-Fres Garment Modeling, In order to focus more on the garments, several recent work make some attempts to reconstruct human and garments separately using implicit function. [15] predicts a layered implicit function, which makes it possible to separate garments from human body, [14] builds a garment specific implicit function over the canonical- posed human body, which could be controlled by latent codes that describe garment cut and style. Not relying on any specific template could be advantage (e.g. more degrees of freedom). But in some applications e.g. re-posing, re-targeting, texture mapping, it is disadvantage because of poor interpretability. To achieve higher level of control, [15, 14] have register the garment meshes, which are reconstructed from implicit functions, to the SMPL+D model.
Parametric Clothed Human Modeling. Most previous garment reconstruction work that use parametric methods tend to take human body and garment as a whole. A common practice, SMPL+D [7, 8, 26-30], is predict displacements from SMPL [31] vertices. However, SMPL+D has difficulty modeling loose garments or garments that do not share same topology as human body, which limits the expression ability of SMPL+D.
Parametric Garment Modeling. In order to reconstruct garments separately using parametric models, PCA decomposition is used by [16, 17] to model the single layered garment meshes, which independent of any parametric human body model. To further increase the expressive ability of an individual parametric garment model, [32] maps garments to two types of UV-maps according to whether the garment is homotopy to human body. As for the deformation of the garment models, most existing methods rely the skinning process of human body. [16] samples the skinning weights of garment vertices from the closest human body vertices [17] predicts skinning weights of garment vertices using neural network. Only relying on the skinning process would lead to overly smoothed results, which is addressed by [33] through additional displacement prediction before skinning.
Conclusion
In this work, we present the first attempt at garment reconstruction from point cloud sequences. garment model, with the novel interpolated LBS, is introduced to adapt to loose garments. A garment mesh registration method is proposed prepare sequential meshes for the garment reconstruction network. Taking point clouds sequences as input, the corresponding canonical garment is estimated by predicting PCA coefficients. Then Proposal- Guided Hierarchical Feature Network is introduced feature aggregation. Efficient Iterative GCN
to perform semantic, geometric and human- along with Temporal Transformer utilized to further encode the features and predict displacements in an iterative way. Qualitative and quantitative results from extensive experiments demonstrate the effectiveness of the proposed framework.
Acknowledgments This study is supported by NTU NAP, MOE ACRF Tier (2021-T1-001-088) and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
10

---

References
[1] Igor Santesteban, Miguel A Otaduy, and Dan Casas. Learning- Leasning-based animation of clothing for virtual try-on. In Computer Graphics Forum, volume 38, pages 355-366, Wiley Online Library, 2019.
[2] Pascal Volino, Christiane Luible, and Nadia Thalmann Virtual clothing. Wiley Encyclopedia of Computer Science and Engineering, 2007.
[3] Simon Spielmann, Volker Heizle, and Rahul Nair. On-set depth capturing for vfx productions using time of flight. In ACM SIGGRAPH 2013 Talks, pages 1-1. 2013.
[4] Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang. Ersin Yumer, Ivan Laptev, and Cordelia Schmid. Bodynet: Volumetric inference of 3d human hody shapes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 20 30-36, 2018. 10
[5] Aaron S Jackson, Chris Manafas, and Georgios Tzimiropoulos. 3d human body reconstruction from single image via volumetric regression. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018. 10
[6] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. arXiv preprint arXiv:1905.055 2019. 10
[7] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1175-1186, 2019. 3, 10
[8] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Moll. Detailed human avatars from monocular video. In 2018 International Conference 3D Vision (3DV), pages 98-109. IEFF, 2018. 2, 10
[9] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from single image. In Proceedings of the IEEE/CVF International Conference Computer Vision,
pages 7739-7749, 2019. 10
[10] Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Mor- ishima. Siclope: Silhouett--aseed clothed people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4480-4490, 2019.
[11] Sicong Tang, Feitong Tan, Kelvin Cheng, Zhanyang Li, Siyu 7hu, and Ping Tan. neural network for detailed human depth estimation from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7750-7759, 2019.
[12] Valentin Gabeur, Jean-Sébbstiee Sébastien Franco, Xavier Martin, Cordelia Schmid, and Gregory Rogez. Moulding humans: Non- -orraaamettric 3d human shape estimation from single images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2232 2241, 2019.
[13] David Smith, Matthew Loper, Xiaochen Hu, Paris Mavroidis, and Javier Romero. Facsimile: Fast and accurate scans from image in less than second. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5330-5330300 5339, 2019. 10
[14] Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and Francesc Moreno-Noguer. Smplicit: Topology-aware generative model for clothed people. arXiv preprint arXiv:2103.0687), 2021. 10
[15] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combin- ing implicit function learning and parametric models for 3d human reconstruction. arXiv preprint arXiv:2007.1133: 2020. 1, 10
[16] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garment net: Learning to dress 3d people from images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 24200,430, 2019. 1, 2, 3, 7, 8, 9, 10
[17] Boyi Jiang, Juyong Zhang, Yang Hong. Jinhao Luo. Ligang Liu, and Hujun Bao. Benet: Learning body and cloth shape from single image. In European Conference on Computer Vision, pages 18-35. Springer, 2020. 1, 2, 10
[18] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: Clothed 3d humans. In European Conference Computer Vision, pages 344 359. Springer, 2020. 2,6

---

[19] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 10X.091009002007, 2016. 5
[20] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. Learning dress 3d people in generative clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1469-6478 2020.
[21] Charles Qi. Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets metric space, arXiv preprint arXiv: 02413, 2017. 7.8
[22] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function high-resolution human digitization. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 84-93, 2020. 10
[23] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2886-2899 2021.
[24] Xu Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating rigid neural implicit shapes. arXiv preprint (XXv:2104.03953 2021.
[25] Mare Habermann, Weipeng Xu, Michael Zollhofer, Gerard Pons-Moll, and Christian Theobalt. Deep- cap: Monocular human performance capture using weak supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5052-5063, 2020. 10
[26] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8387-838 877-88972220 2018. 10
[27] Thiemo Alldicck, Gerard Pons Pons-Moll, Christian Thcobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from single image. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 2293 2303, 2019.
[28] Akihiko Sayo, Hayato Onizuka. Diego Thomas, Yuta Nakashima, Hiroshi Kawasaki, and Katsushi Ikeuchi. Human shape reconstruction with loose clothes from partially observed data by pose specific deformation. In Pacific-RRie Rim Symposium on Image and Video Technology, pages 225-239. Springer, 2019.
[29] Hao Zhu, Xinxin Zuo, Sen Wang. Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hicrarchical mesh deformation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4491-4500 2019.
[30] Donglai Xiang, Fabian Prada, Chenglei Wu, and Jessica Hodgins. Monoclothcap: Towards temporally coherent elothing capture from monocular rgb vidco. In 2020 International Conference on 3D Vision (3DV), pages 322-332. IEEE, 2020. 10
[31] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 3446):248:1-24816 October 2015. 10
[32] Zhaoqi Su, Tao Yu, Yangang Wang. Yipeng Li, and Yehin Liu. Deepeloth: Neural garment representation for shape and style editing. arXiv preprint orXiv:201 1.1999 2020. 10
[33] Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. Tailomet: Predicting clothing in as function of human pose, shape and garment style. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7365-7375, 2020. 10
12


Reviews:
[{"review_id": "fN2vlcmiInZ", "reviewer": "Reviewer_1Hss", "summary": "This manuscript designs a parametric garment model and utilize techniques such as point cloud network and GCN to reconstruct garment mesh. Quantitative and qualitative experiments on synthetic data show the effectiveness of this method to a certain extent. Compared to MGN [16], the method can better process loose garments and produce more natural reconstructions. The technical description is also clear and reproducible.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several concerns:\n1. Its parametric garment representation, including the interpolated LBS skinning weights computation, is very similar with the garment model of BCNet[17], except it adds displacements on posed garments. I think this will decrease the manuscript's novelty.\n2. The authors claim that they are the first to explore garment reconstruction from point cloud sequences. It may be true while I do not think it is very importmant. First, some works have explore the clothed body reconstruction from point clouds [A,B], while they use implicit representations. Second, point cloud input can bring other problems. For example, real point cloud data often has holes and may be partial, these can greatly affect the prediction of the network, while these situations are not analysized in the manuscript.\n3. If I understand correctly, the method not only needs the garment point cloud, but also the dressed body point cloud, which is used in the Hierarchical Body Surface Encoding. I think this requirement severely restricts the application of this method to real scene. Because in a realistic scanning environment, it is difficulty to get the body scanning hidden under the garments. Therefore, I doubt the practicality of this method.\n4. All the experiments are based on the synthetic data, while some real point cloud should be tested to present the practicality. Besides, the demo presented reconstruction results are not satisfactory  for me.\n\n[A] Saito S, Yang J, Ma Q, et al. SCANimate: Weakly supervised learning of skinned clothed avatar networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2886-2897.\n[B] Chen X, Zheng Y, Black M J, et al. SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes[J]. arXiv preprint arXiv:2104.03953, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As discussed above, the authors should analyze more about the practicality and scalability of the method.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "XJ8_twHtMDu", "reviewer": "Reviewer_jo5B", "summary": "The paper presents an approach for 4D garment reconstruction. To this end, it first presents a statistical shape model for garments which is learned using sequences of garment meshes. Since garment meshes across different sequences may have different topologies, the paper aligns them by first registering them to a template mesh and then re-meshing using barycentric interpolation. Given the statistical garment model, the paper then presents an approach for 4D garment reconstruction from a point-cloud sequence as input. The point cloud is first segmented into different body parts and then fed into an MLP to predict the PCA coefficients of the garment model. This provides the garment mesh in the canonical pose. Subsequently, for each body pose, the canonical mesh is posed using the LBS skinning weights interpolated from the body model. These posed meshes are further refined by iteratively predicting additional displacements using hierarchical feature extractors and a transformers-based temporal model. Experiments are performed on the CLOTH3D dataset and compared with adapted baselines. The proposed method is shown to outperform the baselines. Additional ablative experiments demonstrate the importance of different design choices. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n- The paper addresses the problem of garment reconstruction from point-cloud sequences which has not been addressed previously and is of great interest, though I am not sure if the paper aligns well with NeurIPs (may be better suited to CVPR, etc.) \n- The proposed approach is intuitive and makes sense, although it's incremental in nature. \n- Appropriate baselines are developed to validate the contributions of the paper. \n- Sufficient experiments are performed to validate the design choices. \n- I like the idea of interpolated LBS to avoid cliff-like skinning artifacts. \n- I like the idea of using interpenetration loss to explicitly build a relationship between body surface and clothing. \n\n**Cons:**\n- No qualitative samples for real data are provided. \n- Also, no examples of very fast body movements is provided in the supplemental video. I mainly wanted to evaluate the impact of laplacian smoothening. For slow motion, the resulting reconstruction looks good, but how does it look for fast motion? Is there any lag in general?\n- The paper is a bit hard to follow at times. I sometimes missed the intuitions behind different design choices which makes the paper less interesting to read. Also, the paper requires a thorough revision. I have provided some typos below. \n- In Section 3.4, when replacing a specific component of the proposed approach, it's not clear what is used instead. Please make it clear by also providing the details of alternative components. \n- The related work is not enough and requires a thorough revision. \n\n**Questions:**\n- For the BC term in eq (6), how are the corresponding boundaries obtained?\n- How are the human body meshes obtained? Are ground-truth meshes used in this case? \n\n**Missing References:**\nThe following papers should be referenced in the paper as they also tackle estimation of clothing: \n1. MonoClothCap: Towards temporally coherent clothing capture from monocular RGB video, 3DV'20. \n2. DeepCap: Monocular Human Performance Capture Using Weak Supervision, CVPR'20.\n\nAlso, 27 references are too few to do justice with the large amount of literature on human clothes modeling. Please thoroughly revise the related work. \n\n**Typos:**\nL67: learns -> learn \nL76: are formulated -> is formulated. \nL77: afterwards and successively are redundant. \nL87: re presents -> represents \nL117: normalized -> normalizes \nL174: Biases\nL228: homotopy -> homotopic \nL229: of all three aspects. \nL334: and help in capturing the deformations caused by body movements. \n\n**Other suggestions:**\nThe work canonical-posed can be replaced with `Canonical` for ease of reading. This, however, is a personal preference and not critical. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations and societal impacts are provided. I suggest authors to kindly address this by providing results on faster motions, different body shapes, and providing additional comments about societal impacts. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "A105Fj8ylq", "reviewer": "Reviewer_cJ86", "summary": "This paper proposed a framework, Garment4D, to reconstruct 3D garment models from 3D point cloud sequences of dressed humans instead of 2D RGB images. There are three key components in the proposed framework, including garment registration, garment shape estimation in the canonical pose, and garment reconstruction in the target pose. Experiments demonstrate the effectiveness of the proposed framework with more accurate and natural reconstruction results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission claims itself as the first attempt to reconstruct the 3D garment model from 3D point cloud sequences, which eliminates many limitations of those solutions using 2D images. The three key steps in the proposed framework also include new technical designs to address the challenges in modeling the detailed deformation and garment dynamics. Overall, this paper is well-written and technically sound.\n\nThere are a few concerns regarding this paper:\n\n- Guo et al. have also investigated using point cloud as input for the reconstruction of dynamic geometry of clothes. Though it seems that these two papers are concurrent with each other, it is recommended to add discussions or comparisons with Guo et al. in the final version of the submission. \n[Guo et al., Inverse Simulation: Reconstructing Dynamic Geometry of Clothed Humans via Optimal Control, In CVPR, 2021]\n\n- In Sec. 2.3, point-wise semantic segmentation is needed to select the garments from the input point cloud. In practice, it may be harder for algorithms to perform clean segmentation from real-world point clouds. How about the robustness of the estimation results over the segmentation errors?\n\n- In Sec. 2.4 and Fig. 3, are all hierarchical garment and body features concatenated together as the input of GCN? It is not very clear in the presentation and figure since the vertex numbers of the garment and body are different.\n\n- In Sec. 3.4 and Fig. 6, how about the trade-off between the K value and the reconstruction performance? How about the additional computational cost raising by the increase of K?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Reconstructing 3D garment models from 3D point cloud sequences is interesting and appealing. However, this submission does not show or disccuss the performance on real-world point cloud sequences. In comparsion of the solutions using 2D images as inputs, the generazation of the proposed framework remains unclear.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
This paper proposes a framework  to reconstruct 3D garment models from 3D point cloud sequences of dressed humans. The paper raised concerns regarding robustness of the model to segmentation errors, limited real world results, similarities to recent works, requirement for body point clouds (under the garment), and accuracy of the proposed method. The rebuttal submitted by the authors addressed these concerns by showing real world results and the feasibility of obtaining body pointclouds using the SMPL model. Though the relevance of the paper to a wide NeurIPS audience maybe still be under question,  all reviewers are positive about the paper, and it is suggested for publication.  

Decision: Accept (Poster)