Garment4D: Garment Reconstruction from Point Cloud Sequences

1 Introduction . Garment reconstruction is a crucial technique in many applications , e.g . virtual try-on [ 1 ] , VR/ AR [ 2 ] and visual effects [ 3 ] . Extensive efforts [ 4–15 ] have been put into reconstructing the human body and garments as a whole with the help of implicit representation or volumetric representation . However , it is desirable to have a controllable garment model in many applications . In this work , we focus on the parametric modeling of 3D garment , which has advantages in the following two ways . Firstly , we can separate garments from human body . Secondly , the topology of the reconstructed meshes can be controlled , and therefore allows downstream tasks that require high interpretability . Different from previous parametric methods for garment reconstruction [ 7 , 8 , 16 , 17 ] that take 2D RGB images as input , we choose to approach the problem from the perspective of 3D , specifically point cloud sequences ( shown in Fig . 1 ) , for the following three reasons . Firstly , 3D inputs eliminate scale and pose ambiguities that are difficult to avoid when using 2D images . Secondly , exploiting temporal information is important for garment dynamics capturing , at which there are few attempts . Thirdly , recent development in 3D sensors ( e.g . LiDAR ) has reduced the cost and difficulties in obtaining point clouds , which makes it easier to leverage 3D point clouds for research problems and commercial applications . However , garment reconstruction from point cloud sequences has not been properly explored by previous works . We contribute an early attempt at this meaningful task in this work . Our advantages against other garment reconstruction methods are listed in Tab . 1 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The proposed garment reconstruction framework , Garment4D , consists of three major parts : a ) sequential garments registration , b ) canonical garment estimation and c ) posed garment reconstruction . In the registration part , we have several sequences of garment meshes which have different mesh topology across the sequences but share same topology inside one sequence . For each type of garments ( e.g . T-shirt , trousers , skirt ) , we use an optimization-based method to register one frame from each sequence to a template mesh . Then , inside each sequence , a barycentric interpolation method is used to re-mesh other frames to unify their topology with the template mesh . Thereafter , as the first step of garment reconstruction , following the practice of previous parametric methods [ 16 , 17 ] , we estimate the canonical garment mesh for each sequence by a semantic-aware garment PCA coefficients encoder , which takes the dressed human point cloud sequences as input . The posed garment reconstruction part is where the challenges come in for two reasons . Firstly , due to unordered and unstructured nature of point clouds , it is non-trivial to learn low-level geometric features directly . Secondly , it is challenging to capture garment dynamics caused by interactions between the human body and garments . Especially , it is difficult to model the non-rigid deformation of loose garments ( e.g . skirts ) , which depends on both current human pose and previous human motions . To address the challenges , we first apply the Interpolated Linear Blend Skinning ( LBS ) to the estimated canonical garments as proposals . Unlike [ 7 , 8 , 16 ] that depends on SMPL+D model to skin the garment , the proposed Interpolated LBS has the ability to perform skinning on loose garments without artifacts . Moreover , our method does not require learning , which is different from [ 17 ] . After obtaining the proposals , we present the Proposal-Guided Hierarchical Feature Network along with the Iterative Graph Convolution Network ( GCN ) for efficient geometric feature learning . Meanwhile , a Temporal Transformer is utilized for temporal fusion to capture smooth garment dynamics . Our contributions can be summarised as : 1 ) We propose a practical pipeline for the reconstruction of various garments ( including challenging loose garments ) that are driven by large human movements , which does not rely on any particular human body parametric model or extra annotations other than human and garment meshes . 2 ) Many novel learning modules , such as Proposal-Guided Hierarchical Feature Network , Iterative GCN , and Temporal Transformer are proposed , which effectively learn spatio-temporal features from 3D point cloud sequences for garment reconstruction . 3 ) We establish a comprehensive dataset consisting of 2 , 300 3D point cloud sequences by adapting CLOTH3D [ 18 ] , which serves as the first benchmark dataset for evaluating posed garment reconstruction based on point cloud sequences . In addition , this work presents an early attempt at exploiting temporal information for capturing garment motions . To the best of our knowledge , we are the first to explore garment reconstruction from point cloud sequences . 2 Our Approach . We introduce our approach , Garment4D , in this section . Firstly , our garment model is formulated in Subsection 2.1 . Afterwards , we elaborate the three major steps of Garment4D : 2.2 sequential garments registration , 2.3 canonical garment estimation , and 2.4 posed garment reconstruction . Training loss functions are introduced in Subsection 2.5 . An overview of the proposed Garment4D is shown in Fig . 2 . 2.1 3D Garment Model . We design the garment model by the following three parts : Canonical Garment Model . Formally , the canonical garment T ∈ RN×3 can be formulated as T ( α ) = G + Cα , where G represents the mean shape of current type of garment , C is the PCA components that form garment shape sub-spaces , α represents the PCA coefficients of the corresponding garment . Interpolated LBS . One critical problem in skinning step is to get a set of skinning weights of garment vertices that can provide reasonable proposals of posed garments . Previous methods [ 7 , 8 , 16 ] approximate the garment skinning weights by using the weights of the closest human mesh vertex . However , these methods can result in mesh artifacts , and hence limit the quality of reconstructed garments for the following two facts . Firstly , the resolutions of our garment meshes is higher than that of human meshes . Secondly , loose garments like skirts do not always share same topology with human body . To avoid artifacts , we propose to interpolate the skinning weights from K nearest neighbors of human mesh vertices Th . Laplacian smoothing is also performed to further ensure the smoothness of the skinning weights . Displacement Prediction . For those challenging loose garments like skirts and dresses , the interpolated results are mostly not satisfactory . To this end , we further refine the results of interpolated LBS by predicting displacements D. In general , the whole garment model can be formulated as M ( α , θ , K , Wh , D ) = W ( T ( α ) , J , θ , Wg ( α , Th , Wh , K ) ) +D , ( 1 ) Wg ( α , Th , Wh , K ) =WKg ( T ( α ) , Th , Wh , K ) Iw ( T ( α ) , Th , K ) , ( 2 ) where θ ∈ R|J|×3 represents the axis-angle of each joint , Wh ∈ RM×|J| , Wg ∈ RN×|J| are the skinning weights of human and garment mesh vertices , J is the joint locations of human body , WKg ∈ RM×|J|×K is the garment skinning weights of K nearest body vertices , Iw ∈ RM×K is the inverse distance weights used for interpolation . 2.2 Garment Registration . In order to perform PCA decomposition to the same type of garments , we unify the mesh topology by registering the raw meshes to a template mesh T . A visualization of the registration process is illustrated in Fig . 2 ( a ) . Considering that source meshes have large shape variance and diverse sleeve/ hem length , we use a boundary-aware optimization method to align the source meshes to the target mesh . The registration loss function Lreg can be formulated as Lreg = λcLCD ( R′ , T ) + λeLEL ( R′ ) + λnLNC ( R′ ) + λbLBCD ( R′ , T ) , ( 3 ) where LCD ( R′ , T ) is the Chamfer Distance ( CD ) between source and target mesh vertices , LEL ( R′ ) normalizes Edge Length of the source mesh , LNC ( R′ ) maintains the Normal Consistency between neighbouring faces , LBCD ( R′ , T ) minimizes the CD between corresponding Boundaries of source and target meshes ( e.g . cuffs and neckline ) , and λ∗ are corresponding weights for each loss term . The LBCD term makes the optimization process aware of the boundaries of the meshes , which is essential for aligning garments with large shape variance . For each sequence , the aligned garment mesh R′ ∈ RX×3 is then used to calculate barycentric interpolation weights against the vertices of template mesh T , which is further used to re-mesh garment meshes in other frames . Specifically , for each vertex x of T , the nearest face f in R′ is found . We project x to f and then calculate the barycentric coordinate ( w1 , w2 , w3 ) of the projected point x′ on f .Using the barycentric coordinates , it is straight-forward to re-mesh all the other frames of garment meshes to the topology of the template mesh T . 2.3 Canonical Garment Estimation . To estimate canonical garments T ( α ) , we predict the PCA coefficients α by using the segmented 3D points of garments from input point cloud sequences P ∈ RT ×p×3 , where T is the sequence length . As shown in Fig . 3 ( a ) , we first perform point-wise semantic segmentation to select the 3D points of garments for each frame , which results in a sequence of garment point clouds Pg ∈ RT ×pg×3 . Subsequently , Pg is used to predict the PCA coefficients α for each sequence , which is further used to reconstruct target canonical garments by T ( α ) = G+ Cα . 2.4 Posed Garment Reconstruction . Proposal-Guided Hierarchical Feature Network . As introduced in Subsection 2.1 , the skinned garments Mp = W ( T ( α ) , J , θ , Wg ) of the proposed garment model can provide reasonable initial proposals for posed garments . However , due to complex human motions and garment flexibility , especially for intense movements and loose skirts , the initial proposals of garment poses can be far different from real poses . To address these problems , we propose a novel Proposal-Guided Hierarchical Feature Network , consisting of two major parts : 1 ) Hierarchical Garment Feature Pooling , and 2 ) Hierarchical Body Surface Encoder . Displacement D is predicted to refine the initial garment poses . Detailed architectures of these two modules are demonstrated in Fig . 3 ( b ) . The proposed Hierarchical Garment Feature Pooling can effectively capture local geometric details from garments . Hierarchical Body Surface Encoder facilitates predicting correct garment poses by considering interactions between garment and human body . By estimating proper contacts between garments and skin , our body surface encoder highly alleviates the interpenetration problems . • Hierarchical Garment Feature Pooling aims at pooling accurate geometric features and rich semantic information for each vertex of mesh proposals . Specifically , for each vertex of Mp , several query balls with different radii are constructed to sample points from the segmented point clouds of garments . To make use of the rich semantic information extracted when predicting the PCA coefficients α , different levels of down-sampled garment point clouds and the corresponding features from the previous stage are used for the query process with different radii and sample numbers . Then for each vertex , the pooled garment features are further encoded to aggregate low-level geometric features and high-level semantic information . • Hierarchical Body Surface Encoder makes each vertex of the garment mesh aware of its neighbouring human body surface , which is important for correct interaction encoding between garment and human body , especially under the circumstances of large body movements . For each vertex of Mp , neighbouring human mesh vertices are sampled using different radii of query balls . Both coordinates and normal directions of sampled human mesh vertices are further encoded . Iterative GCN Displacement . Concatenating the per-vertex features obtained from the above feature pooling and surface encoder , along with the coordinates of the vertex , we have aggregated necessary information for the following Iterative Graph Convolution Network ( GCN ) to predict the displacementD . Loose garments skinned by the interpolated LBS are normally not in accordance with the real situation . Therefore , we choose to iteratively predict displacements and gradually approach the ground truth . As shown in Fig . 3 ( b ) , for the j-th iteration , using the previous accumulated displacement predictions ∑j−1 i=1 Di and the proposal Mp , we can calculate current mesh prediction by Mpj = Mp + ∑j−1 i=1 Di . Mpj is then used for the Proposal-Guided Hierarchical Feature Pooling of this iteration . The last GCN layer of each iteration outputs current displacement Dj . Inspired by [ 19 ] , the GCN layer could be formulated as H ( l+1 ) = ReLU ( D̄−1ĀH ( l ) W ( l ) + B ( l ) ) , where Ā = A+ I is the bidirectional mesh graph with self-connections , D̄ = diag ( ∑ j Āij ) , W ( l ) and B ( l ) are the learn-able weights and bias . Temporal Transformer . To support better temporal information extraction , a Transformer is integrated into the Iterative GCN . For the j-th iteration , we take the features extracted by the ( j − 1 ) -th iteration , perform temporal fusion by Transformer and concatenate them to the input of the j-th iteration . Specifically , we denote the output feature of the second last layer of the ( j − 1 ) -th iteration as Fj−1 ∈ RT ×M×K . Query , key and value are obtained by applying MLP to Fj−1 , which could be denoted as Q , K , V = MLP ( Fj−1 ) , where Q , K , V ∈ RT ×MK . Then the value is updated as V ′ = softmax ( QKᵀ/ √ T ) V . V ′ is then concatenated to the pooled features of the j-th iteration .

Summary: The paper presents an approach for 4D garment reconstruction. To this end, it first presents a statistical shape model for garments which is learned using sequences of garment meshes. Since garment meshes across different sequences may have different topologies, the paper aligns them by first registering them to a template mesh and then re-meshing using barycentric interpolation. Given the statistical garment model, the paper then presents an approach for 4D garment reconstruction from a point-cloud sequence as input. The point cloud is first segmented into different body parts and then fed into an MLP to predict the PCA coefficients of the garment model. This provides the garment mesh in the canonical pose. Subsequently, for each body pose, the canonical mesh is posed using the LBS skinning weights interpolated from the body model. These posed meshes are further refined by iteratively predicting additional displacements using hierarchical feature extractors and a transformers-based temporal model. Experiments are performed on the CLOTH3D dataset and compared with adapted baselines. The proposed method is shown to outperform the baselines. Additional ablative experiments demonstrate the importance of different design choices.