Skinning a Parameterization of Three-Dimensional Space for Neural Network Cloth

1 INTRODUCTION . Cloth is particularly challenging for neural networks to model due to the complex physical processes that govern how cloth deforms . In physical simulation , cloth deformation is typically modeled via a partial differential equation that is discretized with finite element models ranging in complexity from variational energy formulations to basic masses and springs , see e.g . Baraff & Witkin ( 1998 ) ; Bridson et al . ( 2002 ; 2003 ) ; Grinspun et al . ( 2003 ) ; Baraff et al . ( 2003 ) ; Selle et al . ( 2008 ) . Mimicking these complex physical processes and numerical algorithms with machine learning inference has shown promise , but still struggles to capture high-frequency folds/wrinkles . PCA-based methods De Aguiar et al . ( 2010 ) ; Hahn et al . ( 2014 ) remove important high variance details and struggle with nonlinearities emanating from joint rotations and collisions . More recently , Gundogdu et al . ( 2019 ) ; Santesteban et al . ( 2019 ) ; Patel et al . ( 2020 ) ; Jin et al . ( 2020 ) leverage body skinning Magnenat-Thalmann et al . ( 1988 ) ; Lander ( 1998 ) ; Lewis et al . ( 2000 ) to capture some degree of the nonlinearity ; the cloth is then represented via learned offsets from a co-dimension one skinned body surface . Building on this prior work , we propose replacing the skinned co-dimension one body surface parameterization with a skinned ( fully ) three-dimensional parameterization of the volume surrounding the body . We parameterize the three-dimensional space corresponding to the volumetric region of air surrounding the body with a tetrahedral mesh . In order to do this , we leverage the work of Lee et al . ( 2018 ; 2019 ) , which proposed a number of techniques for creating and deforming such a tetrahedral mesh using a variety of skinning and simulation techniques . The resulting kinematically deforming skinned mesh ( KDSM ) was shown to be beneficial for both hair animation/simulation Lee et al . ( 2018 ) and water simulation Lee et al . ( 2019 ) . Here , we only utilize the most basic version of the KDSM , assigning skinning weights to its vertices so that it deforms with the underlying joints similar to a skinned body surface ( alternatively , one could train a neural network to learn more complex KDSM deformations ) . This allows us to make a very straightforward and fair comparison between learning offsets from a skinned body surface and learning offsets from a skinned parameterization of three-dimensional space . Our experiments showed an overall reduction in error of approximately 50 % ( see Table 2 and Figure 8 ) as well as the removal of visual/geometric artifacts ( see e.g . Figure 9 ) that can be directly linked to the usage of the body surface mesh , and thus we advocate the KDSM for further study . The neural network we trained for a particular body can also be used to infer cloth with unique wrinkle patterns on different body shapes and T-shirt sizes without retraining ( see supplemental material ) . In order to further illustrate the efficacy of our approach , we show that the KDSM is amenable to being used with recently proposed works on texture sliding for better three-dimensional reconstruction Wu et al . ( 2020b ) as well as in conjunction with networks that use a postprocess for better physical accuracy in the L∞ norm Geng et al . ( 2020 ) ( see Figure 10 ) . In summary , our specific contributions are : 1 ) a novel three-dimensional parameterization for virtual cloth adapted from the KDSM , 2 ) an extension ( enabling plastic deformation ) of the KDSM to accurately model cloth deformation , and 3 ) a learning framework to efficiently infer such deformations from body pose . The mean error of the cloth predicted in Jin et al . ( 2020 ) is five standard deviations higher than the mean error of our results . 2 RELATED WORK . Cloth : Data-driven cloth prediction using deep learning has shown significant promise in recent years . To generate clothing on the human body , a common approach is to reconstruct the cloth and body jointly Alldieck et al . ( 2018a ; b ) ; Xu et al . ( 2018 ) ; Alldieck et al . ( 2019a ; b ) ; Habermann et al . ( 2019 ) ; Natsume et al . ( 2019 ) ; Saito et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Bhatnagar et al . ( 2019 ) ; Onizuka et al . ( 2020 ) ; Saito et al . ( 2020 ) . In such cases , human body models such as SCAPE Anguelov et al . ( 2005 ) and SMPL Loper et al . ( 2015 ) can be used to reduce the dimensionality of the output space . To predict cloth shape , a number of works have proposed learning offsets from the body surface Guan et al . ( 2012 ) ; Neophytou & Hilton ( 2014 ) ; Pons-Moll et al . ( 2017 ) ; Lahner et al . ( 2018 ) ; Yang et al . ( 2018 ) ; Gundogdu et al . ( 2019 ) ; Santesteban et al . ( 2019 ) ; Patel et al . ( 2020 ) ; Jin et al . ( 2020 ) such that body skinning can be leveraged . There are a variety of skinning techniques used in animation ; the most popular approach is linear blend skinning ( LBS ) Magnenat-Thalmann et al . ( 1988 ) ; Lander ( 1998 ) . Though LBS is efficient and computationally inexpensive , it suffers from well-known artifacts addressed in Kavan & Žára ( 2005 ) ; Kavan et al . ( 2007 ) ; Jacobson & Sorkine ( 2011 ) ; Le & Hodgins ( 2016 ) . Since regularization often leads to overly smooth cloth predictions , additional wrinkles/folds can be added to initial network inference results Popa et al . ( 2009 ) ; Mirza & Osindero ( 2014 ) ; Robertini et al . ( 2014 ) ; Lahner et al . ( 2018 ) ; Wu et al . ( 2020b ) ; Patel et al . ( 2020 ) . Most recently , Patel et al . ( 2020 ) parameterized cloth as a submesh of the SMPL body mesh and decomposed cloth deformation into low-frequency and high-frequency components . However , this parameterization limits cloth to be bound by the topology of SMPL , and the high-frequency folds/wrinkles added by the network are not constrained to match those in the ground truth data . In contrast , our method allows one to predict cloth deformation independent of a predefined PCA basis , and using Geng et al . ( 2020 ) ensures that folds/wrinkles are physically consistent . 3D Parameterization : Parameterizing the air surrounding deformable objects is a way of treating collisions during physical simulation Sifakis et al . ( 2008 ) ; Müller et al . ( 2015 ) ; Wu & Yuksel ( 2016 ) . For hair simulation in particular , previous works have parameterized the volume enclosing the head or body using tetrahedral meshes Lee et al . ( 2018 ; 2019 ) or lattices Volino & Magnenat-Thalmann ( 2004 ; 2006 ) . These volumes are animated such that the embedded hairs follow the body as it deforms enabling efficient hair animation , simulation , and collisions . Interestingly , deforming a low-dimensional reference map that parameterizes high-frequency details has been explored in computational physics as well , particularly for fluid simulation , see e.g . Bellotti & Theillard ( 2019 ) . 3 SKINNING A 3D PARAMETERIZATION . We generate a KDSM using red/green tetrahedralization Molino et al . ( 2003 ) ; Teran et al . ( 2005a ) to parameterize a three-dimensional volume surrounding the body . Starting with the body in the T-pose , we surround it with an enlarged bounding box containing a three-dimensional Cartesian grid . As is typical for collision bodies in computer graphics Bridson et al . ( 2003 ) , we generate a level set representation separating the inside of the body from the outside ( see e.g . Osher & Fedkiw ( 2002 ) ) . See Figure 1a . Next , a thickened level set is computed by subtracting a constant value from the current level set values ( Figure 1b ) . Then , we use red/green tetrahedralization as outlined in Molino et al . ( 2003 ) ; Teran et al . ( 2005a ) to generate a suitable tetrahedral mesh ( Figure 1c ) . Optionally , this mesh could be compressed to the level set boundary using either physics or optimization , but we forego this step because the outer boundary is merely where our parameterization ends and does not represent an actual surface as in Molino et al . ( 2003 ) ; Teran et al . ( 2005a ) . Skinning weights are assigned to the KDSM using linear blend skinning ( LBS ) Magnenat-Thalmann et al . ( 1988 ) ; Lander ( 1998 ) , just as one would skin a co-dimension one body surface parameterization . In order to skin the KDSM so that it follows the body as it moves , each vertex vk is assigned a nonzero weight wkj for each joint j it is associated with . Then , given a pose θ with joint transformations Tj ( θ ) , the world space position of each vertex is given by vk ( θ ) = ∑ j wkjTj ( θ ) v j k where v j k is the untransformed location of vertex vk in the local reference space of joint j . See Figure 1d . Importantly , it can be quite difficult to significantly deform tetrahedral meshes without having some tetrahedra invert Irving et al . ( 2004 ) ; Teran et al . ( 2005b ) ; thus , we address inversion and robustness issues/details in Section 5 . 4 EMBEDDING CLOTH IN THE KDSM . In continuum mechanics , deformation is defined as a mapping from a material space to the world space , and one typically decomposes this mapping into purely rigid components and geometric strain measures , see e.g . Bonet & Wood ( 1997 ) . Similar in spirit , we envision the T-pose KDSM as the material space and the skinned KDSM as being defined by a deformation mapping to world space for each pose θ . As such , we denote the position of each cloth vertex in the material space ( i.e . T-pose , see Figure 2a ) as umoi . We embed each cloth vertex u mo i into the tetrahedron that contains it via barycentric weights λmoik , which are only nonzero for the parent tetrahedron ’ s vertices . Then , given a pose θ , a cloth vertex ’ s world space location is defined as ui ( θ ) = ∑ k λ mo ik vk ( θ ) so that it is constrained to follow the KDSM deformation , assuming linearity in each tetrahedron ( see Figure 2b ) . Technically , this is an indirect skinning of the cloth with its skinning weights computed as a linear combination of the skinning weights of its parent tetrahedron ’ s vertices , and leads to the obvious errors one would expect ( see e.g . Figure 3 , second row ) . The KDSM approximates a deformation mapping for the region surrounding the body . This approximation could be improved via physical simulation ( see e.g . Lee et al . ( 2018 ; 2019 ) ) , which is computationally expensive but could be made more efficient using a neural network . However , the tetrahedral mesh is only well suited to capture deformations of a volumetric three-dimensional space and as such struggles to capture deformations intrinsic to codimension one surfaces/shells including the bending , wrinkling , and folding important for cloth . Thus , we take further motivation from constitutive mechanics ( see e.g . Bonet & Wood ( 1997 ) ) and allow the cloth vertices to move in material space ( the T-pose ) akin to plastic deformation . That is , we use plastic deformation in the material space in order to recapture elastic deformations ( e.g . bending ) lost/recovered when embedding cloth into a tetrahedral mesh . These elastic deformations are encoded as a pose-dependent plastic displacement for each cloth vertex , i.e . di ( θ ) ; then , the pose-dependent , plastically deformed material space position of each cloth vertex is given by umi ( θ ) = u mo i + di ( θ ) . Given a pose θ , umi ( θ ) will not necessarily have the same parent tetrahedron or barycentric weights as umoi ; thus , a new embedding is computed for umi ( θ ) obtaining new barycentric weights λmik ( θ ) . Using this new embedding , the position of the cloth vertex in pose θ will be ui ( θ ) = ∑ k λ m ik ( θ ) vk ( θ ) . Ideally , if the di ( θ ) are computed correctly , ui ( θ ) will agree with the ground truth location of cloth vertex i in pose θ . The second row of Figure 4 shows cloth in the material space T-pose plastically deformed such that its skinned location in pose θ ( Figure 4 , first row ) well matches the ground truth shown in the first row of Figure 3 . Learning di ( θ ) for each vertex can be accomplished in exactly the same fashion as learning displacements from the skinned body surface mesh , and thus we use the same approach as proposed in Jin et al . ( 2020 ) . Afterwards , an inferred di ( θ ) is used to compute umi ( θ ) followed by λmik ( θ ) , and finally ui ( θ ) . Addressing efficiency , note that only the vertices of the parent tetrahedra of um ( θ ) need to be skinned , not the entire tetrahedral mesh . In order to compute each training example ( θ , d ( θ ) ) , we examine the ground truth cloth in pose θ , i.e . uGT ( θ ) . For each cloth vertex uGTi ( θ ) , we find the deformed tetrahedron it is located in and compute barycentric weights λGTik ( θ ) resulting in u GT i ( θ ) = ∑ k λ GT ik ( θ ) vk ( θ ) . Then , that vertex ’ s material space ( T-pose ) location is given by umi ( θ ) = ∑ k λ GT ik ( θ ) v m k where v m k are the material space ( T-pose ) positions of the tetrahedral mesh ( which are the same for all poses , and thus not a function of θ ) . Finally , we define di ( θ ) = umi ( θ ) − u mo i .

Summary: This paper proposes to model 3D cloth by embedding it into kinematically deforming skinned mesh (KDSM)[1], a tetrahedral mesh that parametrizes the volumetric region around the underlying body. A KDSM can be created and deformed using a variety of skinning and simulation techniques introduced in [1]. This paper extends KDSM by enabling plastic deformation in material space (T-pose), and accurately models the cloth deformation as per-vertex offsets. Inspired by [2], this paper trains a neural network to learn the per-vertex offset as a function of body pose. Once trained, the network is able to infer the 3D cloth on a particular body. Experiments show that the proposed 3D cloth parameterization method is better than the 2D UV parameterization method used in [2].