Skinning a Parameterization of Three-Dimensional Space for Neural Network Cloth
Authors: Jane Wu
Zhenglin Geng
Hui Zhou
Ronald Fedkiw
Venue:  (2021)

Abstract: We present a novel learning framework for cloth deformation by embedding virtual cloth into a tetrahedral mesh that parametrizes the volumetric region of air surrounding the underlying body. In order to maintain this volumetric parameterization during character animation, the tetrahedral mesh is constrained to follow the body surface as it deforms. We embed the cloth mesh vertices into this parameterization of three-dimensional space in order to automatically capture much of the nonlinear deformation due to both joint rotations and collisions. We then train a convolutional neural network to recover ground truth deformation by learning cloth embedding offsets for each skeletal pose. Our experiments show significant improvement over learning cloth offsets from body surface parameterizations, both quantitatively and visually, with prior state of the art having a mean error five standard deviations higher than ours. Without retraining, our neural network generalizes to other body shapes and T-shirt sizes, giving the user some indication of how well clothing might fit. Our results demonstrate the efficacy of a general learning paradigm where high-frequency details can be embedded into low-frequency parameterizations.

Under review as a conference paper at ICLR 2021
SKINNING A PARAMETERIZZATIN OF THREE-DIMENSIONAL SPACE FOR NEURAL NETWORK CLOTH
Anonymous authors Paper under double-blind review
ABSTRACT
We present novel learning framework for cloth deformation by embedding virtual cloth into a tetrahedral mesh that parametrizes the volumetric region of air surround- ing the underlying body. In order to maintain this volumetric parameterization during character animation, the tetrahedral mesh is constrained to follow the body surface it deforms. We embed the cloth mesh vertices into this parameterization of three-dimensional space in order to automatically capture much of the nonlinear deformation due both joint rotations and collisions. We then train convolutional neural network to recover ground truth deformation by learning cloth embedding offsets for each skeletal pose. Our experiments show significant improvement over learning cloth offsets from body surface parameterizations, both quantitatively and visually, with prior state of the having mean error five standard deviations higher than ours. Without retraining, our neural network generalizes to other body shapes and T-shirt sizes, giving the user some indication of how well clothing might fit. Our results demonstrate the efficacy of a general learning paradigm where high-fiee gegh-frequency details can be embedded into low-frequency parameterizations.
INTRODUCTION
Cloth is particularly challenging for neural networks to model due to the complex physical processes that govern how cloth deforms. physical simulation, cloth deformation is typically modeled via a partial diffcrential equation that is discretized with finite element models ranging in complexity from variational energy formulations basic masses and springs, see e.g. Baraff Witkin (1998); Bridson et al. (2002; 2003); Grinspun et al. (2003); Baraff et al. (2003): Selle et al. (2008). Mimicking these complex physical processes and numerical algorithms with machine learning inference has shown promise, but still struggles to capture high-frequency folds/wrinkles. PCA-based methods De Aguiar et al. (2010); Hahn et al. (2014) remove important high variance details and struggle with nonlincarities emanating from joint rotations and collisions. More recently. Gundogdu et al. (2019); Santesteban et al. (2019); Patel et al. (2020); Jin et al. (2020) leverage body skinning Thalennn-TThammnnn et al. (1988); Lander ( (1998); Lewis et al. (2000) to capture some degree of the nonlinearity; the cloth is then represented via learned offsets from a co-dimension one skinned body surface. Building on this prior work, we propose replacing the skinned co-dimension one body surface parameterization with a skinned (fully) three-dimensional parameterization of the volume surrounding the body.
We parameterize the three-dimensional space corresponding to the volumetric region of air sur- rounding the body with a tetrahedral mesh. In order to do this, we leverage the work of Lee al. (2018; 2019). which proposed number of techniques for creating and deforming such a tetrahedral mesh using variety of skinning and simulation techniques. The resulting kinematically deforming skinned mesh (KDSM) was shown to be beneficial for both hair animation/simulatioo Lee et al. (2018) and water simulation Lee et al. (2019). Here, we only utilize the most basic version of the KDSM, assigning skinning weights to its vertices so that it deforms with the underlying joints similar to a skinned body surface (alternatively, one could train a neural network to learn more complex KDSM deformations). This allows make a very straightforward and fair comparison between learning offsets from a skinned body surface and learning offsets from a skinned parameterization of cires-immnsiioal space. Our experiments showed overall reduction in error of approximately

---

Under review as a conference paper at ICLR 2021
50% (see Table and Figure 8) as well as the removal of visual/geometric artifacts (see e.g. Figure 9) that can be directly linked to the usage of the body surface mesh, and thus we advocate the KDSM for further study. The neural network we trained for particular body can also be used to infer cloth with unique wrinkle patterns on different body shapes and T-shirt sizes without retraining (see supplemental material). In order further illustrate the efficacy of our approach, we show that the KDSM is amenable to being used with recently proposed works on texture sliding for better three-dimensional reconstruction Wu et al. (2020b) as well as in conjunction with networks that use postprocess for better physical accuracy in the L norm Geng et al. (2020) (see Figure 10).
In summary, our specific contributions are: 1) a novel diceeniimnnii parameterization for virtual cloth adapted from the KDSM, 2) an extension (enabling plastic deformation) of the KDSM to accurately model cloth deformation, and 3) learning framework efficiently infer such deformations from body pose. The mean error of the cloth predicted in Jin et al. (2020) is five standard deviations higher than the mean error of our results.
2 RELATED WORK
Cloth: Data-driven cloth prediction using deep learning has shown significant promise in recent years. To generate clothing on the human body, a common approach is to reconstruct the cloth and body jointl Alldieck et al. (2018a;;); Xu et al. (2018); Alldieck et al. (2019a;b); Habermann et al. (2019); Natsume et al. (2019); Saito et al. (2019); Yu et al. (2019); Bhatnagar et al. (2019); Onizuka et al. (2020): Saito et al. (2020). In such cases, human body models such as SCAPE Anguelov et al. (2005) and SMPL Loper et al. (2015) can be used to reduce the dimensionality of the output space. To predict cloth shape, a number of works have proposed learning offsets from the body surface Guan et al. (2012); Neophytou & Hilton (2014); Pons-Moll et al. (2017); Lahner et al. (2018); Yang et al. (2018); Gundogdu et al. (2019); Santesteban et al. (2019); Patel et al. (2020); Jin et al. (2020) such that body skinning can be leveraged. There variety of skinning techniques used in animation; the most popular approach is lincar blend skinning (LBS) Magnenat-Thalman (1988); Lander (1998). Though LBS is efficient and computationally inexpensive, it suffers from well-known artifacts addressed in Kavan & Zára (2005); Kavan al. (2007); Jacobson & Sorkine (201 Le & Hodgins (2016). Since regularization often leads to overly smooth cloth predictions, additional wrinkles/folds can be added to initial network inference results Popa al. (2009); Mirza & Osindero (2014); Robertini et al. (2014); Lahner et al. (2018); Wu et al. (2020b); Patel et al. (2020). Most recently, Patel et al. (2020) parameterized cloth submesh of the SMPL body mesh and decomposed cloth deformation into low-frequency and high-frequency components. However, this parameterization limits cloth be bound by the topology of SMPL, and the folds/wrinkles added by the network are not constrained to match those in the ground truth data. In contrast, our method allows one to predict cloth deformation independent of a predefined PCA basis, and using Geng et al. (2020) ensures that folds/wrinkles are physically consistent.
et al.
Parameterizing the air surrounding deformable objects a way of treating collisions during physical simulation Sifakis et al. (2008); Müller et al. (2015); Wu & Yuksel (2016). For hair simulation in particular, previous works have parameterized the volume enclosing the head or body using tetrahedral meshes Lee et al. (2018; 2019) or lattices Volino & Thallamn (2004; 2006). These volumes are animated such that the embedded hairs follow the body as deforms enabling efficient hair animation, simulation, and collisions. Interestingly, deforming low-dimensional reference map that parameterizes high-frequency details has been explored in computational physics well, particularly for fluid simulation, see e.g. Bellotti & Theillard (2019).
3D Parameterization:
3 SKINNING A 3D PARAMETERIZATION
We generate KDSM using red/green tetrahedralization Molino et al. (2003); Teran et al. (2005a) to parameterize a three-dimensional volume surrounding the body. Starting with the body in the T-pose, we surround it with enlarged bounding box containing a three-dimennionnll Cartesian grid. As is typical for collision bodies in computer graphics Bridson et al. (2003), we generate a level set representation separating the inside of the body from the outside (see e.g. Osher & Fedkiw (2002)). See Figure la. Next, a thickened level set is computed by subtracting a constant value from the current level set values (Figure lb). Then, we use red/green tetrahedralization as outlined in Molino
2

---

Under review as a conference paper at ICLR 2021
al. (2003); Teran et al. (2005a) generate suitable tetrahedral mesh (Figure Ic). Optionally, this mesh could be compressed to the level set boundary using either physics or optimization, but we forego this step because the outer boundary is merely where paramcterization ends and does not represent an actual surface as in Molino et al. (2003); Teran et al. (2005a).
Skinning weights assigned to the KDSM using linear blend skinning (LBS) Magnenat- Thaamnn et al. (1988); Lander (1998), just as one would skin a co-dimension one body surface parameterization. In order skin the KDSM so that follows the body as moves, each vertex eg is assigned a nonzero weight Waj for each joint it is associated with. Then, given a pose 0 with joint transformations T3 (0), the world space position of each vertex is given by vx) Σ; Wky W (0)og, where v is the untransformed location of vertex in the local reference space of joint j. See Figure 1d. Importantly, it can be quite difficult to significantly deform tetrahedral meshes without having some tetrahedra invert Irving et al. (2004); Teran et al. (2005b); thus, we address inversion and robustness issues/details in Section 5.
(b)
(c)
(d)
Figure 1: We build a tetrahedral mesh surrounding the body to parameterize the enclosed three- dimensional space. First, level set representation of the body (a) is generated and subsequently thickened (b) to contain the clothing worn on the body. Then, we use red/green tetrahedralizattio Molino et al. (2003): Teran et al. (2005a) to create a tetrahedral mesh (c) from the thickened level set. This tetrahedral mesh is skinned to follow the body as it moves (d). Note that the tetrahedral mesh surrounds the whole upper body to demonstrate that this parameterization can also be used for long-sleeve shirts.
EMBEDDING CLOTH IN THE KDSM
In continuum mechanics, deformation is defined as a mapping from a material space to the world space, and one typically decomposes this mapping into purely rigid components and geometric strain measures, see e.g. Bonet & Wood (1997). Similar in spirit, we envision the T-pose KDSM as the material space and the skinned KDSM as being defined by deformation mapping world space for each pose 0. As such, we denote the position of each cloth vertex in the material space (i.e. T-pose, see Figure 2a) as u: We embed each cloth vertex u.!" into the tetrahedron that contains it via barycentric weights X4t which are only nonzero for the parent tetrahedron's 's vertices. Then, given a pose 6, a cloth vertex's 's world space location defined as (,) = Σ, >T constrained to follow the KDSM deformation, assuming linearity in each tetrahedron (see Figure 2h). Technically, this an indirect skinning of the cloth with its skinning weights computed as linear combination of the skinning weights of its parent tetrahedron's 's vertices, and leads to the obvious errors one would expect (see e.g. Figure 3, second row).
u ux (0) so that is
The KDSM approximates deformation mapping for the region surrounding the body. This ap- proximation could be improved via physical simulation (see e.g. Lee et al. (2018; 2019)). which computationally expensive but could be made more efficient using a neural network. However, the tetrahedral mesh is only well suited to capture deformations of volumetric three-dimensiona space and as such struggles to capture deformations intrinsic codimension one surfaces/shells including the bending, wrinkling, and folding important for cloth. Thus, we take further motivation from constitutive mechanies (see e.g. Bonet & Wood (1997)) and allow the cloth vertices to move in material space (the T-pose) akin to plastic deformation. That is, we use plastic deformation in the material space in order to recapture elastic deformations (e.g. bending) lost/recovered when embedding cloth into tetrahedral mesh. These elastic deformations are encoded pose-dependent plastic displacement for each cloth vertex, i.c. d0); then, the pose-dependent, plastically deformed material space position of each cloth vertex is given by tith (0) u + (0).
3

---

Under review as a conference paper at ICLR 2021
Given pose 8. u," (f) will not necessarily have the same parent tetrahedron or barycen- tric weights as u, thus, new embedding is computed for uz (0) obtaining new barycentric weights (O). Using this new embedding. the position of the cloth vertex in pose 0 will be (,( = Σx A (0)) Ideally, if the di (0) are computed correctly, 2, (6) will agree with the ground truth location of cloth vertex i pose The second row of Figure 4 shows cloth in the material space T-pose plastically deformed such that its skinned location in pose 0 (Figure 4, first row) well matches the ground truth shown in the first row of Figure 3. Learning (0) for each vertex can be accomplished in exactly the same fashion learning displacements from the skinned body surface mesh, and thus the same approach as proposed in Jin et al. (2020). Afterwards, an inferred d0( is used to compute uf" (0) followed by AgR (0), and finally (0). Addressing efficiency, that only the vertices of the parent tetrahedra of u² (0) need to be skinned, not the entire tetrahedral mesh.
(a)
(b)
Figure 2: One can embed the cloth into the T-pose KDSM (a) and fix this embedding as the KDSM deforms (b). However, this results in undesired artifacts in the cloth (see e.g. Figure 3, second row).
(a)
(b)
Figure 3: (a) The ground truth cloth and (b) skinning the cloth using a fixed tetrahedral embedding. Note how poorly this naive embedding of the cloth into the KDSM matches the ground truth (especially as compared to a more sophisticated embedding using our plastic deformation as shown in Figure 4).
(a)
(b)
Figure 4: (a) The hybrid cloth embedding method (see Section 5) produces cloth u(0) that closely matches the ground truth shown in the first row of Figure 3. (b) This is accomplished, for each pose, by plastically deforming the cloth material space (the T-pose) before embedding it to follow the deformation of the KDSM.
In order to compute each training example (8, d()). we examine the ground truth cloth in pose i.e. GT (0). For each cloth vertex u,T (0), find the deformed tetrahedron it is located in and
4

---

Under review as a conference paper at ICLR 2021
compute barycentric weights X (0) resulting in (0) Aik (0) Then, that vertex's 's material space (T-pose) location is given by u." = En 1,k (0)eg" where upg are the material space (T-pose) positions of the tetrahedral mesh (which are the same for all poses, and thus not a function of 0). Finally, we define (0) u (0) u
5 INVERSION AND ROBUSTNESS
Unfortunately, the deformed KDSM will generally contain both inverted and overlapping tetrahedra, both of which can cause a ground truth cloth vertex u, (0) to be contained in more than one deformed tetrahedron, leading multiple candidates for (0) and d, d0) Although physical simulation can be used to reverse some of these inverted elements Irving et al. (2004); Teran et al. (2005b) as was done in Lee et al. (2018; 2019), it typically not feasible remove all inverted tetrahedra. Additionally, overlapping tetrahedra occur quite frequently between the arm and the torso, especially because the KDSM needs to thick enough to ensure that contains the cloth as it deforms.
Before resolving which parent tetrahedron each vertex with multiple potential parents should be embedded into, we first robustly assemble list of all such candidate parent tetrahedra as follows. Given a deformed tetrahedral mesh v(0) in pose 0, we create a bounding box hierarchy acceleration structure Hahn (1988); Webb & Gigante (1992); Barequet et al. (1996); Gottschalk et al. (1996); Lin & Gottschalk (1998) for the tetrahedral mesh built from a slightly thickened bounding box around each tetrahedron. Then given a ground truth cloth vertex, u, CT (0). we robustly find all tetrahedra containing (or almost containing) it using minimum barycentric weight of -c with this list to remove tetrahedra that may be subject to numerical precision errors that could cause a vertex to erroneously be identified inside multiple or tetrahedra. This is done by first sorting the tetrahedra on the list based on their largest minimum barycentric weight, i.c. preferring tetrahedra the vertex is deeper inside. Starting with the first tetrahedron on the sorted list, we identify the face across from the vertex with the smallest barycentric weight and prune all of that face's vertex neighbors (and thus face/edge neighbors too) from the remainder of the list. Then, the next (non-deleted) tetrahedron on the list is considered, and the process is repeated, etc.
O. We prune
Method 1: Any of the parent tetrahedra that remain on the list may be chosen to obtain training exam- ples with zero error compared to the ground truth, although different choices lead to higher/lower variance d()) and thus higher/lower demands on the neural network. To establish a baseline, we first take the naive approach of randomly choosing u," (0) when multiple candidates exist. This can lead to high variance in d(f) and subsequent ringing artifacts during inference. See Figure 5.
Figure 5: (a) shows a training example where overlapping tetrahedra led to cloth torso vertices being embedded into arm tetrahedra, resulting high variance in d(O). Although there are various ad hoc approaches for remedying this situation, it is difficult to devise a robust strat- egy in complex regions such as the armpit. (b) shows that the ground truth u GT (0) is still cor- rectly recovered in spite of this high variance in un (0) and d(0): however, (c) shows that this high variance leads spurious ringing oscilla- tions during subsequent inference.
(a)
(b)
Method 2: Aiming for lower variance in the training data, we leverage the method of Jin al. (2020) where UV texture space and normal direction offsets from the skinned body surface are calculated for each pose 0 in the training examples. These same offsets can be used any pose, since the UVN coordinate system is still defined (albeit deformed) in every pose. Thus, we utilize these UVN offsets in our material space (T- (ooe in order to define u²"(() (f) and subsequently d(f). In particular, given the shrinkwrapped cloth in the T-pose, we apply UVN offsets corresponding to pose 8. Although this results in lower variance than that obtained from Method 1, the resulting d(0) do not exactly recover the ground truth cloth un GT (0). See Figure 6.
5

---

Under review as a conference paper at ICLR 2021
Figure 6: (a) shows the result obtained using Method 2 to compute um (0) in material space (the T-pose) for pose (b) shows the result obtained using this embedding compute u(O) as compared to the ground truth u GT (0) (c). Al- though the variance in u" (0) and d(0) is lower than that obtained using Method 1, the training examples now contain errors (shown with heat map) when compared to the ground truth.
(a)
(b)
(c)
Hybrid Method: When vertex has only one candidate parent tetrahedron, Method 1 is used. When there is more than one candidate parent tetrahedron, we choose the parent that gives an embedding closest the result of Method (in the T-pose) as long as the disagreement is below threshold (1 cm). As shown (for particular training example) in Figure 7a, this can leave number of potentially high variance vertices undefined. Aiming for smoothness, we use the Poisson morph from Cong et al. (2015) to morph from the low variance results of Method 2 to the partially-defined cloth mesh shown in Figure 7a, utilizing the already defined/valid vertices as Dirichlet boundary conditions. See Figure 7b. Although smooth, the resulting predictions may contain significant errors, and thus we only validate those that are within threshold (1 cm) of the results of Method See Figure 7c. The Poisson equation morph guarantees smoothness, while only utilizing the morphed vertices close to the results of Method limits errors (as compared to the ground truth) to some degree. This process is repeated until no newly newly morphed vertices are within the threshold (1 cm). At that point, the remaining vertices are assigned their morphed values despite any errors they might contain. See Figure 7d.
(a)
(b)
(c)
(d)
Figure 7: (a) Subset of vertices for which some choice of a parent tetrahedron using Method 1 reasonably agrees with Method 2. (b) The rest of the mesh can be filled in with the 3D morph proposed in Cong et al. (2015). (c) Subset of vertices from (b) that reasonably agree with Method 2. (d) Final result of our hybrid method (after repeated morphing).
EXPERIMENTS
Dataset Generation: We the cloth dataset from Jin et al. (2020), which consists of T-shirt meshes corre-
100
Hybrid Jin al. 2020
80
sponding to about 10,000 poses for a particular body Wu et al. (2020a). For each pose, the cloth was simulated on the scanned body, taking into account gravity, elastic and damping forces, and collision, contact and friction forces. We applied an 80-10-10 split to obtain train- ing, validation, and test datasets, respectively. Table 1 compares the maximum 2 and D norms as compared to the ground truth for each of the three methods used to generate training examples. While Method min- imizes cloth vertex errors, the resulting d(0) contains high variance. Method has significant vertex errors, but significantly lower variance in d(f). We leverage the advantages of both using the hybrid method.
Frequency
60
40
20
8.0
0.5
1.0
1.5
2,0
2.5
3.0
Avg vertex (cm)
Figure 8: Histogram of average vertex er- rors over every example in the test dataset.
Network Training: We adapt the network architecture from Jin et al. (2020) for learning the displacements d(f), i.. by storing the displacements d(l) bixel-based cloth images for the front and back sides of the T-shirt. Given joint transformation matrices of shape 1 x 1 x 90 for pose 8, the network applies transpose convolution, batch normalization, and ReL.U activation layers. The

---

Under review as a conference paper at ICLR 2021
Method
Max Vertex Error
Avg Vertex Error
Max ||dd]
Avg ||
Method Method 2
8.9 x 10-6
9.8 x 10-7
136.5
9.35
12.7
0.549
14.9
0.75
Hybrid Method
11.6
0.021
14.7
0.79
Table 1: Dataset generation analysis (in cm). To measure variance in d(0), we calculate the change in d(0) between any two vertices that share an edge in the triangle mesh, denoted by Ad(0).
output of the network is 128 x 128 x 6, where the first three dimensions represent the predicted displacements for the front side of the T-shirt, and the last three dimensions represent those for the back side. We train with an L loss on the difference between the ground truth displacements d(0) and network predictions d(O). using the Adam optimizer Kingma & Ba (2014) with a 10-3 learning rate in Py Torch Paszke et al. (2017).
Network Inference: From the network output á(O), we define ú (0) nh + d(0), which is then embedded into the material space (T-pose) tetrahedral mesh and subsequently skinned to world space to obtain the cloth mesh prediction (0). Table summarizes the network inference results the test dataset (not used in training). While all three methods detailed in Section 5 outperform the method proposed in Jin et al. (2020), the hybrid method achieved the lowest average vertex error and standard deviation. Fig- ure 8 shows histograms of the average vertex error over all examples the test dataset for the hybrid method and Jin et al. (2020). Note that the mean error of Jin et al. (2020) five standard deviations above the mean of the hybrid method. Table 3 shows the errors volume enclosed by the cloth (after capping the neck/sleeves/torso)..
Network
Vertex Error
Jin et al. (2020)
1.19 + 0.20
KDSM (Method 1)
1.06 + 0.63
KDSM (Method 2)
0.78 + 0.17
KDSM (Hybrid)
0.52 + 0.12
Table 2: Test dataset, average vertex errors (em).
Network
Volume Error
Jin et al. (2020) KDSM (Hybrid)
2991 + 715
194 + 161
Table 3: Test dataset, average volume errors (cm²).
There are significant visual improvements as well, see c.g. Figure 9. In addition, we evaluate the hybrid method network
motion capture sequence from emu and compare the inferred cloth to the results in Jin et al. (2020). The hybrid method is able to achieve greater temporal consistency; the supplemental video. To demonstrate the efficacy of our approach in conjunction with other approaches, we apply texture sliding from Wu et al. (2020b) and the physical post process from Geng al. (2020) to the results of the hybrid method network predictions, see Figure 10.
7 DISCUSSION
In this paper, we presented a framework for learning cloth deformation using volumetric parameter- ization of the air surrounding the body. This parameterization was implicitly defined via a tetrahedral mesh that was skinned follow the body it animates, i.e. KDSM. neural network was used to predict offsets in material space (the T-pose) such that the result well matched the ground truth after skinning the KDSM. The cloth predicted using the hybrid method detailed in Section 5 exhibits half the error as compared to state-of-the-art; in fact, the mean error from Jin et al. (2020) is five standard deviations above the mean resulting from our hybrid approach. Our results demonstrate that the KDSM is promising foundation for learning virtual cloth and potentially for hair and solid/fluid interactions as well. Morcover, the KDSM should prove useful for treating cloth collisions, multiple garments, and interactions with external physics.
The KDSM intrinsically provides a more robust parameterization of dimee-dimensional space, since it contains true extra degree of freedom compared to the degenerate co- oo-dimension one body surface. In particular, embedding cloth into a tetrahedral mesh has stability guarantees that do not exist when computing offsets from the body surface. See Figure 11. We believe that the significant
7

---

Under review as a conference paper at ICLR 2021
(a) GT
(b)
(c) Jin et:
Figure 9: Test dataset example predictions (b) compared to the ground truth cloth in (a) and the results from Jin et al. (2020) in (c). Regularization can smooth the body surface offsets predicted using Jin et al. (2020) and as such reveals the underlying body shape, e.g. the belly button (indicated with red square).
(a)
(b)
(c)
Figure 10: Given the hybrid method network prediction in (a), apply texture sliding from Wu et al. (2020b) and the physics postprocess from Geng et al. (2020) as shown in (b), compared to the ground truth (c). The shown example is the in Figure 14 of Wu al. (2020b).
decrease network prediction errors least partially attributable to increased stability from using volumetric parameterization.
Figure 11: (a) Embedding cloth in a tetrahedral mesh guarantees that each transformed vertex will remain inside and thus be bounded by the displacement of its parent tetrahedron. (b) How- ever, no such bounds exist when the cloth is defined via UVN offsets from the body surface, since angle perturbations of the surface cause the cloth to move along arclength C' = vr where even small & can lead to large for large enough r.
gb
C vr
r
T
(a)
(b)
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Cmu graphics lab motion capture database. http::/ cmu. cmw.edu/.
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In 2018 International Conference on 3D Vision (3DV), pp- 98-109. IEEE, 2018a.
Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8387-8397, 2018b.
Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1175-1186, 2019a.
Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Mareus Magnor. Tex2shape: Detailed full human body geometry from single image. In Proceedings of the International Conference on Computer Vision (ICCV). IEEE, 2019b.
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM transactions on graphics (TOG), volume 24, pp. 408-416. ACM, 2005.
David Baraff and Andrew Witkin. Large steps in cloth simulation. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pp. 43-54, 1998.
David Baraff, Andrew Witkin, and Michael Kass. Untangling cloth. In ACM SIGGRAPH 2003 Papers, SIGGRAPH '03, pp. 862-870, New York, NY, USA, 2003. ACM. ISBN 1-58113-700-5.
Gill Barequet, Bernard Chazelle, Leonidas J Guibas, Joseph SB Mitchell, and Ayellet Tal. Boxtree: hierarchical representation for surfaces in 3d. Computer Graphics Forum, volume 15, pp.
387-396. Wiley Online Library, 1996.
Thomas Bellotti and Maxime Theillard. coupled level-set and reference map method for interface representation with applications to two-phase flows simulation. Journal of Computational Physics, 392:266-290 290, 2019.
Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garment net: Learning to dress 3d people from images. Proceedings of the IEEE International Conference on Computer Vision, pp. 5420-5430, 2019.
Javier Bonet and Richard D Wood. Nonlinear continuum mechanics for finite element analysis. Cambridge university press, 1997.
R. Bridson, S. Marino, and R. Fedkiw. Simulation of clothing with folds and wrinkles. In Proceedings of the 2003 ACM SIGGRAPH//Euographicc Symposium on Computer Animation, SCA '03, pp. 28-36. Aire-la-Viie Ville, Switzerland, Switzerland, 2003. Eurographics Association. ISBN 1-58113- 659-5.
Robert Bridson, Ronald Fedkiw, and John Anderson. Robust treatment of collisions, contact and friction for cloth animation. ACM Trans. Graph., 21(3):594 July 2002. ISSN 0730-0301.
Matthew Cong, Michael Bao, Jane L. E, Kiran S. Bhat, and Ronald Fedkiw. Fully automatic generation anatomical face simulation models. In Proceedings of the 14th ACM SIGGRAPH/ Eurographics Symposium on Computer Animation, SCA '15, pp. 175-183, New York, NY, USA, 2015. Association for Computing Machinery.
Edilson De Aguiar, Leonid Sigal, Adrien Treuille, and Jessica K Hodgins. Stable spaces for real-time clothing, ACM Transactions on Graphics (TOG), 29(4): 2010.
Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. Coercing machine learning to output physically accurate results. Journal of Computational Physics, 406: 109099, 2020.

---

Under review as a conference paper at ICLR 2021
Stefan Gottschalk, Ming C Lin, and Dinesh Manocha. Obbtree: A hierarchical structure for rapid interference detection. In Proceedings of the 23rd anmual conference on Computer graphics and interactive techniques, pp. 171-180, 1996.
Eitan Grinspun, Anil Hirani, Mathieu Desbrun, and Peter Schröder. Discrete shells. In Proceedings symposium on Computer animation, pp. 62-67.
of the 2003 ACM Eurographics Association, 2003.
Peng Guan, Loretta Reiss, David A Hirshberg, Alexander Weiss, and Michael Black. Drape: Dressing any person. ACM Transactions on Graphics (TOG), (1 2012.
Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, and Pascal Fua. Garnet: A two-stream network for fast and accurate 3d cloth draping. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8739-8748, 2019.
Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll and Christian Theobalt. Livecap: Real-time human performance capture from monocular video. ACM Transactions on Graphics (TOG), 38(2): 2019.
Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. Subspace clothing simulation using adaptive bases. ACM Transactions on Graphics (TOG), 33 2014.
James K Hahn. Realistic animation of rigid bodies. Acm Siggraph Computer Graphics, 22(4): 299-308, 1988.
Geoffrey Irving, Joseph Teran, and Ronald Fedkiw. Invertible finite elements for robust simulation of large deformation. In Proceedings of the 2004 ACM SIGGRAAHH/Eurgraphiss symposium on Computer animation, pp. 131-140. 2004.
Alec Jacobson and Olga Sorkine. Stretchable and twistable bones for skeletal shape deformation, In Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1-8, 2011.
Ning Jin, Yilin Zhu, Zhenglin Geng, and Ronald Fedkiw. pixel-based framework for data-driven clothing. In Proceedings of the 19th ACM SIGGRAPH Eurographics Symposium on Computer Animation, volume 39, Association for Computing Machinery, 2020.
Ladislav Kavan and Jiří Zára. Spherical blend skinning: a real-time deformation of articulated models. In Proceedings of the 2005 symposium on Interactive 3D graphics and games, pp. 9-16, 2005.
Ladislav Kavan, Steven Collins, Jifí Žára, and Carol O' O'Sulliaan. Skinning with dual quaternions. In Proceedings of the 2007 symposium on Interactive 3D graphics and games, pp. 39-46, 2007.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6990 2014.
Zorah Lahner, Daniel Cremers, and Tony Tung. Deepwrinkles: Accurate and realistic clothing modeling. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 667-684, 2018.
Jeff Lander. Skin them bones: Game programming for the web generation. Game Developer Magazine, 5(1):10-18, 1998.
Binh Huy Le and Jessica K Hodgins. Real-time skeletal skinning with optimized centers of rotation.
ACM Transactions on Graphics (TOG), 35(4):1-10, 2016.
Minjae Lee, David Hyde, Michael Bao, and Ronald Fedkiw. A skinned tetrahedral mesh for hair animation and hair-water interaction. IEEE transactions on visualization and computer graphics, 2443):1449-145, 2018.
Minjae Lee, David Hyde, Kevin Li, and Ronald Fedkiw. A robust volume conserving method for character-water interaction. In Proceedings of the 18th anmual ACM SIGGRAPH/Eunggraphic Symposium Computer Animation, pp. 1-12, 2019.
10

---

Under review as a conference paper at ICLR 2021
John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: unified approach to shape interpolation and skeleton-dre driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 165-172, 2000.
Ming Lin and Stefan Gottschalk. Collision detection between geometric models: A survey. In Proc. of IMA conference on mathematics of surfaces, volume 1, pp. 602-608, 1998.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skimned multi pulti-person linear model. ACM transactions on graphics (TOG), 34(6):248, 2015.
Nadia Magnenat- Magnenat-Thalman Thalmann, Richard Laperrire, and Daniel Thalmann. Joint-dependent local defor- mations for hand animation and object grasping. In Proceedings Graphics Interface '88, pp. 26-33, 1988.
Mehdi Mirza and Simon Osindero. arXiv: 1411: 1784, 2014.
Conditional
generative adversarial nets. arXiv preprint
Neil Molino, Robert Bridson, Joseph Teran, and Ronald Fedkiw. crystalline, red green strategy for meshing highly deformable objects with tetrahedra. In IMR, pp. 103-114 114. Citeseer, 2003.
Matthias Müller, Nuttapong Chentanez, Tae- Yong Kim, and Miles Macklin. Air meshes for robust collision handling. ACM Transactions on Graphics (TOG), 34(4): 9, 2015.
Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima. Siclope: Silhouette-based clothed people. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4480-4490, 2019.
Alexandros Neophytou and Adrian Hilton. A layered model of human body and garment deformation. In 2014 2nd International Conference on 31) Vision, volume pp. 171-178. IEEE, 2014.
Hayato Onizuka. Zehra Hayirci, Diego Thomas, Akihiro Sugimoto, Hideaki Uchiyama, and Rin- ichiro Taniguchi. Tetratsdf: 3d human reconstruction from a single image with a tetrahedral outer shell. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6021-6020, 2020.
Stanley Osher and Ronald Fedkiw. Level Set Methods and Dynamic Implicit Surfaces. Springer, New York, 2002.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Dito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. Tailornet: Predicting clothing in 3d as function of human pose, shape and garment style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7365-7375 2020.
Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. Clothcap: Seamless 4d clothing capture and retargeting. ACM Transactions on Graphics (TOG), 36(4):1-15, 2017.
Tiberiu Popa, Quan Zhou, Derek Bradley, Vladislav Kraevoy, Hongbo Fu, Alla Sheffer, and Wolfgang Heidrich. Wrinkling captured garments using space-time data-driven deformation. In Computer Graphics Forum, volume 28, pp. 427-435. Wiley Online Library, 2009.
Nadia Robertini, Edilson De Aguiar, Thomas Helten, and Christian Theobalt. Efficient multi-view performance capture of fine-scale surface detail. In 2014 2nd International Conference on 3D Vision, volume 1, pp. 5-12. IEEE, 2014.
Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the International Conference Computer Vision (ICCV). IEEE, 2019.
Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 84-93, 2020.

---

Under review as a conference paper at ICLR 2021
Igor Santesteban, Miguel A Otaduy, and Dan Casas. Learning based animation of clothing for virtual try-on. In Computer Graphics Forum, volume 38, pp. 355-366. Wiley Online Library, 2019.
Andrew Selle, Jonathan Su, Geoffrey Irving, and Ronald Fedkiw. Robust high-resolution cloth using parallelism, history-based collisions, and accurate friction. IEEE transactions on visualization and computer graphics, 15(2):339-35, 2008.
Eftychios Sifakis, Sebastian Marino, and Joseph Teran. Globally coupled collision handling using vol- ume preserving impulses. Proceedings of the 2008 ACM SIGGRAPH//Eurgrrphics Symposium Computer Animation, pp. 147-153, 2008.
Joseph Teran, Neil Molino, Ronald Fedkiw, and Robert Bridson. Adaptive physics based tetrahedral mesh generation using level sets. Engineering with computers, 21 2005a.
Joseph Teran, Eftychios Sifakis, Geoffrey Irving. and Ronald Fedkiw. Robust quasistatic finite elements and flesh simulation. In Proceedings of the 2005 ACM SIGGRAPH/Eunogappies symposium on Computer animation, pp. 181-190, 2005b.
Pascal Volino and Nadia Magnenat- Thailmnn. Animating complex hairstyles in real-time. In Proceedings of the ACM symposium on Virtual reality software and technology, pp. 41-48, 2004.
Pascal Volino and Nadia Magnenat- Thalmann. Real-time animation of complex hairstyles. IEEE Transactions Visualization and Computer Graphics, 12(2):131-142, 2006.
Maagenaa-Thallmann.
Robert Webb and Mike Gigante. Using dynamic bounding volume hierarchies improve efficiency of rigid body simulations. In Visual Computing, pp. 825-842. Springer, 1992.
Jane Wu, Zhenglin Geng, Ning Jin, and Ronald Fedkiw. Physbam virtual cloth dataset, 2020a. http: / physpam. stanford. edu.
Jane Wu, Yongxu Jin, Zhenglin Geng, Hui Zhou, and Ronald Fedkiw. Recovering geometric information with learned texture perturbations. arXiv preprint arXiv:2001.07253 2020b.
Kui Wu and Cem Yuksel. Real-time hair mesh simulation. In Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pp. 59-64, 2016.
Weipeng Xu, Avishek Chatterjee. Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. Monoperfcap: Human performance capture from monocular video. ACM Transactions on Graphics (ToG), 37(2):27, 2018.
Jinlong Yang, Séan-Sébastien Franco, Franck Ketroy-Wheelle and Stefanie Wuhrer. Analyzing clothing layer deformation statistics of 3d human motions. In Proceedings of the European Conference Computer Vision (ECCV), pp. 237-253, 2018.
Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, and Yebin Liu. Simulcap: Single-view human performance capture with cloth simulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
12

---

Under review as a conference paper at ICLR 2021
A MODIFIED BODY SHAPES AND CLOTHING SIZES
We demonstrate that our network trained to infer cloth on a particular body, e.g. from Wu et al. (2020a), can used for other body parameterizations and body shapes without retraining. Using the trained hybrid method network (see Section 6), the inferred T-shirt for a given pose is transferred to the SMPL body model Loper et (2015) as follows. First, we generate skinned KDSM for the SMPL body as described in Section 3. Next, we transfer the T-pose cloth mesh to the SMPL body in the T-pose via quasistatic simulation. Then, for any skeletal pose, KDSM embedding offsets for the cloth on the SMPI. body are inferred using the trained network. See Figure 13. The cloth can also be scaled different sizes depending on user preference. See Figure 14.
Figure 13: The network inferred cloth for the body from Wu et al. (2020a) be transferred to the SMPL body model with any given pose and shape. Column corresponds to Wu et al. (2020a), and columns 2-4 correspond to thinner, template, and thicker SMPL bodies, respectively. Note that the cloth exhibits unique wrinkling patterns depending on body shape, as expected.
13

---

Under review as a conference paper at ICLR 2021
Figure 14: The network inferred cloth can be resized based on user preference without network
retraining. The size of the T-shirt increases from left to right for three different poses.
14


Reviews:
[{"review_id": "h5VYRb0zc_0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to model 3D cloth by embedding it into kinematically deforming skinned mesh (KDSM)[1], a tetrahedral mesh that parametrizes the volumetric region around the underlying body. A KDSM can be created and deformed using a variety of skinning and simulation techniques introduced in [1]. This paper extends KDSM by enabling plastic deformation in material space (T-pose), and accurately models the cloth deformation as per-vertex offsets. Inspired by [2], this paper trains a neural network to learn the per-vertex offset as a function of body pose. Once trained, the network is able to infer the 3D cloth on a particular body. Experiments show that the proposed 3D cloth parameterization method is better than the 2D UV parameterization method used in [2].\n\nStrengths\n- This paper proposes a new approach to modeling 3D cloth deformation. Experiments demonstrate that it outperforms the UV parameterization method on modeling the per-vertex offset as a function of body pose. It has the potential to be applied to other cloth-related tasks.\n- This paper successfully adapts KDSM, which is originally used for hair modeling, to clothes. The inversion and robustness issues are addressed. It would be inspiring for other researchers to apply the similar idea on other types of objects.\n- This paper is clearly organized and well-written. There are sufficient technical details presented in the paper.\n\nWeaknesses\n- Some existing works parameterize cloth deformation based on SMPL, e.g., SMPL+D [3], TailorNet [4], Tex2Shape [5].  This paper lacks the comparison with these closely related methods. Therefore, it is not clear whether the proposed KDSM-based cloth parameterization method is better than existing SMPL-based methods.\n- There is only experimentation on synthetic data. It is not clear how it performs on real world data like BUFF [6].\n- At the end of Sec. 6, the authors claim that “the hybrid method is able to achieve greater temporal consistency”. It is not clearly which component of the method enforces temporal consistency.\n\nSuggestion\n- In Sec. 5, the authors elaborate on how to robustly handle tetrahedra inversion and overlapping when generating training examples. Being depicted in natural language, the entire process is too complicated for readers to follow. For example, the sentence “We prune this list of tetrahedra, keeping only the most robust tetrahedron near each element boundary …” makes readers wonder how the robust tetrahedron and element boundary are defined. It would be better if the authors can provide a piece of pseudocode to explain the entire process in a compact and precise manner. A graphic illustration of the key operations would also be helpful for readers to understand the process.\n \n[1] Lee, Minjae, et al. \"A skinned tetrahedral mesh for hair animation and hair-water interaction.\" IEEE transactions on visualization and computer graphics 25.3 (2018): 1449-1459.\n[2] Jin, Ning, et al. \"A pixel-based framework for data-driven clothing.\" In Proceedings of the 19th ACM SIGGRAPH / Eurographics Symposium on Computer Animation, volume 39. Association for Computing Machinery, 2020.\n[3] Bhatnagar, Bharat Lal, et al. \"Multi-garment net: Learning to dress 3d people from images.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.\n[4] Patel, Chaitanya, Zhouyingcheng Liao, and Gerard Pons-Moll. \"Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[5] Alldieck, Thiemo, et al. \"Tex2shape: Detailed full human body geometry from a single image.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.\n[6] Zhang, Chao, et al. \"Detailed, accurate, human shape estimation from clothed 3D scan sequences.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xr69viBcMfO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn the cloth deformation of a t-shirt given\nthe skeletal pose of an upper body. The method skins a thick tetrahedral mesh to\nthe skeleton and embeds the t-shirts cloth within. At inference time a network\npredicts a rest pose displacement before conducting skinning via barycentric\nlookup in the tet mesh. The method is trained (as far as I can tell) on some\ngroundtruth cloth simulation method (this is not revealed).\n\nI recommend rejecting this paper from ICLR 2021 on several grounds: 1) the\nresults are poor, 2) the description is hard to follow, 3) the methodological\nchoices are not well motivated, 4) the method as written is not reproducible,\nand 5) the claims are too general.\n\n1) In terms of topic and methodology this paper would be an appropriate\nsubmission to SIGGRAPH or SCA. Callibrating for the expected result quality at\neither venue, I would recommend acceptance. Since the machine learning component\nof this paper is not a contribution besides being an application of\n\"off-the-shelf\" tools, I do not see reason to lower these standards for ICLR.\n\n2) After reading the paper, I eventually feel I understand this method. \nImportant details are left out effecting replicability (see below). The paper\ndoes not clearly state what the input and output is. The paper does not describe\nhow groundtruth data is generated. \n\n\"This is done by first sorting the tetrahedra on the list based on their largest\nminimum barycentric weight, i.e. preferring tetrahedra the vertex is deeper\ninside\" I don't understand this. Barycentric weights are largest when near a\nvertex. Meanwhile tetrahedra can be very pancake/sliver-shaped, so that\nregardless of the barycentric coordinates, a point is never deep inside. Using\nbarycentric coordinates to measure depth of penetration is misguided.\n\nI didn't understand \"method 2\". In that method is the tet mesh entirely ignored?\n\nIs Figure 4 showing training data or withheld testing poses?\n\n3) The paper immediately jumps into the idea of embedding the cloth of a t-shirt\nin a bulbous tetrahedral mesh around the upper body. This isn't questioned until\nlater when all sorts of issues appear due to overlapping and inverted elements.\nThere was no reason to think that skinning such a thick tet mesh was a good idea\nin the first place. So the \"INVERSION AND ROBUSTNESS\" section is describing ad\nhoc heuristics to a problem that could have been avoided by starting with a more\nsound premise.\n\nThe working premise is that pose space deformations can be used for efficient\ncloth simulation. This is reasonable and traces its heritage to \"A powell\noptimization approach for example-based skinning in a production animation\nenvironment,\" which should probably be cited. From there, the choice of using a\nthick tet mesh comes without solid motivation. Why not, for example, instead\nlearn the skinning weights and displacements directly? So, that for a point p on\nthe cloth the final deformation is:\n\n∑ wi(p,θ) Ti(θ) (p + d(θ,p))\n\n?\n\nTo generalize across body types etc., rather than the heavy handed proposed\napproach of sharing this mysteriously skinned tet mesh, the learned w and d\nfunction could be predicted based on some relative position to the rest pose and\nt-shirt size etc.\n\n4) This paper is far from replicable. How is the groundtruth data computed? Some\ncloth simulation? Which method? Are collisions handled in that method? \n\nWhy does the surface boundary in Figure 1 (c) look so spiky yet the input level\nset is smooth? This does not appear in the results of the red/green\ntetrahedralization results. This looks more like a simply clipped regular grid\ntetrahedralization.\n\nHow are the weights wkj determined? Manually? Automatically? Optimized during\ntraining? This appears to be crucial to the method but left out.\n\n5) Finally, this paper claims to provide a \"Skinning a parameterization of\nthree-dimensional space for neural network cloth\". Even if I accepted this paper\nas successful in its results (I do not), then this paper could at best claim to\nhave skinned a parameterization of t-shirt deformations for upper-body motions.\nThe fragility of the method as discussed above makes this overclaiming\nespecially dubious.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ubkqy_k8oe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors derive a volumetric extension of the surface parameterization approach developed by Jin et.al. Towards this, they propose to use tetrahedral parameterization using well known techniques in computer graphics community. The kinematically deforming skinned mesh (KDSM) formulation for tetrahedral parameterization is borrowed from Lee at. al. \n\nThe combination of the above techniques coupled with some heuristics to increase robustness to inversion suggest improvement over Jin et. al. This is a very niche topic and I am not confident that the general audience stands to benefit from this specific formulation for clothes. The ICLR community would benefit by demonstrating the approach on other deformations of solids/liquids and validating the generality of the approach compared to other representations beyond virtual cloth. Only comparing to Jin et. al significantly limits the scope of the paper. \n\nThe computational complexity of the approach is completely ignored. As the gains over Jin et.al. seem to stem from extending the formulation to 3d domain, the compute should be compared. This becomes important for high dimensional solid/liquid simulation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "K30YcCcE7tO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm not very familiar with 3d cloth animation, so may not provide an entirely adequate evaluation. However it looks to me that this paper is more suited to the graphics conference like SIGGRAPH.\n\nThe paper present a new method for cloth deformation based on tetrahedral KDSM mesh, to make the deformation more natural the neural network is used to predict the offsets  that match the ground truth deformation. Differently from Jin et al. (2020), in this paper volumetric region of air surrounding the underlying body  is used.\n\nThe are several questions about the experimental part of the paper.\n\n1. The paper only compares on a single dataset of the Tshirts. So it is not clear how well the model will perform on other types of  clothes: regular shirts, dresses and so on.  The paper however claims that the method could be potentially to more broad categories such as hair and fluids. Is it possible to add more comparisons on the other clothes or object types?\n\n2. How the dataset of Tshirt meshes been obtained, is it synthetically generated? If it is synthetically generated what is the benefit of using data-driven learning method? Could the method be applied on the real world scanned meshes?\n\nMinor issues:\n- Figure 1(d) looks not intuitive, does not look like (d) is modification of (c) and (d) does not look as shirt at all. It is better to use some small modification to make it more intuitive.\n- Supplementary material should go along with the paper as a set of Appendixes. Only videos and source code should go as zip archive.\n- Paper contain weird green artifacts in the end of first and second pages which should be removed.\n- It is hard to switch attention from Figure 3 to Figure 4, these figures should be joined.\n\n—-------—---------------\n\nThe rebuttal did not change neither my confidence, nor my impression about the paper. So I did not change my rating, however I acknowledged that other reviews may be more proficient to judge this paper properly. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three of the four reviewers recommend rejection; one additional reviewer considers the paper to be marginally above threshold for acceptance but is very uncertain and this is taken into account.  The AC is in consensus with the first three reviewers that this paper is not ready yet for publication.  

There is concern from the reviewers that ICLR is not the right venue for this submission.  The author response in "Submission Update" does not clarify this concern.  Training a neural network to solve the problem does not automatically mean that ICLR or other ML conferences are necessarily the right venue.  Regardless, due to the many other raised concerns e.g. limited experimental results and comparisons as well as clarity,  the AC recommends rejection for this paper and resubmission at a more appropriate venue.  
Decision: Reject