Title: Learning 3D Garment Animation from Trajectories of A Piece of Cloth
Venue: NeurIPS 2024
Accepted: yes
Keywords: Garment

Cloth

Simulation

Constitutive Model

3D
Abstract: Garment animation is ubiquitous in various applications, such as virtual reality, gaming, and film producing. Recently, learning-based approaches obtain compelling performance in animating diverse garments under versatile scenarios. Nevertheless, to mimic the deformations of the observed garments, data-driven methods require large scale of garment data, which are both resource-wise expensive and time-consuming. In addition, forcing models to match the dynamics of observed garment animation may hinder the potentials to generalize to unseen cases. In this paper, instead of using garment-wise supervised-learning we adopt a disentangled scheme to learn how to animate observed garments: 1). learning constitutive behaviors from the observed cloth; 2). dynamically animate various garments constrained by the learned constitutive laws. Specifically, we propose Energy Unit network (EUNet) to model the constitutive relations in the format of energy. Without the priors from analytical physics models and differentiable simulation engines, EUNet is able to directly capture the constitutive behaviors from the observed piece of cloth and uniformly describes the change of energy caused by deformations, such as stretching and bending. We further apply the pre-trained EUNet to animate various garments based on energy optimizations. The disentangled scheme alleviates the need of garment data and enables us to utilize the dynamics of a piece of cloth for animating garments. Experiments show that while EUNet effectively delivers the energy gradients due to the deformations, models constrained by EUNet achieve more stable and physically plausible performance comparing with those trained in garment-wise supervised manner.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to transfer the deformations of the observed garments to any other garment. Previous methods either rely on a large-scale dataset for training or analytical physics model with limited expressive ability. On the contrast, the proposed method first learns the constitutive relations from the observation by a neural network (EUNet), then use it as an energy prior to regularize the training of garment deformation model. This design addresses the limitation of previous works and show better results.\n\nSTRENGTHS:\nThe strength of the paper is the proposed method does not need to collect huge amount of data with varied body pose and shape and garment types for training. Through theoretical analysis, they prove that they can learn a more physically accurate energy model to describe the deformation of garment. In this way, they do not need an explicit physical model, which tends to have limited expressive power. The derivation is theoretical sound.\n\nWEAKNESSES:\nTo learn this energy model by EUNet, the authors rely on the synthetic data simulated with blender. A cloth with known geometry (vertices and faces) is assigned with a specific material type. However, this setting is too ideal. In real scenarios, we are more interested in transferring the material of a real cloth to another garment. But having the geometry of a real cloth usually is not feasible. Even though we can have the mesh of the cloth through some registration process, how to get the shape of mesh when it is hanging and dangling is still a problem. In this paper, I do not see the possibility of using the proposed method in real applications. This is the critical weakness.\n\nQUESTIONS:\n1. In Fig. 4, the results of MGN-S+EUNet on dress are not similar to the ground truth data. \n2. What is the unit of the errors in Table 1? The errors on the leather and denim seem too big compared with the others.\n\nLIMITATIONS:\nSee the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission presents a method that could effectively learn the dynamic patterns of different garments from a single piece of cloth. The key insight is that the motion of different cloths is governed by both external forces and the constitutive relations rather than specific garment topologies. Thus, an Energy Unit Network (EUNet) is proposed to learn the topology independent constitutive laws. Then the animation is performed by energy optimizations. Experimental results shows improved results comparing to previous methods and baseline methods.\n\nSTRENGTHS:\nThe paper is well written and easy to read.\n1)\tThe paper is well structured.\n2)\tMany terms are well defined and explained.\nThe proposed method is both novel and interesting.\n1)\tThe disentangled learning scheme of using a network to learn the constitutive law, which generalizes to different garment types, is physically intuitive and natural. More importantly, this design helps alleviate the needs for large amount of training data of various cloth shapes in dynamic for learning based animation.  This disentanglement between topology and energy is achieved by using mesh edge as a unit instead of the whole cloth mesh.\n2)\tThe proposed disturbance training strategy helps stabilize the training and improves the generalization of EUNet. As a constraint, it accompanies the direct supervision on the energy form by taking into account the physical meaning of equilibrium state. This helps the network to learn a more reasonable manifold of the energy distribution.\nExperimental results:\n1)\tImproved results over previous methods are shown both qualitatively and quantitatively.\n2)\tAccording to the ablation study, the design of both the contrastive loss and dissipation unit loss are validated as effective.\n\nWEAKNESSES:\nSome details regarding the design of the EUNet is missing. Although some descripsions on the EUNet design is provided at the experimental section, it is relatively hard for the reader to follow and develop a more coherent understanding of the presented work.\n\nSome limitations and open questions that the work might not cover:\nWhat about anisotropic materials? How to adapt the current model design to also fit to cloths where its material is anisotropic? How well does the method handles cloths with more complex topology that goes beyond a single layer of cloth? As also pointed by the authors, the method does not handle self-collision. It would be interesting to see how it can be adapted in that axis.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to learn the constitutive model of cloth materials from observed cloth trajectory using a neural network. It adopts an MLP that operates on individual edges and predicts per-edge distortion based on the deviation of edge geometry from rest shape and trains the network using a combination of supervision on potential energy change with ground truth and optimality of incremental potential. The learned potential energy can be used as a constraint to train neural simulators for garment animation.\n\nSTRENGTHS:\n- I appreciate the novelty in the idea of learning the constitutive model of cloth materials in a data-driven manner. Potentially this formulation could allow the neural networks to understand the intrinsic physical property instead of mimicking the behavior of specific examples, thus of scientific significance if implemented correctly.\n- The paper is well-written and mostly clear.\n\nWEAKNESSES:\n- On the methodology side, the major question is probably the design of dissipative energy. On the one hand, why it is and is merely a function of \\(X^t - X^{t-1}\\) is questionable. In fact, whether it should be modeled as an absolute quantity is a question because the total amount of dissipative energy seems not that meaningful. The only observable quantity is the relative change of dissipative energy in a physical process.\n- On the other hand, with the presented framework, it is very hard to learn the major sources of energy dissipation: collision, and friction, since they are neither present in the training data, nor fully modeled (e.g. self-collision) in the formulation. While the dissipative energy is not the focus, the problem is that without correctly modeling dissipative forces, I doubt the possibility of learning an accurate elastic potential energy function.\n- On the evaluation side, the problem is that the method is only evaluated in a simplified setting, without comparing against methods or in settings that are practically useful (see below). In my opinion, there are two ways to demonstrate that the learned constitutive model is useful: either 1. demonstrate that it is more accurate than an analytical model on real data, or 2. show that it leads to more realistic animation than existing methods (including traditional numerical models).\n- The evaluation section only shows that the MGN trained with the learned constitutive model is better than those trained with ground truth garments or analytical constitutive model. On the one hand, it does not compare with other state-of-the-arts like HOOD, SNUG, and PBNS that are also formulated in a self-supervised manner. On the other hand, the claim that it is better than the analytical constitutive model is not convincing because the discrepancy may be caused by the limited accuracy in the neural simulator (or even by the mini-network mentioned in Sec 3.3). To truly demonstrate that it is better than an analytical one, it must be compared using a numerical integrator that is guaranteed to converge to the energy minimum.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nThe discussion seems adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for animating garments by learning from a single piece of cloth. This approach circumvents the need for large-scale garment datasets, which are resource-intensive and time-consuming to create. The core idea is to use a disentangled scheme where constitutive behaviors are learned from observed cloth and then applied to animate various garments. The proposed Energy Unit Network (EUNet) captures constitutive relations in the form of energy, bypassing the need for traditional physics models.\n\nSTRENGTHS:\nThe paper introduces a novel disentangled approach that separates the learning of constitutive behaviors from the animation process.\n\n The EUNet models constitutive behaviors using energy units, allowing for direct learning from observed cloth trajectories without traditional physics models.\n\nThe approach significantly reduces the data requirement, relying on a single piece of cloth for training, making it more practical and less resource-intensive.\n\nThe method produces animations that are both robust and generalizable, capable of handling various garment types and materials.\n\nWEAKNESSES:\nThe energy optimization process, although effective, can be computationally intensive and may require fine-tuning to achieve optimal results.\n\nThe paper would benefit from more extensive experimental validation, including comparisons with a broader range of existing methods and more diverse garment types.\n\nQUESTIONS:\nHow does the performance of EUNet compare with traditional physics-based models in terms of computational efficiency and accuracy?\n\nWhat are the limitations of using a single piece of cloth for training, and how can these limitations be mitigated in future work?\n\nLIMITATIONS:\nComparisons with a broader range of existing methods would be appreciated, for example, one of the SOTA method named 'HOOD: hierarchical graphs for generalized modelling of clothing dynamics' which has been cited in the paper is worth comparing with.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to learn garment dynamics using a disentangled learning framework and the Energy Unit Network (EUNet). Instead of relying on extensive garment datasets, the approach learns constitutive behaviors from a single cloth piece and dynamically animates garments through energy optimization.\n\nSTRENGTHS:\nThe writing is clear and technical details are described clearly.  The visual aids and diagrams are well-integrated, enhancing understanding.\n\nWEAKNESSES:\nMy main problem with the paper is that the problem of learning/recovering cloth dynamics from structured sample tests has been studied intensively for a long time, and the authors seem not to be aware of this whole field.  This is a well-studied problem, and the authors do not position their method against significant prior works. Many existing works have also attempted learning from real-world fabric sample tests or indirect representations (video), which is a much harder problem. To mention a few:\n\n1. \"Predicting the Drape of Woven Cloth Using Interacting Particles\" Breen et al., 1994\n2. \"Estimating Cloth Simulation Parameters from Video\" Bhat et al., 2003\n3. \"Data-driven elastic models for cloth: Modeling and measurement\" Wang et al., 2011\n4. \"How Will It Drape Like? Capturing Fabric Mechanics from Depth Images\" Rodriguez-Pardo et al., 2023\n5. \"Estimating Cloth Simulation Parameters From Tag Information and Cusick Drape Test\" Ju et al., 2024\n\nThe authors should thoroughly review the literature and reposition their contribution and provide experimental comparisons against existing works.  Additionally, the literature review section does not include important works from the physics simulation community. Including these references and discussing how the proposed method builds upon or differs from them would strengthen the paper significantly.\n\nQUESTIONS:\n1. Literature Positioning: Can you clarify your awareness and positioning of your method in relation to the existing body of work on learning/recovering cloth dynamics from structured sample tests? As discussed above, many significant studies in this area were not referenced.\n2. Physics Simulation/Graphics Community References: The literature review section did not include important works from the physics simulation and graphics community. How does your approach relate to or differ from the significant contributions in this field? Including a discussion of this could provide a better contextual grounding for your work.\n3. Experimental Validation: Can you provide more details on how your experiments validate the proposed method against these existing works? Specific comparisons and metrics would help clarify the effectiveness and novelty of your approach. Can you validate your approach against different datasets, including synthetic datasets generated from different simulation engines as well as real-world datasets used by current works?\n\nLIMITATIONS:\nThe authors provided a discussion on the limitations of edge-wise discretization and a lack of self-collision handling.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-18"}