Multi-Layered 3D Garments Animation
Authors: Yidi Shao
Chen Change Loy
Bo Dai
Venue: Submitted to ICLR 2023 (2023)

Abstract: Most existing 3D garment animation datasets are restricted to human bodies with single-layered garments. Even though cases with upper shirts and lower pants are included, only a few overlap areas among such garment combinations exist. Moreover, they often regard human body movement as the only driving factor that causes garment animation. Approaches developed on top of these datasets thus tend to model garments as functions of human body parameters such as body shape and pose. While such treatment leads to promising performance on existing datasets, it leaves a gap between experimental environments and real scenarios, where a body can wear multiple layered garments and the corresponding garment dynamics can be affected by environmental factors and garment attributes. Consequently, existing approaches often struggle to generalize to multi-layered garments and realistic scenarios. To facilitate the advance of 3D garment animation toward handling more challenging cases, this paper presents a new large-scale synthetic dataset called LAYERS, covering 4,900 different combinations of multi-layered garments with 700k frames in total. The animation of these multi-layered garments follows the laws of physics and is affected by not only human body movements but also random environmental wind and garment attributes. To demonstrate the quality of LAYERS, we further propose a novel method, LayersNet, for 3D garment animation, which represents garments as unions of particles and subsequently adopts a neural network to animate garments via particle-based simulation. In this way, the interactions between different parts of one garment, different garments on the same body, and garments against various driving factors, can be naturally and uniformly handled via the interactions of particles. Through comprehensive experiments, LayersNet demonstrates superior performance in terms of animation accuracy and generality over baselines. The proposed dataset, LAYERS, as well as the proposed method, LayersNet, will be publicly available.

Under review as a conference paper at ICLR 2023
MULTI-LAYERED 3D GARMENTS ANIMATION
Anonymous authors Paper under double-blind review
Tome
(a) Jacket Falling
(b) Hood Fallling
(c) Triple Garments
(d) Wind Blowing
Figure 1: We propose new large-scale 3D garment animation dataset LAYERS, which improves pre- vious datascts by considering multi-layered 3D garments and more driving factors for garment animation, egg. environmental wind, besides human body movements. In (a)-(d) we show the new and realistic challenges covered LAYERS but are omitted in previous datasets.
ABSTRACT
Most existing 3D garment animation datasets are restricted to human bodies with single-layered garments. Even though cases with upper shirts and lower pants are included, only few overlap areas among such garment combinations exist. Moreover, they often regard human body movement as the only driving factor that causes garment animation. Approaches developed on top of these datasets thus tend to model garments as functions of human body parameters such as body shape and pose. While such treatment leads to promising performance datasets, it leaves a gap between experimental environments and real scenarios, where body can wear multiple layered garments and the corresponding garment dynamics can be affected by environmental factors and garment attributes. Conse- quently, existing approaches often struggle to generalize to multi-layered garments and realistic scenarios. To facilitate the advance of 3D garment animation toward handling more challenging cases, this paper presents new aarge-scale synthetic dataset called LAYERS, covering 4,900 different combinations of multi-layered garments with 700k frames total. The animation of these multi-layered gar- ments follows the laws of physics and is affected by not only human body move- ments but also random environmental wind and garment attributes. To demon- strate the quality of LAYERS, 3D garment animation, which represents garments as unions of particles and sub- sequently adopts a neural network to animate garments via particle- based simula- tion. In this way, the interactions between different parts of one garment, different garments on the same body. and garments against various driving factors, can be naturally and uniformly handled via the interactions of particles. Through com- prehensive experiments, LayersNet demonstrates superior performance terms of animation accuracy and generality over baselines. The proposed dataset, LAY- ERS, as well as the proposed method, LayersNet, will be publicly available.
existing
further propose novel method, LayersNet, for
INTRODUCTION
3D garment animation has been an active and important topic in computer graphics and machine learning, due to its great potential in various downstream tasks, including virtual reality, virtual try- on, gaming and film production. While this topic has been extensively studied the past, generating realistic and faithful animation remains open research question. In particular, existing approaches are still limited in modeling diverse garments of different topologies and appearances. In addition, the complex interactions between the garment and the human body under the challenging setting of multi-layer garments and with external environmental factors remain much less explored in the literature.

---

Under review as a conference paper at ICLR 2023
To support the development of data-driven approaches for 3D garment animation (Patel al. 2020 Bertiche et al., 2020; Santesteban et al., 2021). researchers have built various datasets on real- scans and synthetic data generated by Physically Based Simulation (PBS) (Narain et al., (0122] Li 2018). However, most of existing datasets (Bertiche et al.) 2020, Patel et al. 2020 Tiwari ett al.
al. 2020 consider only human bodies with single-layered garments, where each human body wears either a single dress or an upper t-shirt with lower pants that have limited overlap. The an- imation of multi-layered garments, such as a t-shirt with a jacket, that obey sophisticated physical dynamics, remain unexplored. In addition, in existing datasets, the moving human body is com- monly regarded as the default and only driving factor in animating garments. Other factors, such as wind and friction, are left unconsidered. Such a simplification thus leads to a significant gap be- tween experimental environments and real-world applications, making most approaches developed on top of these datasets less applicable in real life.
To bridge the gap between experimental environments and real-world applications and facilitate the advance of 3D garment animation, this paper introduces a new challenging dataset called LAYERS. muLti-lAYerEd gaRmentS dataset, which is carefully generated based on a simulation engine. LAY- ERS focuses the animation of multi-layered garments, while also taking the wind, another im- portant driving factor besides the human body, into consideration. Specifically, in LAYERS, multi- layered garments are prepared combinations of inner and outer clothes, as shown in Figure [ The inner and outer garments adopt different attribute values, e.g., bend stiffness and frictions. All garments on the same human body will interact with each other, constrained by the laws of physics. They are also simultancously affected by the wind with randomly sampled direction and strength.
To demonstrate the quality of LAYERS, we further propose novel data-driven method, dubbed as LayersNet, for multi-layered 3D garment animation. The core of LayersNet is a neural network based simulation system (Shao et al. 2022) that represents garments as unions of particles. Con- sequently, all kinds of interactions during garment animation, including the interactions between different parts of one garment, the interactions between different garments on the same body, and the interactions between garments and various driving factors, can be naturally and uniformly re- garded as the interactions between particles. Hence, instead of being restricted to a specific driving factor (e.g., the human body) as previous methods, the proposed LayersNet possesses strong gener- alization ability across diverse types of human body movements, multi-layered 3D garments, as well as driving factors. As the number of particles in LayersNet are considerably large when fine-graine details of garments and human bodies are preserved, we further exploit the redundancy of garments and extend LayersNet to establish two-level structural hierarchy for garments where garments made of patches, and patches are constituted of particles of a fixed configuration. Since the number of patches is much smaller than the number of particles, the interactions between all particles can be efficiently captured by the interactions of patches.
Our contributions can be summarized as follows:
1. We propose LAYERS, a large-scale and new dynamic dataset for 3D garment animation. The dataset focuses on multi-layered 3D garments, introducing random wind and friction as additional driving factors besides human body movements.
2. On top of LAYERS, we further propose LayersNet, a novel method for 3D garment an- imation that uniformly captures interactions among garment parts, different garments, as well as garments against driving factors. The notion of unifying various interactions as particle- based simulations novel in the literature.
2 RELATED WORK
3D Garment Datasets. Publicly available 3D garment datasets are in great need. Existing datasets
are generated either from synthesis (Pumarola et al.) 2019 Patel et al. 2020; Santesteban et al.) Bertiche et al.) 2020) or real-world scans (Zhang et al. 2017 Zheng et al.| 2019) Ma et al, Tiwari et al.) 2020 Cai et al. 2022). For synthetic datasets, PDPeleee (Pumarola et al. 2019) contains mu multi-view images including RGB, depth, normal, and scene flow data. TailorNet Patel
2021
2020
et al. 2020) provides synthetic dataset with 20 different garments simulated in 1.782 static SMPL poses for nine body shapes. (Santesteban et al. (202I) contributes a dataset composed of two different garments simulated on 56 human motion sequences with 17 body shapes. Cloth3D (Bertiche al.,
2

---

Under review as a conference paper at ICLR 2023
2020) is the largest synthetic dataset, with 11,300 outfits generated from the combinations of several prototypes, such as t-shirts, tops, trousers and skirts. Most existing datasets only contain single- layered 3D garment models. Even though there exist combinations with multiple garments, such as an upper t-shirt and lower pants in Multi-Garment Net [Bhttaggrr et al. (2019), there are very few overlapping areas among different cloth pieces. Recently, Layered-Garment
Garment Net Aggarwal et al. (2022) proposes a static multi-layered garments dataset in 7 static poses for 142 bodies to generate layers of outfits from single image. However, the garments in Layered-Garment Net, which are mostly skinning clothes, do not follow physics laws and the interpenetration is solved by simply foreing penetrated vertices out of inner garments.
To our best knowledge, LAYERS is the first dataset containing dynamic multi- multi-layered 3D garments, e.g., the human model wears a dress and outer jacket which deform according to the body motions. Different layers of garments have different attributes and interact with each other, obeying the laws of physics. Moreover, we introduce wind as extra driving factor to animate the garments, enriching their dynamies given similar human movements. Our dataset includes all necessary 3D information, which is able to easily generalize to other tasks, such as reconstructions from single images.
Data-driven Cloth Model, Most existing approaches aim to estimate a function that outputs the
deformations of garments for any input. A common strategy is to learn a parametric garment model to deform the corresponding mesh templates. For example, garments are modeled as functions of human pose (Wang al., 2019), shape (Vidaurre et al. (020), pose-and-shape (Bertiche et al.,
2021 Tiwari & Bhowmick, (021). motions (Santesteban et al. [021), garment type (Ma) 2020
ct al.
2020: Patel al., (2200). The approaches mentioned above rely heavily on SMPL-based
human models and animate garments by the blend weight according to the registered templates. The generalization of such approaches is limited to skinning clothes. Some recent studies explore bone-driven motion networks (Pan et al. (022) to animate loose garments by virtual bones, which can be regarded as extra anchors besides SMPL model. SCALE (Ma et al. 2021 adopts local elements to model registered garments based on minimum-clothed human. To handle obstacles with arbitrary topologies, N-Cloth (Li et al. 2022) predicts garments deformations given the states of initial garments and target obstacles. Other studies [Shen et al. (2020) Zhang et al. (022) generate 3D garments based on UV maps. SimulCap Yu et al. (2019) segments garments into upper and lower clothes as multiple separated meshes. SMPLicit Corona et al. (2021) generates garments by controlling the clothes' shapes and styles, but intersection-free reconstruction is not guaranteed.
In contrast, our data-driven method LayersNet animates garments by inferring garments' future posi- tions through the interactions between garment particles and other driving factors. Since the driving factors are also represented by particles, the garment animation thus equals to simulate particle- wise interactions. which is inaeeeindeeendent and is highly generalizable to unseen scenarios.
Physies Simulation by Neural Network. Learning-based methods for physics simulation can be applied to different kinds of representations, e.g., approaches for grid representation (Thuerey et al.,
2020
Wang et al. 2020), meshes (Nash et al., 2020 Qiao et al. 2020: WWnn et al. [022 Pfaff et al.
2021 and particles (Li ct al. | Ummenhofer et al. 2020;
et al. 2020 Shao
let al. 2022). Some methods include Graph Neural Network (GNN)-based methods (Li et al.
2019
Saachez-GGonzaler et al., 2020 Pfaff et al.) 2021). Transformer-basse methods (Shao et al, 2022)
adopt modified attention to recover interactions' semantics. Other work Liang al. (2019) designs algorithms to accelerate gradient computation for collision response as plug-ins for neural network.
Our LayersNet follows TIE in the notion of modeling particle-wise interactions, which is topology- independent and easy to generalize to unseen scenarios. In contrast to TIE, we exploit the redun- dancy of garments and establish a two-level hierarchy structure for them, where garments are made of deformable patches. Our method also differs in learning to predict the patches' dynamics by in- teracting with neighbor patches and other driving factors. We devise decoder to learn a topology- independent descriptor for each patch, enhancing the generalization abilities to unseen scenarios.
LAYERS DATASET
Most existing datasets are limited to single-layered garments driven only by human bodies. Different garments, such as the upper T-shirt and lower pants, rarely interact with each other. Consequently,
3

---

Under review as a conference paper at ICLR 2023
We compare LAYERS with existing 3D datasets. Our dataset is composed of multi-layered clothes, with unique attribute data, such as stiffness and friction, attached to each garment. Moreover. we include data of wind with its strength and direction randomly sampled. **: 3DPeople (Pumarola el al. 2019) does not specify the exact number of garments, while it claims to dress cach subject with different outfits. *-; The multi-layered garments in Layered-Garment Garment do not follow physics laws and the penetrated vertices are forced to move out of inner garments hard-coded manner.
Table I:
Dataset
Dynamics Subjects Garments Multi-layered Attributes Wind
3DPeople Pumarola et al.
2019)
80
N/A
TailorNet Patel et al. 2020)
9
20
N/A
Cloth3D (Bertiche
2020
8.5K
11.3K
Layered Garment (A Aggarwal ct al.
2022
142
101
.2
N/A
LAYERS (Ours)
4.9K
9.9K
9.9K
V
the problem can be easily solved by modeling garments as functions of human bodies and consider- ing only single-layered outfits predictions (Patel et al. 2020 Bertiche et al., 202T].
Generating dataset with multi-layered garments is non-trivial interpenetration between garments should be avoided, and their dynamics should obey the physics rules. Thanks to recent developments in thysics-based methods, several software, such as Blender can infer the interactions among different clothes and generate faithful garments with multiple layers.
The proposed LAYERS is built with Blender. It is the first dynamic multi-layered garments dataset that considers the wind factor beyond just human bodies. To construct the dataset, we first collect & Lee (021). which includes various types of
the garment templates from WwWPaateen(Kooostee garments, such as jackets with hood, and dresses with waist helts. Then, we generate multi-layered combinations with outer-layer and inner-layer clothes. Each combination of multi-layered garments is then draped to SMPL human body (Loper et al. (015)). This is followed by a warm-up simulation in Blender resolve interpenetrations. Finally, we simulate the dynamics of garments given the human motion sequences (Mahmood et al. 2019) and sampled winds. Specifically, after we drape the garments to a human body model, we scale up the human mesh and garment mesh ten times the real-world size before simulation. This strategy can reserve more
details in Blender.
We compare our dataset with existing datasets in Table T
Since our dataset includes the 3D meshes and attributes of garments, as well as the detailed scene settings for each sequence, LAYERS can be easily extended to other formats of data facilitate the explorations of alternative topics, such as optical flow estimations, 3D reconstructions from images, and physics parameters estimations. In the following, we detail the key settings LAYERS.
Multi-layered Garments. Each multi-layered outfit is composed of inner and outer outfit. In LAYERS, the outer outfit is either jacket or jacket with a hood, giving us a clear view of interac- tions from inside and outside. Inner outfits refer to whole-body outfits, such as dresses, jumpsuits, and t-shirts with pants or skirts. We generate 4,900 combinations of multi-layered garments, with 9,872 different garments in total. The garment templates are in high fidelity, with vertices ranging from 5,000 to more than 15,000 for each garment, enabling us to capture more details in simulation.
The main challenge is to have interpenetration-fis simulations for multiple objects. To achieve that, we first drape the multi-layered outfit to SMPL human body in T-pose, followed by warm-up simulation to solve the interpenetrations among garments. We adopt a large collision distance to ensure all the interpenetrations mesh and conduct a simulation driven by the human body and wind. Since all garments belong to one mesh after merging, the interactions among garments are computed through the self-collision mechanism in Blender, which generates interpenetration- free results in simulation.
resolved. Afterwards, we merge the garments into garment
Wind. Most existing datasets simplify real-life scenarios through driving the animation of gar- ments only by human bodies. To enrich the settings and enable researchers to further explore gar- ment animations driven by multiple factors, in LAYERS we introduce randomly sampled wind, a common and obvious force field to influence the animation. Specifically, we randomly select sev- cral spans of frames in a sequence, and apply winds of different directions and strengths as force fields. The directions and strengths are uniformly sampled (0 to 400 in Blender). Within each span,
https://www. /ww. blender.ooo orr//
4

---

Under review as a conference paper at ICLR 2023
UV mappings
==t
Eincoder
Simolaer
Docader
3
Paich Embeddings
T-i+1
T-1+1
Wind
T
Overview of LayersNet. Given driving factors at time t ++ 1, i.e. the human body model and environmental wind in our study, Layerseet aims animate target garments at time t and predicts the new states garments at time I. Human body model, wind, garments and corresponding attributes are represented by particles, where we adopt abstract particles to denote
Figure 2:
wind and garments' attrihutes. We further establish a two-level structural hierarchy for garments, as shown on the top left of the figure, where garments are made of patches, and patches composed of particles of fixed configuration given the UV mappings. Then encode the particles and model the interactions among them by simulator, which outputs the embeddings for each patch, Finally, we apply a decoder to decode the patches into corresponding vertices at time 4
make reasonable assumption that the wind would affect the whole 3D space, where wind's direction and strength remain constant.
Garments' Attributes. Existing datasets cover limited choice for garment attributes, e.g., cotton or fabric. Tasks like physics parameter estimations could barely benefit from those datasets. To make garments animations more diverse and flexible, we uniformly sample different garments' attributes, such as mass, stiffness, and friction. addition, we introduce different attributes to the inner and outer outfits, leading to more varieties.
Human Motion Sequences. We adopt the SMPL-based human motion sequences from CMU MoCap in AMASS (Mahmood et al. 2019), which includes 2,600 sequences with 30FPS in total. During simulation, we randomly sample the human shapes and genders for each sequence and ex- tract sub-sequences with maximum of 600 frames. To accurately simulate garments human body, collision-free human meshes are required to avoid invalid simulations. Since our dataset fo- cuses on garments generations, we adopt linear regressions models (Loper et al., (0155) and leave a minimum gap of 0.004 meters before scaling up the human mesh. We skip the unresolvable collisions and discard the corresponding frames.
solve the self-collisions from SMPL
METHODOLOOGY
Our goal is to faithfully animate the garments regardless of the garments' topology and the type of driving factors, where the latter include the rigid human bodies and winds in case. To this end, we propose LayersNet to animate garments a simulation manner. The novelty of LayersNet is that view garment animations driven by human bodies and winds interactions between particles. This unified perspective allows our framework to exploit the semantics of interactions among all particles, e.g., the energy transition when constrained by physics laws. In addition, the animation of garments becomes shape-independent and highly generalizable. Figure shows an overview of LayersNet. In the following, first formulate garment animation in the form of particle simulation, followed by an explanation of our patch- based garment model, and the introduction of the simulation pipeline to animate garments.
4.1 LAYERSNET
Problem Formulation. We denote each mesh time i by M (V', EM EEW }. where V (x), à N :/i=1 are the vertices' positions, velocities, and accelerations, and EM denote the mesh edges. EW are the world space edges (Pfaff et al. (021), where we dynamically connect node and node j if x 2 < R, excluding node pairs already exist in the mesh. In particle-base based system, each mesh is represented by particles, which are the corresponding vertices from the mesh. During simulation, particle and particle j will interact with each other iff edge eig E EM U E
5

---

Under review as a conference paper at ICLR 2023
connects them. The interactions guided by EM enable learning internal dynamics of mesh, while interactions indicated by EW serve to compute external dynamics such as collisions.
We adopt abstract particles to represent the garments' attributes and the wind. In particular, ag to denote each garment's attribute, such as the friction and stiffness, and w" to denote the wind. Since the wind has constant strength in the whole 3D space,
the quaternion rotation q' and the strength s to represent the wind as w' = = J. In this way, given the human body and wind i++1 as well as their previous l states, we aim predict the garments' states at time t+1 given the current states at / and corresponding previous meshes {M'-1 ... Mt-k). In practice, we choose === in all experiments. Our approach can be described as:
.
vi++1
a(ag (M-- M MM+i-i, w w 1i=0),
(1)
where My and M are the meshes of garments and human body, respectively, o(-) is the simulator and runs recursively during predictions, and Vet+1 the garment's new vertices' states at time t t 1.
g
Patch-based Garment Model. Inspired by Ma et (2021), we establish two-level structural hierarchy for garments and represent each garment by patches. Patch modeling holds several advan- tages. First, basic units to represent garments, patches are topology independent. By modeling the dynamics of each patch, our model is more flexible and generalizable unseen garments. Second, instead of simulating each vertex in a mesh, simulating patches signficantly reduces the computa- tional overhead, especially when the mesh is of high- high-fidelity.
Formally, we find a mapping q(-) to map the vertex- based mesh to patch-based representation by:
g = (MM)),
(2)
where P3 ( (Vi. """,""" The patches' states V are the averaged vertices' states within the patches, and y are computed given V The mapping g(-) is based on the garments' uv maps as shown in Figure 2 In this way, method can be updated as:
-1+ 4++1 = o(a AP-ii,M M,,,,,++,, w 111-111 (
(3)
Simulation- Garment Animation. After collecting the set of particles, including the patch particles of garments, the vertex particles of human body, well the abstract particles of garment attributes and environmental wind, we can then animate garments by predicting the future state of each particles through particle-based simulation, which is topology independent and highly general- izable. While our method is orthogonal to the choice of particle simulator, in practice we adopt TIE (Shao et al.) 2022) as our simulator due to its promising results and high computational efficiency. Specifically, it assigns each particle i three tokens, namely state token a sender token 8; and receiver token r;- The sender token 8, describes how the particle influence others, and the receiver token r; indicates how the particle i can be affected. The updating formulas of all tokens can be summarized as follow:
8, = W.Us,
Ti = W,v, r/ r: Hr:
(4)
8j -Mx,
8/
(5)
Oria;
Tr,35
fr. fr..s r/ + 8/j:
(6)
v. = Σ Ewiffruies
(7)
Wij = softmax((Wqu:) ((.
(8)
where Ms, are the means of tokens r;, 8j respectively, and orey is the standard deviation. In practice, we generate two different attention masks indicated by EM and EW for different heads in multi-head attention.
Formally, the simulator can be described as
E
S(ay; (P!-1, Mo+1-s, - )=

---

Under review as a conference paper at ICLR 2023
To recover each vertex's details within the corresponding patch, apply decoder
a s(ve, c+1 w/(l)))
pp.
(10)
0 At att v p At At pp
(11)
where we concatenate i-th vertex's state of with its corresponding patch embedding epp' and the states of the nearest point on human mesh hi, time f + 1. We then calculate the corresponding position and velocity at time t + 1 given the time interval At between each frame.
++1 imputs, and output the new acceleration a at
4.2 TRAINING DETAILS
To train simulation model, we first apply a standard mean square enror (MSE) loss the positions of vertices as:
1 N
C
P
(12)
where p1+1 is the ground truth at time + and N the number of vertices. We adopt a loss term for the vertex normal maintain the smoothness and consistence of the garments:
c++
Σ N
n 1+1 - n
n 11
(13)
where n and na reduce the collision rates between garments and human bodies, we adopt a collision loss:
+++ the vertex normal for prediction and ground truth respectively. To further
Ct
N. Σ (d. min (1p)+) Ph )) -nn
++1 ,d.))²
(14)
where Ph is the nearest point P p-- on the human mesh, ni 1 is the normal vector of point Ph No the number collided vertices, and de is the minimum distance of penetration. Thus, for predictions at time / 1, our training loss is written as:
c++
Am Am + Xn Cn Lt+1
(15)
During training, we predict the garments' positions for two future timestamps, namely + and t t+2 2. The final loss L is
C c++ + 2
(16)
EXPERIMENTS
5.1 BASELINE AND IMPLEMENTATION DETAILS
baselines. DeePSD achieves -tate-of-the-art performance in terms of 3D garment animations. Importantly, it claims to support the animations of multi-layered garments. We mainly compare DeePSD with our model and make the following extensions to DeePSD: 1. we add wind as extra inputs; 2. we add the collision loss between different layers of garments for multi-layered clothes settings. Since MGNet is a garment-specific model for single-layered clothes, we compare MGNet with only inner garments. We also include the wind as extra feature map. All models are trained with ten epochs. We do not apply any postprocessing for both training and predicting. During evaluation, we calculate emrors as the mean of Euclidean errors for each frame, then average the errors of all frames within each sequence. The final results are the mean of errors from all sequences.
We implement DeePSD (ertiche et al. 202 and MGNet (Zhang et al. 2022)
5.2 ABLATION STUDY ON LAYERS
To investigate the influence of mulli-layerra garments and random wind in LAYERS, we divide dataset into four different splits shown in Table 2 tight inner garments without wind (T);

---

Under review as a conference paper at ICLR 2023
analyze the challenges in LAYERS, we sample four splits from our dataset: inner garments are tight clothes without wind (T); inner garments are tight clothes with strong wind (T+W); inner garments are loose clothes without wind (L); inner garments are loose clothes with strong wind (L+W), Specifically, jackets are either with without hood, while dress are either with without waist belt. Notice that group winds with a strength less than 50 as not windy, where the wind has little influence on the garments.
Table 2:
Components
Tight (T) Tight+ Wind (T+W) Loose (L) Loose+ Wind (L+W)
Inner Garments
Jumpsuit
Jumpsuit
Dress
Dress
Outer Garments Jacket
Jacket
Jacket
Jacket
Wind Strength
50
> 250
S 50
250
Table 3: Euclidean errors (mm) on four splits. To display the challenges brought by the outer garments and the interactions between layers of clothes, we further train models with only the inner garments as marked by in the table. MGNet has worse generalization abilitics due to garment-speeifie design. LayersNet has slightly higher emrors since the inner garments simulated logether with outer clothes do not follow physics laws by themselves. LayersNet achieves superior and robust performance on all splits with multi-layered garments.
Methods
Tight (T)
Tight Wind (T+W)
Loose (L)
Loose + Wind (L+W)
DeePSD*
225.3++00.4
139.5==03.9
501.3=300.1
577.5+#37399
MGNet"
5219.2+1565.8
1156.8=7754.8
4432.7=1438.
4595.0=#1215.2
LayersNet'
260.3::254.4
278.61:328.6
378.0-293.0
363.6:311.4
DeePSD
1068.2:::93.8
2782.6=1239.77
868.0=495.6
1707.9::503.4
LayersNet(Ours)
611.3:1:54.3
578.8
603.0=529.1
472.5:t:69.5
tight inner garments with strong wind (T+W); loose inner garments without wind (L); loose inner garments with strong wind (L+W). Each split contains 36K frames for training, 2K frames for validation, and 2K frames for test. Note that the strength of the wind ranges from 400. We group winds with a strength less than 50 as not windy, where the wind has little influence on the garments. We consider winds with a strength more than 250 as strong wind. Since the outer garments exhibit more flexible dynamics, such as falling off or waving in the air, animating them is already a challenging task for existing methods, let alone considering the interactions with inner garments. To further simplify our dataset and have better comparison with existing synthetic dataset, Cloth3D (Bertiche et al. 2020), we first exclude the outer garments on all splits and train models with only inner garments, which is indicated in the first three rows of Table
When trained with only inner garments, DeePSD achieves reasonable performance comparing with that when it is trained on Cloth3I) (Bertiche et al. 2020), suggesting that the settings of Cloth3I) are similar to our simplified data settings. MGNet fails in our dataset due to the garment-specific design and low generalization abilities. LayersNet has lower errors especially on split I. and L+W, suggesting the effectiveness and higher generalization abilities of animating loose clothes, Please refer to Appendix for more details and qualitative comparisons. On split T+W and L+W, DeePSD shows higher errors due to the random wind. Since jumpsuits in splits T and T+W are tight garments, the wind has less influence on them.
When trained with multi-layered garments. the Euclidean errors by DeePSD increase dramatically, especially with the influence of wind in split T+W and L+W, suggesting the challenges brought by multi-layered garments. The high errors on split T+W and L+W, compared with split T and L., respectively, suggest that DeePSD is less generalizable to driving factors beyond human bodies. In contrast, LayersNet achieves superior performance on all splits with both inner and outer garments. The Euclidean errors are close to each other on different splits, suggesting that our model is more robust the garments' various topologies well the driving factors beyond human bodies.
5.3 GARMENT ANIMATION
We sample 50K frames from LAYERS for training, 6K frames for validation, and 6K frames for test. There is no overlapping among different sets of samples. All the samples are composed of both inner and outer garments, as well as random wind as the external factor.
The vanilla DeePSD without collision loss exhibits high Euclidean errors on all types of garments. When adding collision loss including collisions between layers of garments, the collision rates are reduced. Nonetheless, DeePSD does not improve in terms of Euclidean errors. Since the vanilla DeePSD has difficulties in learning reasonable dynamics of garments, the collision terms introduce more noise while training DeePSD, pushing the garments away from the bodies. In contrast, Lay- ersNet achieves superior performance in terms of Euclidean errors, suggesting the effectiveness of our simulation-basee methods. Moreover, our LayersNet is more generalizable and shows more
8

---

Under review as a conference paper at ICLR 2023
Table Euclidean error (mm) on sampled LAYERS with maximum sequenee length of frames. The colli- sion rates between different layers of garments are shown under L-Collision, while the collision rates between garments and human bodies are shown under H-Collision. Models trained with collision loss marked by +. When training DeePSD with collision loss, extend Equation and include the collisions between different layers garments as extra loss term. In constrast, our model only applies basic collision loss between garments and human bodies following Equation| [4] Our LayersNet achieves superior results on all types of garments. Even without explicitly punishing collisions between layers of garments, LayersNet still achieves low collision rates among garments and makes good balance between Euclidean emrors and penetrations.
Methods
Jacket
Jacket + Hood
Dress
Jumpsuit
Skirt
DeePSD DeePSD+
2863.0+8811.
2956.7=799.88
2606.3++792.0
2876.2++00..3
2498.6-618.6
1607.7+1147.0
3434.41781.9
4046.6.1962.9
4773.1
4232.6:1713.9
LayersNee(Ours)
717.6:1609.8
577.5:155..1
448.2
277.3=293.11
274.6.14,44
(ayersNet+(OUrs))
684.915554.9
566.2:1425,4
501.211466.9
321.1
378.61143.0
Methods
Pants
T-shirt
Overall
L-Collision
H--Collisin
DeePSD
107..1
2618.5-1729.1
2851.81699
23.82%:111.2 11.25%
18.36%.16.777 6.7%%
DeePSD+
1260.411183..4
3144.9-994.3
3907.6:1790.
3.82% 13.72%
0.63%
Layers)Ne(Ours)
291.2:1:01.7
272.4=198.2
560.61155..2
4.51%=2.98%
10.01%
LayersNet+(Ours)
349.3+233..4
331.7=226.9
567.2
4.94%=2.67%
3.88%
Time
Time
Lazaneheei
LayanaNee
Layershees
LayanaMece
Figure 3: Qualitative results by LayersNet. The left sequence shows a human model walking down some steps. The magnified regions highlight the vivid dynamics of the jacket. Human model in the right sequence is moving towards her left. Without collision loss, LayersNet generates some body-to-cloth penetrations near the thigh and even head on the last frame of the right sequence. When trained with collision loss, LAYERS, which marked by +. reduces the collision rates more obviously. Even though we do not explicitly penalize the collisions among different layers of garments, LayersNet is able to solve those collisions implicitly through exchanging semantics by interactions, suggesting the robustness and effectiveness of LayersNet.
robust performance across different types of garments. When adding collision loss as mentioned in Equation 14 the body-to-cloth collisions and human bodies are reduced. Though never plicitly penalize collisions between different layers garments, the collision rates among clothes are low. Since the key idea of simulation is model the interactions among objects, such as the energy transition and collisions, LayersNet can resolve collisions implicitly. We show the qualita- tive results by our LayersNet in Figure 3 More qualitative comparisons can be found in Appendix. LayersNet with collision loss, which is marked by +, shows fewer body-to- body-to-garment 1o-garment penetrations, suggesting the effectiveness and robustness of our model.
CONCLUSION
We have presented a new large-scale synthetic dataset called LAYERS, which covers 4,900 different combinations of multi-layered garments with 700K frames in total. The animations of multi-layered garments follow the laws of physies, allowing the interactions among different layers of garments. In addition, LAYERS takes the environmental wind, another important driving factors besides human body, into consideration to animate garments. To demonstrate the quality of LAYERS, we further propose LayersNet, a simulation-based method for garment animations. We model the various driv- ing factors as (abstract) particles, while represent garments as unions of particles as patches. The animations of garments driven by different factors are naturally and uniformly achieved via model- ing the interactions among particles. As shown by the experiments, our model achieves superior and robust performance with compelling abilities in generalization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alakh Aggarwal, Jikai Wang, Steven Hogue. Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. Layered-garment net: Generating multiple implicit garment layers from single image. In Com- puter Vision ACCV 2022 16th Asian Conference on Computer Vision, Macau, China, December December 8, 2022, 2022.
Hugo Bertiche, Meysam Madadi, and Sergio Escalera. CLOTH3D: clothed 3d humans. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XX, 2020.
Hugo Bertiche, Meysam Madadi, Emilio Tylson, and Sergio Escalera. Deepsd: Automatic deep skinning and pose space deformation for 3d garment animation. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garment net: Learning dress 3d people from images. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, 2019.
Zhongang Cai, Daxuan Ren, Ailing Zeng. Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang. and Ziwei Liu, Humman: Multi-modal 4d human dataset for versatile sensing and modeling. CORR, 2022.
Enric Corona, Albert Pumarola, Guillem Alenyå, Gerard Pons-Moll, and Francesc Moreno-Noguer. Smplicit: Topology-aware generative model for clothed people. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual, June 19-25, 2021, 2021.
Maria Korosteleva and Sung-Hes Lee. Generating datasets of 3d garments with sewing patterns. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Jie Li, Gilles Daviet, Rahul Narain, Florence Pertails-Descouteee Matthew Overby, George E. Brown, and Laurence Boissieux. An implicit frictional contact solver for adaptive cloth simu- lation. ACM Trans. Graph., 2018.
Y. D. Li, M. Tang, Y. Yang, Z. Huang, R. F. Tong, S. C. Yang, Y. Li, and Dinesh Manocha. N-cloth: Predicting 3d cloth deformation with mesh-based networks. Comput. Graph. Forum, 2022.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Junbang Liang, Ming C. Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse prob- lems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model. ACM Trans. Graph. 2015.
Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to dress 3d people in generative clothing. In 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J. Black. SCALE: modeling clothed humans with a surface codec of articulated local elements. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: archive of motion capture as surface shapes. In 2019 IEEE/CVF International Confer- ence on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, 2019.
10

---

Under review as a conference paper at ICLR 2023
Rahul Narain, Armin Samii, and James F. O'Brien. Adaptive anisotropic remeshing for cloth simu- lation. ACM Trans. Graph., 2012.
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. PolyGen: An autoregressive generative model of 3d meshes. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, 2020.
Xiaoyu Pan, Jiaming Mai, Xinwei Jiang, Dongxue Tang, Jingxiang Li, Tianjia Shao, Kun Zhou, Xiaogang Jin, and Dinesh Manocha. Predicting loose-fitting fitting garment deformations using bone- driven motion networks. In SIGGRAPH '22 Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 11, 2022, 2022.
Chaitanya Patel. Zhouyingcheng Liao, and Gerard Moll.Moll. Tailomet:
Predicting clothing in 3d as function of human pose, shape and garment style. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzallee and Peter W. Battaglia. Learning mesh- based simulation with graph networks. In 9th International Conference on Learning Representa- fions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Albert Pumarola, Jordi Sanchez, Gary P. T. Choi, Alberto Sanfeliu, and Francese Moreno. 3dpeople: Modeling the geometry of dressed humans. In 20/9 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, 2019.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Scalable differentiable physics for learning and control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, 2020.
Alvaro Sanchez-Gonzale Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, 2020.
Igor Santesteban, Nils Thuerey, Miguel A. Otaduy, and Dan Casas. Self-supervised collision han- dling via generative 3d garment models for virtual try-on. In IEEE Conference Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-base physics simulation. CORR, 2022.
Yu Shen, Junbang Liang, and Ming C. Lin. Gan-based garment generation using sewing pattern images. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII, 2020.
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 2020.
Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. SIZER: dataset and model for parsing 3d clothing and learning size sensitive 3d clothing. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 111, 2020.
Lokender Tiwari and Brojeshwar Bhowmick. Deepdraper: Fast and accurate 3d garment draping a 3d human body. In IEEF/CVF International Conference Computer Vision Workshops. ICCVW 2021, Montreal, BC, Canada, October 1 1-17 2021, 2021.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simula- tion with continuous convolutions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Raquel Vidaurre, Igor Santesteban, Elena Garces, and Dan Casas. Fully convolutional graph neural networks for parametric virtual try-on. Comput. Graph. Forum, 2020.
11

---

Under review as a conference paper at ICLR 2023
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In KDD 20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27,
2020, 2020.
Tuanfeng Y. Wang. Tianjia Shao, Kai Fu, and Niloy J. Mitra. Learning an intrinsic garment space for interactive authoring of garment animation. ACM Trans. Graph., 2019.
Zehang Weng, Fabian Paus, Anastasiia Varava, Hang Yin, Tamim Asfour, and Danica Kragic. Graph-based task-specific prediction models for interactions between deformable and rigid ob- jects. CoRR, 2021.
Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, and Yebin Liu. Simulcap Single-view human performance capture with cloth simulation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. Detailed, accurate, human shape estimation from elothed 3d scan sequences. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017.
Meng Zhang, Duygu Ceylan, and Niloy J. Mitra. Motion guided deep dynamic 3d garments. CoRR, 2022.
Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human recon- struction from single image. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, 2019.
A APPENDIX
A.I LAYERS DATASET
Although Blender does not support collisions among multiple objects, it able to solve collisions within one object. Thus, by merging multiple garments single mesh, we regard the collisions among different garments as the interactions within one mesh, which can be solved by Blender. In other words, different layers of garments will interact with each other following the physics rules.
Before simulation, we properly dress the human body in T-pose and scale up all objects 10 times the real-world size, which can preserve more details of the garments such wrinkles.
We adopt different garment attributes to different layers of clothes. Specifically, we uniformly sam- ple the following attributes: vertex mass from 0.2 to 0.8; stiffness of tension, compression, shear, and bending from 15 to 100; friction from 40 to 80.
Quantitiave comparison with existing synthetic dataset. We train our implementation of DeePSD on Cloth3D following the original paper (Bertiche et al. 202I). Results are shown in Table 5 suggesting that our implementation of DeePSD is similar to the official one. We sample four splits from LAYERS as mentioned in main text. Specifically, the inner garments in split T and T+W are jumpsuits, while those in split L and L+W are dress. The outer garments in all splits are either jackets or jackets with hood. For convenience, we copy the table in main text for reference as shown in Tan Table 7 Table the results trained on four splits of our dataset. Specif- ically, the results on the first three rows, where models are marked by *, are obtained by training models with only inner garments. This setting most similar to Cloth3D's settings. The remaining results are obtained by training on both inner and outer garments on all splits. Notice that we scale up the human mesh and garment mesh 10 times the real-world size, the corresponding errors are also scaled up. Thus, DeePSD* achieves similar results on both datasets: the Euclidean errors of jumpsuit and dress are similar on both LAYERS and Cloth3D, suggesting that the quality of our dataset is not worse than Cloth3D. In addition, when comparing DeePSD* with DeePSD, DeePSD achieves higher errors, which mainly come from the outer garments. MGNet fails in LAYERS due to the garment-specins sNet achieves reasonable results especially on loose garment settings, suggesting the effectiveness and higher generalization abilities of animating loose garments.
design and low generalization abilities shown in Figure 4 Our Layer-
12

---

Under review as a conference paper at ICLR 2023
We verify our implementation DeePSD on Cloth3D accroding to official paper (Bertiche et al. 2021). The results are similar to original paper, suggesting that our implementation of DeePSD is similar to official one.
Table 5;
Method
T-shirt
Top
Trousers
Skirt
Jumpsuit
Dress
DeePSD 25.01=20.94
16.90 15.38 20.02:8..50 20.43=31.10 24.31=6.36 42.10+21.41
Table analyze the challenges LAYERS, we sample four splits our dataset: inner garments are tight clothes without wind (T); inner garments are tight clothes with strong wind (T+W); inner garments are loose clothes without wind (L): inner garments are loose clothes with strong wind (L+W). Specifically, jackets either with without hood. while dress are either with without waist belt. Notice that group winds with strength less than 50 as not windy, where the wind has little influence the garments.
Components
Tight (T) Tight+Wind (T+W) Loose (I.) Loose+ Wind (L+W)
Inner Garments
Jumpsuit Jacket
Jumpsuit
Dress
Dress
Outer Garments
Jacket > 250
Jacket S 50
Jacket
Wind Strength
S 50
> 250
Table Euclidean errors (mm) on four splits. To display challenges brought by the outer garments and the interactions between layers clothes, we further train models with only the inner garments as marked by in the table. MGNet has worse generalization abilities due to garment-specific design. LayersNet has slightly higher errors since the inner garments simulated together with outer clothes do not follow physies laws by themselves. LayersNet achieves superior and robust performance all splits with both inner and outer garments.
Methods
Tight (T)
Tight Wind (T+W)
Loose (L.)
Loose + Wind (L+W)
DecPSD*
225.3++
1.9.5=+003.9
501.3=300.1
577.5+337399
MGNet"
5219.2+
1186.8=1754.8
1432.7=1438..
4595.0=12155.2
LayersNet'
260.3::254.4
278.6::28.6
778.0-293.0
363.6+311.4
DecPSD
1068.2-:693.8
2782.6+12399
868 868.0=995.6
1707.9+503.4
LayersNet((Orrs
614.3
578.8
603.0=529.1
472.5
MGNei
DeePSD
The left are training samples while right is test samples. All samples are from split T, where we train models with only inner garments (jumpsuit). MGNet is able to generate 3D garments on training examples on the left while has difficulties to generalize unseen examples in test set due to the carment-speciffe design. DeePSD has faithful predictions simplified dataset. Our LayersNet faithfully rollouts 3D garments when only with inner clothes.
Figure 4:
Qualititative comparisons. We include more qualitative comparisons in this section. As shown in Figure 4 when trained with only inner garments (jumpsuit), DeePSD (Bertiche al. 2021 is
13

---

and Under review as a conference paper at ICLR 2023
GT
DeePSD
DeePSD+
LayersNat+
Time
The outer garments more flexible in dataset and are able to respond to different garment attributes, such friction. In this sample, the jacket falls off the shoulder due to the motion of human and small friction, bringing more challenge for DeePSD to converge. The collision loss for DeePSD further generates noises and forces DeePSD to push clothes away from human model to achieve lower collision rates. In contrast, our LayersNet is able to faithfully rollout the garments even on this challenging case.
Figure 5:
Table 8:
We train LayersNet on our LAYERS test on Cloth3D, also compare DeePSD which trained on Cloth3D. We re-sample the test samples from Cloth3D and include continuous sequences for better comparisons. The test set and training set on Cloth3D have no overlaps. Our LayersNet is able to achieve superior results all scenarios.
Method
T-shirt
Top
Trousers
DeePSD (trained Cloth3D)
36.69-11.33 26.52.19.15 29.86.113.47
LayersNet (trained LAYERS)
23.84=15.81 14.17=5.92 13.16
Method
Skirt
Jumpsuit
Dress
DeePSD (trained on Cloth3D) LayersNet (trained LAYERS) 41.48:117.73 -9.91 35.51
55.66=22.11 26.86=6.65 54.71
able to predict faithful rollouts. MGNet Zhang et al. 2022 is able to generate 3D garments on training samples while struggles to generalize unseen garments from test examples due to the garment-specific design. In original paper of MGNet, they train MGNet with only on 300 frames of data with the same garment topology, while in our dataset each garment is unique with different topology. In contrast, our LayersNet achieves faithful predictions in this simplified case.
On the other hand, shown in Figure the garments in dataset, especially the outer clothes, more flexible and are able to respond to various garment attributes, such as falling off the shoulder due to small frictions in this case. The high flexibility brings more challenges to DeePSD, leading to difficulties in convergence. The collision loss for DeePSD further introduces noises due to inaccurate garment meshes and forces DeePSD to push the clothes away from human to achieve lower collision rates. In contrast, our simulation based LayersNet animates garments in unified manners and still achieves faithful rollouts even on such challenging case.
A.2 MODEL GENERALIZATION
Since our LayersNet learns a topology-independeer simulation model and is highly generalizable unseen scenarios, test LayersNet, which is trained on dataset LAYERS, Cloth3ID
14

---

Under review as a conference paper at ICLR 2023
DoePSD
LayersNet
GT
DeaPSD
Time
Figure 6: Samples are from Cloth3D Bertiche et al. 22020 The human model on the left wears jumpsuit and is walking, while the human on the right wears dress and is spinning. Our LayersNet achieves faithful rollouts. DeePSD is able animate the garments with stiff dynamics, while LayersNet can predict garments following physics laws, as the inertia shown dress.
We grouped garment's vertices into patches given the corresponding UV mapping. Based the coordinates in UV, group the vertices according their positions. Subsequently, we divide the 3D garment mesh patches based groups.
Figure 7:
(erttche 2020 for generalization and compare the original DeePSD trained Cloth3D. We re-sample the test set to include continuous sequence of samples. The test set and training set for training DeePSD on Cloth3D have no overlaps. As shown in Table g our LayersNet achieves superior performance on all types of garments. As shown in Figure 6 when test on unseen samples, predictions by DeePSD tend be stiff while LayersNet still achieves faithful and vivid rollouts, suggesting the effectiveness of our simulation based method which animates garments in unified manner.
A.3 IMPLEMENTATION DETAILS
Patched Garment Model. We group the particles garment's mesh given corresponding UV mapping. Specifically, divide the UV mapping into square patches according to the UV coordi- nates, as shown in Figure 7 Then, we group the garment's vertices in 3D space given the grouped UV mapping. When building the connections EM of the patched garments, connect patch and patch j iff there at least one pair of vertices within the patches are connected in 3D space.
LayersNet details. To obtain the world space edges E we adopt R = 0.4 to calculate the neighbors from the human mesh and R 0.6 for different layers of garments. We adopt h 1 for
15

---

Under review as a conference paper at ICLR 2023
Table 9: Euclidean error (mm) on sampled LAYERS with maximum sequence length 35 frames. The collision rates between different layers of garments are shown under L-Collision, while the collision rates between garments and human bodies are shown under H-Collision. Models trained with collision loss are marked by +. When training DeePSD with collision loss, extend Equation [4 and include the collisions between different layers of garments loss between garments and human bodies following Equation [4 Our LayersNet achieves superior results on all types of garments. Even without explicitly punishing collisions between layers of garments, Ney still achieves low collision rates among garments.
loss term. constrast, our model only applies the basic collision
Methods
Jacket
Jacket + Hood
Dress
Jumpsuit
Skirt
LayersNet
556.3-
588.2-t-46055
436.3+:343.7
289.3+194.4
322.9+-58.9 274.6++94.4
LayersNett&on
717.6+6009
577.5++458.1 1 566.2+44554
448.2++452.2
277.3+293.1
LayersNett (fulll)) 684.9+5554.9
501.2+466.9
321.1+274.7
378.6=143.0
Methods
Pants
T-shirt
Overall
L-Collision
H-Collision
LayersNet
007.2+1644
282.9+
558.9+409.0
5.31%
16.41 % 18%
LayersNett&.n
291.2 349.3+238.4
272,4+198.2 2 331.7++22.99
4.51% 567.2+432.8
560.6=445..2
10.01 5.66%
LayersNet+Cn,, Cffful)
4.94%++2.67%
3.58%
the inputs of all objects' states. The hyperparameters Am, An, de in our loss term are set to 1. We adopt Adam optimizer with an initial learning rate of 0.001 and decreasing factor of 0.5 every two epochs. The batch size is set to 4.
We adopt three different encoders for meshes, garment attributes, and wind attributes. This is be- cause the three components belong to different domain space and have different dimensions. Since the states of garments mesh and human bodies mesh are from the same domain, we share the en- coder for them. All the encoders are two-layer MLPs with dimensions 128. The only difference is the input dimensions. We adopt 4 blocks modified Transformer block in LayersNet, with hidden dimensions 128 for each block. The number of head in multi-heas attention is set to 8, while 4 heads apply the attention mask generated by EM, and 4 heads adopt the attention mask generated by EW For the decoder, we adopt three-layer MLPs with forward dimension of 128 and an out- put dimension of 3. When concatenating the nearest point on human mesh in Equation TO we mask the point + zeros if the there is no edge ch,i EW connecting the point + and garment point g+ For the inputs of garment mesh and human body mesh, we adopt relative positions to the root of human body mesh. Since the wind's attributes are still measured in global coordinates, we also convert them to the relative coordinates in implicit manner, i.e. convert the value of strength in global coordinates to local coordinates defined by the root of human body. Specifically, we con- catenate the position, velocity, and acceleration of the human body's ' root point of as extra features w = {q', s², of and let LayersNet learns the wind's features in relative coordinates. We apply A = in our experiments, which is independent from the real time interval between each frames, which is 0.33s. When training LayersNet, we normalize the meshes' states across the whole training set before feeding into the model, which is a commonly adopted processing in literature.
A.4 ABLATION STUDY ON LAYERSNET
In this section, we analyze our model in the following aspects: (a). vanilla LayersNet; (b). Layer- sNet with normal loss; (c). LayersNet with both normal loss and collision loss.
As shown in Table 9 the vanilla LayersNet achieves low Euclidean errors on all types of garments, suggesting the effectiveness of our simulation-based method. With the normal loss term L.. which aims to smooth the surface of garments, achieves lower collision rates in terms of both body-to-cloth penetrations and collisions between different layers of garments, leading to more faithful rollouts comparing with vanilla LayersNet. When training with both normal loss and collision loss, Layer- sNet further reduce the body-to-cloth penetrations. Though our complete version of LayersNet has slightly higher Euclidean errors, the lower collision rates lead to more convincing and faithful re- sults. A good example is shown in Figure of the main text.
A.5 FUTURE WORK
In this work, we propose a novel multi-layered 3D dataset and make the first attempt to animate garments though simulation pipeline which achieve superior performance in both LAYERS and other generalization scenarios. Since we regard all objects as particles, our model is able to animate various garments with different typologies driven by different types of outer forces, such human
16

---

Under review as a conference paper at ICLR 2023
bodies and wind. However, since the predictions of future frames are based previous rollouts by the model, the errors accumulate as the length of predictions increases, which is a common problem in simulation. In this work, train the model and alleviate the problem by predicting two continuous frames as shown in Equation T6 We will explore more strategies, such as adding noise input data and forcing the model learn correct errors, enable the model to rollout long sequences while keeping the accumulated errors relatively small at the same time.
17


Reviews:
[{"review_id": "M-K9k3Mrpr7", "reviewer": "Reviewer_X85t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new dataset of garment motions under different human movements as well as wind forces. It also introduces a new network that can predict garment motions of different layers at the same time.", "summary_of_the_review": "Although the dataset is a good contribution, the proposed method has some clarity issues that prevent this paper from meeting the bar. I would be happy to change my score as long as the issues above are addressed.", "strength_and_weaknesses": "Strengths:\n1. Multi-layered cloth motion dataset is indeed in great need. This dataset can boost the advances of learning-based garment estimation and animation research.\n2. Results show that LayersNet has a better performance than previous work in terms of predicting multi-layer garment motion.\n\nWeaknesses:\n1. Missing references:\n - Physics Simulation by Neural Network: Liang, Junbang, Ming Lin, and Vladlen Koltun. \"Differentiable cloth simulation for inverse problems.\" Advances in Neural Information Processing Systems 32 (2019).\n - Data-driven Cloth Model: Shen, Yu, Junbang Liang, and Ming C. Lin. \"Gan-based garment generation using sewing pattern images.\" European Conference on Computer Vision. Springer, Cham, 2020.\n2. In Table 3, is there any reason why the LayersNet is not trained and tested with inner garments only as well? Otherwise, I think DeePSD for inner garments only is not needed either.\n3. Does the wind velocity affect the accuracy of the model? How well does the model perform with different wind speeds? In the simulation, very minor differences in the initial setting can cause significantly different end results (i.e., known as instability of the simulation). Does that happen as well for the model? If so, is there any particular way to overcome this so that the prediction errors can make more sense? Please add some discussions regarding this topic.\n4. In Table 4, the proposed method does not perform as well as previous works in regards to L-Collision and H-Collision. The authors should analyze and explain the phenomena but I didn't see any in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Iuoy0Pzk5OX", "reviewer": "Reviewer_9oGC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "MULTI-LAYERED 3D GARMENTS ANIMATION\n\nThis paper presents a novel deep learning-based 3D garments animation method. The method learns to model interactions between 1) cloth-body, 2) multiple cloth layers 3) wind and clothes. Specifically, the algorithm decomposes the clothes into connected patches and evaluates the deformation based on a learning-based particle physics simulator (Shao et al., 2022). A new simulation dataset has been proposed to study this new setting. \n\n", "summary_of_the_review": "Overall I am on the fence. I would like to see how the author addresses my aforementioned concerns and questions. \n\n\n----------------------------------------------\nPost-rebuttal. \n\nThe authors addressed most concerns by providing additional experiments and revised the submission accordingly. \n\nOverall, it is a weak-accept to borderline-ish submission. \n\nThe paper presentation is quite clear, and the technical approach is straightforward, as pointed out by both the authors and reviewers -- fusing particle simulation in TIE (Shao et al.) with patch-based representation in SCALE (Ma et al.) for efficiency. The method achieves the SOTA in runtime and speed and can handle external force and unseen clothes types.  \n\nIt is a well-executed and well-presented paper after the revision. The proposed combination is somewhat new and, indeed, solves practical challenges of prior arts (efficiency, external force, and multi-layer interaction). It's not a ground-breaking or surprising idea, but it sets a new state for the field and brings new capacity. Hence it doesn't harm to get accepted for its practicality. ", "strength_and_weaknesses": "Pros: \n- The new capacity for learning-based clothes simulation to handle both wind-cloth, body-cloth, and inter-cloth interactions. \n- Superior performance compared against DeePSD through a thoughtful exp design. \n- The paper is clearly presented. Equations/tables/figures are all easy to parse. The intro is clear. Related covers most of the relevant works.\n- New large-scale dataset with realistic simulation and unique designs (e.g., wind). \n\nCons:\n- The problem setting of data-driven animation here is not well-motivated. \n- The comparison study is somewhat limited. \n- Lack of qualitative demonstrations and comparisons. \n\nMixed:\n- Technical contribution. The paper belongs to a novel application / resemble of existing techniques (1. data-driven particle-based simulation from Shao et al. and 2. patch-based garments representation used in Ma et al. and earlier works. e.g., Kavan et al. Siggraph 11, Feng et al. 2010)\n- The above two core contributions the author claimed to differentiate the proposed work from prior works are not entirely original. That said, I still appreciate the novel ensemble and applications to data-driven simulation under the wind. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3WwmYptsj10", "reviewer": "Reviewer_g4UX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors address the task of simulating multi-layered 3D garments on a human body. Authors fist curate a dataset of multi-layered garments under various poses and wind effects and use this data to train LayersNet that predicts garment deformations as a function of body shape and previous garment state. ", "summary_of_the_review": "I like the task and formulation that authors propose in this work. It is not clear to me how is the existing garment modelling different than the particle based modelling in TIE, Shao 2020. Moreover, parts of the formulation seem to be tailored for modelling single layered garments and not multiple layers. eg: intersection is handled only with the body and garment and not garment-garment.\n\nCan the authors also provide video results from their method. This is important as the authors deal with garment simulation over time and we should see the quality of the simulated data (the proposed dataset) as well as the results from LayersNet. With provided results it is difficult to access whether the clothing deforms smoothly or not.", "strength_and_weaknesses": "Strengths:\n+ The task of modelling multi-layered garments is quite challenging and not a lot of works address this.\n+ Typical works on garment modelling on human body do not consider external factors such as wind, which is interesting in this work.\n+ Authors chose particle based physics modelling for the garments which is topology independent and allows modelling external factors like wind as wind can be treated as a set of particles.\n+ The authors will release the dataset and code.\n\nWeaknesses:\n- Technical novelty seems limited, as the garment modelling is handled by TIE, Shao et al. 2020. Can the author clarify the differences between the approaches.\n\n- In eq. 3 the garment is modelled as a function of body and previous state of the garment. Since the paper primarily deals with multi-layered garments, should't the garment also be a function of other garments on the body?\neg: let's say a person is wearing a loose skirt and a long jacket on top, such that the bottom of the jacket draper around the top of the skirt. Isn't the jacket deformation more dependent on the skirt than the body?\n\n- Eq. 14 only handles collisions between the body and the garments, but since authors deal with multi-layered garments, what about the collisions between different layers of garments?\n\n- Related work on multi-layered garment is completely missing. This important as multi-layered garments are the main contribution of the work. Authors should add a paragraph about it as currently it appears that the authors are the first one to address multi-layered garments which is not true.\n--> SMPLicit, CVPR'21 (not cited) has shown generation of multi-layered garments.\n--> SimulCap, CVPR'19 (not cited) Also use multi-layered garments + physics based simulation for motion capture.\n--> Another concurrent work: Layered-Garment Net, ACCV'22 (not cited) also generates multi-layered clothing using simulated data.\n\nClarifications:\n- Eq. 12: Shouldn't this be a double summation. One over multi-layered garments and other over vertices of the garment.\n- What is f(.) in the line just above eq. 3?\n- There are existing works like TailorNet, Patel et al. 2021 that also use PBS to drape garments on human body. Can the authors also provide details on how simulating multiple layers is more challenging than simulating single layered clothing?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject