GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation
Authors: Haoran Lu
Ruihai Wu
Yitong Li
Sijie Li
Ziyu Zhu
Chuanruo Ning
Yan Shen
Longzan Luo
Yuanpei Chen
Hao Dong
Venue: NeurIPS 2024 poster (2024)
Keywords: Garment Manipulation
Simulation
Benchmark
Primary area: robotics
TL;DR: We propose GarmentLab for garment manipulation with realistic and rich simulations and diverse benchmark tasks.

Abstract: Manipulating garments and fabrics has long been a critical endeavor in the development of home-assistant robots. However, due to complex dynamics and topological structures, garment manipulations pose significant challenges. Recent successes in reinforcement learning and vision-based methods offer promising avenues for learning garment manipulation. Nevertheless, these approaches are severely constrained by current benchmarks, which exhibit offer limited diversity of tasks and unrealistic simulation behavior. Therefore, we present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators. The abundant tasks in the benchmark further explores of the interactions between garments, deformable objects, rigid bodies, fluids, and human body. Moreover, by incorporating multiple simulation methods such as FEM and PBD, along with our proposed sim-to-real algorithms and real-world benchmark, we aim to significantly narrow the sim-to-real gap. We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks, highlighting the challenges faced by current algorithms, notably their limited generalization capabilities. Our proposed open-source environments and comprehensive analysis show promising boost to future research in garment manipulation by unlocking the full potential of these methods. We guarantee that we will open-source our code as soon as possible. You can watch the videos in supplementary files to learn more about the details of our work.

GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation
Haoran Lu 1.-
Ruihai Wu
Yitong Li
Sijie Li
Ziyu Zhu
Chuanruo Ningb
Yan Shen
Longzan Luot.2
Yuanpei Chen
Hao Dong"
CFCS, School of CS, PKU 2School of EECS, PKU WWeyyang College, THU
Abstract
Manipulating garments and fabries has long been critical endeavor in the develop- ment of home-assistant robots. However, due to complex dynamics and topological structures, garment manipulations pose significant challenges. Recent successes reinforcement learning and rision-based methods offer promising avenues for learn- ing garment manipulation. Nevertheless, these approaches are severely constrained by current benchmarks, which offer limited diversity of tasks and unrealistic simu- lation behavior. Therefore, we present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators. The abundant tasks in the benchmark further explores of the interactions between garments, deformable objects, rigid bodies, fluids, and human body. Moreover, by incorporating multiple simulation methods such FEM and PBD, along with our proposed sim-to-real algorithms and real-world benchmark, we aim to significantly narrow the sim-to-real gap. We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks, highlighting the challenges faced by current algorithms, notably their limited generalization capabilities. Our proposed open-source environments and comprehensive analysis show promising boost to future research garment ma- nipulation by unlocking the full potential of these methods. We guarantee that we will open-source our code as soon as possible. You can watch the videos supplementary files to learn more about the details of our work. Our project page is available at: hattps://garmee
Introduction
The next- -exeereentttion assistant robots should possess not only the abilities to separately manipulate wide variety of objects, including rigid, articulated [59], and deformable objects (58). but also the capability to leverage interactions between those physical media, including flow and fluids, order to assist humans (3).. Among various daily tasks [69 59 56), garment manipulation stands out of the most challenging, crucial, and extensively discussed tasks in the robotics and computer vision, due to its demanding requirements for understanding dynamic properties of physical instances and interactions between them. For instance, washing clothes entails the interaction between garments and fluids, while dressing up requires collaboration between robots and humans.
Garment manipulation tasks mainly presents three challenges. Firstly, each individual garment possesses nearly infinite states and exhibits complex kinematic and dynamic properties. Therefore, crucial for models to comprehend the various self-deform states of garments, which usually requires
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

o
MultiPhysics
Teleoperation
Real-Worle Benchmark
Figure 1: GarmentLab provides realistic simulation diverse garments with different physical propoerties.
benchmarking various novel garment manipulation tasks in both simulation and the real world.
large amount of training data [7 o (C1). Secondly, garment manipulation involves interactions with various types objects, including rigid (e.g., clothes hanger) and articulated (e.g., wardrobe) objects, as well as fluids and human body. Consequently, enabling models to understand these interactions across diverse physical media presents great significance (C2). Finally, considering that strategies for manipulating garments are often highly complex, and visual perception of garments more challenging due to their diverse states and patterns, manipulating garments faces a greater sim2real gap
633 29/ (C3).
Training a powerful agent capable of overcoming these challenges necessitates a vast amount of data encompassing robot-object interactions. However, directly collecting data from the real world is impractical. Thus, researchers have long pursued benchmarks for garment manipulation [30 Current deformable simulations suffer from drawbacks such missing garment meshes Additionally, they offer a limited range of tasks, hindering further research endeavors.
30
Therefore, we present GarmentLat (Figure [ a unified environment and benchmark for garment manipulation. GarmentLab) has three novel components satisfy the demands for diversity and realism: The powerful GarmentLab Engine, which is built upon Omniverse Isaac Sim 771 and supports a variety physical simulation methods. The simulator not only supports Position- -asedd- Dynamic (PBD) (3, Settee-lleeent--eeee objects but also makes integration with ROS 42 to provide an efficient teleoperation pipeline for data collection. GarmentLab Assets is a large-scale indoor dataset comprising I) garments models covering 11 categories of daily garments from ClothesNet 70 2) various kinds robot end-effector including gripper, suction and dexterous hands. 3) high-quality 3D assets including 20 scenes and 9000+ object models from ShapeNet [7]. Based on realistic simulation and rich assets, we propose GarmentLab Benchmark containing 20 tasks divided into 5 groups to evaluate state-of-the-art viseon-based and reinforcement learning based algorithm.
(FEM) (IT), to simulate garments, fluid and deformable
To tackle above challenges, our environment has three characteristics:1) Efficient. Garment manipu- lation involves nearly infinite object state and action spaces, requiring substantial data for models to understand garment structure and deformation. To meet this demand, our highly parallelized GPU-based simulator provides significant training advantage. Larger batch sizes enhance RL-based algorithms [33], while faster data collection speeds reduce training time for perception based algo- rithms (tackling C1). 2) Rich. The richness of our simulator can be categorized into two aspects: the diversity of simulation content offered by GarmentLab Assets and the depth of physical inter- action facilitated by Garmentl Engine. We emphasize multi-physics simulation, encompassing rigid-articulated, deformable- garmmn,, fluid dynamics, and flow, along with their interactions. This focus is vital for training agents capable of comprehending real-world physical properties (48] (tack- ling C2). You refer to videos in supplementary material for our simulation effects. 3) Real.

---

Table 1: Comparisons with Other Deformable Object Environments,
Thermalist pnermalistreant (tram//r/e//rr
Multi- Multi-Camera Camera
Scrnes
Rohot
Sim2real
Garment
Fluid
Rigid Body
Articulated
Human
FEM Objects
Flow
RealBenchmurrt Phy Physics
Mobile Tisk
Dexterous Tick
Rendering
Pipeline
Area
Softgyna 20
R
GPU
Manipulation
Orbit 34]
x
x
x
x
x
X
X
X
x
KI
GPU
Industrial
Sapien
CPU
Manipulation
Hahitat 3.0
x
x
x
4
x
X
X
x
x
R
CPU
Navigation
Bchaviod 28
4
4
x
x
X
94
36
X
x
KI
GPU
Manipulation
Mujoco
Manipulation
Pyballet
x
x
R
CPU
Manipulation
Rench 23
X
x
X
X
x
R
CPU
Manipulation
Gibson
CPU
Navigation
DeformableRavens411
x
x
R
CPU
Manipulation
Garmentl ab
V
RT
GPU
Manipulation
As the sim-to-real gap emerges the main obstacle in developing embodied agents, Garment Engine surpasses Omniverse capabilities by providing mature sim-to-real algorithms, such as Tele- operation [4]] utilized in the RL field, and the Visual Sim- Alignment Algorithm employed in perception algorithms. We also make integration with ROS [2] and Movelt [00,, which is beneficial for narrowing sim2real gap by introducing real-world robot motions into simulation (tackling C3).
Our benchmark experiments highlight the significant challenges current algorithms face, even with seemingly simple tasks like unfolding. These difficulties arise from a lack of understanding of physical interactions and the complexities high-dimensional states. Vision-based algorithms demonstrate limited generalization, with performance strongly affected by the initial state of objects. RL-based algorithms also encounter difficulties with tasks requiring long-horizon planning. These insights offer valuable guidance for improving methods for garment and deformable object manipulation.
In summary, we have made the following contributions in GarmentLab:
We propose Garmentl.a Environment, realistic and rich environment for garment manipula- tion, featuring diverse simulation methods, assets, object physics and multi-material interactions.
Based Garmentl Lab Environment, we propose GarmentL.a Lab Benchmark, benchmarking large variety of garment manipulation tasks, and providing the first real-world garment manipulation benchmark that can be reproduced internationally.
We integrate different sim2real methods into GarmentLab lab, providing solutions to narrowing the sim2real and further facilitating the real-world applications.
Extensive experiments and detailed analyses of different types of garment manipulation algo- rithms facilitate and enlight future research on garment manipulation.
2
Related Work
Garment and Deformable Object Benchmarks. Current deformable object environments [30 61
63
usually support only one simulation method (e.g., PBD or FEM), limiting the types of simulated
objects and interactions. Besides, most environments are CPU-based 62 63 12], severely limiting parallel capabilities and often exhibiting& a huge sim-to-real gap due to the absence of comprehensive sim-to-real algorithm designs. In contrast, as GPU-based simulator, GarmentLab provides diverse 3D meshes and supports various simulation techniques. We further integrate ROS and combine with our carefully designed sim-to-real pipeline, offering more comprehensive solution for researchers. Detailed comparisons between our environment and others can be found in Table I and Appendix C
Garment and Deformable Object Manipulation. Although current efforts excel at specific tasks such folding 63 57] and unfolding [T7 6), many real-world tasks are long- oong-horizon and involve interactions between various physical media. While many studies have potential to tackle these problems 48 (2], they are hindered by the lack of mature simulation platform capable of support- ing such diverse and complicated extensions. Furthermore, while current research predominantly emphasizes gripper manipulation tasks 655, we introduce tasks involving suction, dexterous hands, and mobile robots. We believe Garmentlaat will make unique and valuable contribution to the roboties community by providing new platform for developing garment manipulation algorithms and significantly expanding the scope of existing methods.
3

---

GormentLob Engine
Gormentioh ob Assets
GormentLob Benchmark
Fold
Robot Controller
Robot
Garment-Garment
ClothesPiles
Mobile Franks
Desterous hands
Fling
Physx Simulator
Finite Element Method
Scarf
Socls
Fat
Garment-Flud
Glove
Garment
Pasition Base Dynamics
Torb
Drecs
3krls
Germent-Deform
Clean
place
Hang
Garment-fiigdd
Place
Articulation and Rigid
Rigid
Articulated
Garment-Avolar
Dress Up
interface
Simiteal
Sim2ifeal
Reolloorld
Tedeoperationn
Planning
Sensor
Material
Aligmment
Cioject
Scene
The Architecture of GarmentLab. (Left) Built on PhysX5, our environment supports various simulation methods. (Middle) Our environment can deliver realistic simulations of diverse robots, garments, and interactions between multiple physics media. (Right) Subsequently, utilize these assets to construct tasks various categories. (Bottom) The framework supports real-world deployment.
Figure 2:
3 GarmentLab Environment
Garmentl.ar Lab aims to integrate state-of-the-ard physical simulation methods, modern graphics ren- dering engines, and user-friendly robotic interfaces into unified framework (Figure [}. Below we will first introduce Garmentlab Engine (Section 3.1 and Garmentlabb Asset (Section 3.2) to show our diversity function and objects. As we especially focus on the exploration of multiple physical simulation methods and interaction between them, we will introduce GarmentLab Physics in Section section We will talk about our novel-proposed tasks.
3.3
3.1 GarmentLab Engine
Built on NVIDIA's IsaacSim Garmentl .ab offers highly-paralleled data collection pipeline, real- istic rendering, support for various sensors, and integration with Robot Operating System (ROS) [42)
Data Pipeline. Data pipeline mainly consists of two components: Visual Data System and RL- Training System. Visual System provides both RGB-D observations and ground- semantic label including 2D and 3D bounding box, normals and instance segmentation. Based on IsaacGym, RL System can establish multiple agents on GPU at the same time for efficient training.
Rendering, GarmentLab supports multiple camera angles, such as eye-on-hand and eye-on-base perspectives, unlike the single-camera setups of past works 30 61], which employing naive OpenGL framework [49]. Additionally, it utilizes GPU-enabled ray tracing for rendering, which enhances realism and challenge by creating more realistic shadows and lighting [5I], thus reducing the sim2real gap and improving the performance of visual algorithms and mobile navigation tasks.
ROS. ROS generic and widely-used framework for building robot applications. We ROS to align robot in realworld and the simulation, please refer to Section for detailed. Also, although IsaacSim provides traditional Inverse-Kinematie Kinematic (36] and RMPFlow con also provide Movelt framework [10] for motion planning, which is more widely used in the real world.
Sensor. In addition to RGB-D observations, auxiliary observations can be accessed, such as robot joints, cloth particles and object poses. They are required in common RI. framework and teacher- student network [5). Other Omniverse sensors (c.g., tactile, contact-report) could also be available.

---

3.2 GarmentLab Assets
GarmentLab Asset compiles simulation content from a variety of state-of-the-ar datasets, integrating individual meshes or URDF files into complete, simulation-ready scenes with robots and sensors. We employ Universal Scene Description files to store all assets with attributes, including physics, semantics, and rendering properties. Key components along with their sources and categories are shown in Table 2 More details about each asset type are provided in Appendix
Table 2: Key Components of GarmentLab Assets.
Asset Type
Sources
Categories
Garment and Cloth
ClothesNet
Huts, Ties, Masks, Gloves, Socks, Dishcloths,
Rigid and Articulated
ShapeNet. PartNet YCB, PurNNet-Moility
Chairs, Boxes. Washing Machines. Storage Furniture..
Robot
Franka, URS, RidgchaccPranka, Hando
Human Model
Omniverse HumanModel
Materials
Onniverse Material
Fabrie, Carpet, Leather, Silk, Cotton,
Tops
Hat
Fluid
Pillow
Garment-Rigid
Fluid-Garment
Figure GarmentLab .ab Physics. GarmentL.n Lab explores the potential of different simulation methods, and provides different physical parameters, modeling the distinct properties of different materials in the real world.
3.3 GarmentLab Physics
Simulation Method. To ensure physically realistic simulation, tailored methods for different objects based on their physical characteristics. For large garments and fluid, we use Postion-Based Dynamics (3] For small elastic garments like gloves and socks, and everyday objects like toys and sponges, we apply Finite Element Method (F Human simulation involves articulated skeletons with rotational joints and surface skin mesh for high-fidelity rendering. Robot simulation utilizes PhysX articulation system for precise force control, P-D control, and inverse dynamics. Unlike previous works that rely on single simulation method, GarmentLab provides platforms for exploring dynamics and kinematics of various objects and the coupling and interactions among them. Diverse Physies Parameters. To fully exploit the potential of various simulation methods and make garment simulation more diverse and realistic, GarmentLat provides various physics parameter configurations. For example, as cloth is modeled as a grid of particles, altering parameters such as particle size and stiffness will change garment physical behaviors. Likewise, as depicted in Figure 3 diverse physical material parameters are assigned to diverse objects. These parameters encompass, but are not limited to, surface tension and cohesion for fluids, friction for rigid objects, and modulus. worth noting that parameters influencing the interaction between different objects, including contact offset and reset offset, are also adjusted. For further details, please consult Appendix D
5

---

4 GarmentLab Benchmark
GarmentLab Benchmark is motivated by the abilities that an intelligent manipulator agent should possess, including (1) understanding the physics of object interactions, (2) generating accurate action sequences for long-horizon, complex tasks, and (3) transferring this knowledge the real world. To test these abilities, we classified tasks into five categories based on physical interactions. We also proposed several complex, long-hone nong-horioon tasks advance future research. The demos of tasks and real-world experiment are shown in figure 4
Task Categories. To fully exploit the model's capability in understanding physical interactions and conduct comprehensive evaluations of current algorithms, we categorize 20 tasks into 5 groups. The example tasks and corresponding categories shown in Table Examples of task sequences provided in Figure 4 You can refer to Appendix to get more details.
Long- Horizon Tasks. With the advancement of robotics, there is growing focus on completing Long-horizon tasks, which integrate skills tasks such as 3D perception, manipulation and navigation. Thus we propose several long-horizen garment manipulation tasks, including organizing clothes, wash clothes, make up tables and dress up. These tasks go beyond simply executing subtasks in sequence, as they require holistically planning how accomplish the task based on the environment. During the execution, the algorithm needs to consider the positioning of the operation, the placement location, and carefully plan the path to avoid collisions. More analysis are shown in Section 7
Table 3: Task Categories of GarmentLab Benchmark.
Tisk Type
Focus
Example
Garment-Garmen
fundamental garmentt manipulation
fold, unfold, clothes piles retrieval...
Garment- Fluid
interaction beiween gurmentts and fluids as as flow.
Washing clothes, Drying clothes with huindyyer...
Garment-HHae HEMbbeess
manipulating FEM Objects
Packing clothes, Dexterous grasp plush toy
Garment Rigid
common interactions between clothing and rigid bodies
Hanging, Putting clothes into washing machine
Garment-Avatar
collaboration with human
Putting scarf, Dress person in T-shirt
5 Real-World Benchmark
Real-world benchmark is crucial for not only evaluating the real-world performance of different methods, but also providing a standardized platform for researchers reproduce and exchange methods. With the existence of real-world dataset or benchmarks for rigid [5), articulated [31] objects and furnitures [8] we introduce the first real-world benchmark for deformable objects and garments.
Unlike rigid or articulated objects that can be 3D-printed from CAD files, deformable objects are usually purchased without CAD files. Easily influenced by external forces, it is difficult accurately model garments directly using traditional multi- camlia calibration and surface reconstruction methods. commercial scanning devices with lasers and light for mesh scanning.
Therefore,
Selected objects cover diverse garments (tops, trousers, socks, hats), plush toys, household items (bags, clutches), and cleaning supplies. They are primarily selected from well-known international brands for durability and accessibility. To ensure variety, objects have different shapes, sizes, transparencies, deformabilities, and textures. For instance, our dataset features various tops made from materials like assault jackets, down jackets, shirts, and vests, with a wide range of physical attributes.
Additionally, we provide semantic human annotations for object part masks and key points, supporting dexterous manipulation such as grasping specific parts and object tracking using key points. Following 5 we present a systematic approach for defining manipulation protocols and benchmarks. These protocols specify the experimental setup for each task and provide procedural guidelines. A comprehensive description of the real-world benchmark is provided in Appendix F
6 Sim2Real Framework
Transferring models from simulation to reality is crucial yet challenging. Garmentl.at paves way to realistic application by integrating methods for mitigating vision (Sec. 6.1 and action (Sec. 6.2 gap.

---

Simulation Task
Fold
Hang
Fling
ClothesPile
Blow
DressUp
Grasp
StoreCloth
TidyUp
CleanTable
Long-Horizon Task
SetTable
WashHat
Real-World Experiment
Figure 4: Diverse Tasks of GarmentLab Benchmark. We introduced 20 garment and deformable manipulation
tasks including complicated long-horizon tasks. The last row shows the execution of these tasks in the real world.
a. Scan Pipeline
A. Model wearing T-shirt
B. Scan from all angles with scanner
C. Scan results
b. Real-World Benchmark
Figure Real-World Benchmark. Part demonstrates the whole pipeline converting real- weal-world objects into simulation assets. Part demonstrates the performance of different categories of objects in both simulation and the real world (the first row), and the results of these objects being manipulated by the robot (the second row).

---

Visual Sim2Real Algorithms
Learned Representations
(Voualord Poirblevel Comreppnndene))
Mcvelti
Human Demo
Teleoperation
add
before
after
ROS
Recording
offllne data
synchronize
keypoint embedding align
before
after
TealWorke Dealoy
simulation
RealAlu M Dittibbttion
Darributtoon
All gnont
sirnulation
real
replay
point cloud spatial alignn
before
after
Figure Sim2Real Framework. On the left, we highlight our Movelt teleoperation pipeline, a lightweight and casy-to-deploy system built using ROS, On the right, we present our three proposed visual sim-to-ret real algorithms, demonstrating significant improvement in model performance after deploying these algorithms.
6.1 Sim-Real Vision Alignment
Garmentl.a Lab intergrates several automated and self-supervised sim2real methods, and have verified
their effectiveness by predicting dense visual correspondence for manipulation [57] (Figure 6 Right),
with quantitative manipulation success rate in Table
Keypoint Embedding Alignment. Aligning corresponding skeleton point representations can mitigate representation gap between point cloud simulation and the real world [57] By attaching markers to skeleton points and enabling robot to perform self self-play, obtain ground-truth keypoint pairs and employ InfoNCE to align corresponding point representations. Shown in Figure 6 the alignment adapts representations to the real-world distribution. Appendix showws more details. Noisy Observation. Adding noise to point cloud for training can be very effective for sim2real transfer [16]. As shown in Figure 6 initial query results had many errors. By adding noise during training, our model became more robust, leading to smoother and more accurate representations. Point Cloud Alignment. We propose aligning point clouds by optimizing
affine matrix, using chamfer distance as the loss function. As shown in Figure 6 the model initially predicts incorrect results, even for flat surfaces. However, after alignment, it successfully predicts accurate results.
6.2 Real-World Motion Generation
For many algorithms, action trajectories generated in simulation is not align with those in the real world. We introduce two methods for generating trajectories in simulation that closely mimic real-world scenarios by leveraging prior knowledge of real-world manipulation trajectories.
We've developed a lightweight, cost-effective telcoperation system requiring just one-click deployment. It facilitates simultancous control of dexterous hands and grippers in both real-world and simulated settings (Figure 6). This system supports data collection for offline training, like diffusion policy. [67 9.. Implementation details in Appendix
Teleoperation.
Movelt. Incorporating Movelt into our framework elevates motion planning and obstacle avoidance beyond heuristic trajectory methods, noted in previous studies [57 63
Employing Movelt for real- world robot execution also aids visual algorithms. Adapting models to Movelt-generated trajectories during training reduces the sim-to-real gap. Detailed implementations are provided in Appendix H

---

7 Experiments
7.1 Simulation Experiment Setup
Methods. We selected three vision-based and two reinforcement learning (RI.) algorithms for experiment, with details listed in Table For vision-based algorithms, prioritized those utilizing dense representations for garments, as they have demonstrated generalization ability and are suitable for various downstream tasks.
Table 4: Benchmark Methods of GarmentLab.
Method
UniGammnnMaaiip (UGM) 57
DIFT
Affordance [8
RL-State 46
RL Vision
Type
3D Lo0-Visual-Con Corepp--Cre
JD-Visual-Corma Corusl-Corespomddeeee
3D Ro-RRepresnnttion
RL
RL
BackBone
PointNer++
Stable-Diffusion
PointNet++
PPO
PPO
Input
Point
RGB Image
Point
State-Bast Base-BBee GT
Partial Point Cloud
Tasks. Although we proposed many novel tasks, current algorithms cannot fully solve them. Thus, for large garments like tops, dresses, and trousers, we chose folding, hanging, and unfolding tasks. For small items like hats and gloves, selected hanging and placing tasks evaluate visual and RL algorithms. For dexterous and mobile tasks, existing work mostly employs RL algorithms. Hence, evaluated the performance of both RL-state-basser and RL-vision-bars algorithms separately.
Hang
Fold
Source Image
Target Image
T-shirt
One-piece
T-shirt
Drese
One-pioce
Top
Tops
One-picce
Jumpsuit
Socks
Trousers
Gloves
A Affordance
Correspondence
2D Correspondence
Figure 7: Qualitative Results. We visualize the qualitative results of the three vision-hased algorithms: the left, middle, and right sections this image correspond to the Affordance, Correspondence, DIFT algorithms, respectively. Note that the DIFT exhibits query errors. For detailed analysis, please refer to Experiment Section.
7.2 Simulation Result and Analysis
Table 5 present quantitative comparisons between vision- based algorithms and RL-based algorithms. Figure intuitively demonstrates the visual results three different vision algorithms.
Table 5: Simulation Results Traditional Tasks. Numbers in the Large- column represent scores for Tops, Trousers and Skirts. Numbers in the Small-piece column represent scores for Hats and Gloves.
Large-piece
Small-piece
Method
Fold
Unfold
Hang
Place
Hang
UGM
61.5 62.1 59.8
58.3 60.5 57.2
61.8 17.5/5 15.55/99..
33.2/35.4
31.8 29.2
DIFT
32.7 / 36.7 31.2
23.3 17.6
31.2 27.6/ /
63.2/63.2
64.7 / 61.2
Affordance
/3.2//1..8 56.9
32.4/ /3..
64.1 60.2 61.3
63.2
62.6 / 60.4
RL-State
14.8 12.5 /9.8
6.5/8.8 12.7
13.1 19.7 14.7
14.2 12.1
12.8 / 13.2
RI-Visioe Vision
6.7 8.2
5.2/ 6.2/ 8.8
7.3/5.
13.1 14.8
11.3 15.2

---

Vision-Based Algorithm. Among the three vision- based algorithms, UGM performed best on large- piece clothing, emphasizing cross-deforn and cross-object consistency learning representations. DIFT excels with small-piece clothing due to its robustness to object rotation but lacks proficiency in understanding clothing folding. Affordance works well for tasks that do not require precise point selection, such as hanging, but struggles with folded garments.
RL Algorithm. Compared vision-based algorithms, RL performs poorly on garment manipulation due to the complex dynamics of garments. Our analysis of training videos showed that RL often generates abnormal trajectories, causing clothes to get tangled with the robotic arm or be pushed away. This issue is more pronounced with RL-vision-bars bassd methods, as the higher-dimensionn and partial visual observations hinder the model's ability to converge on an effective strategy. For dexterous and mobile tasks, the larger action and search spaces result in suboptimal performance. Further discussion and analysis can found in Appendix
7.3 Real-World Experiments
In our real-world experiments, we focused on testing vision-based algorithms due to the risk associated with RL actions. T-shirts for folding and hats for hanging, were selected for experimentation. Additionally, we conducted ablation study on proposed sim2real methods using UGM (Table (6). Our real-world results align with our simulation findings, indicating GarmentLab environment can enhance real-world applications. For sim2 real algorithm, without point cloud alignment and noise augmentation along with keypoint embedding alignment can improve representation smoothness and accuracy. Qualitative sim2real results are shown in Figure (Right).
Table 6: RerldWorld Experiment and Ablation Results, w/ PA, w/o Noise, and w/o EA respectively represent UniGarmentManip without Pointcloud Alignment, Noise Augmentation, and KeyPoint Embedding Alignment.
Method
UGM
DIFT
Affordance
w/o PA
w/o Noise
w/o EA
Tshirt- Tshirt-Folding
10/15
8/15
6/15
2/15
8/15
7/15
Hanging
10/15
14/15
9/15
5/15
8/15
9/15
Conclusion
We introduce GarmentLab, comprehensive environment and benchmark for manipulating garments and deformable objects. GarmentLab includes the GarmentLab Engine, supporting various simulation methods and ROS integration; GarmentLab Assets, diverse dataset of robots, materials, and garments; and GarmentLab Benchmark, proposing several novel tasks. It also provides the first real-world deformable benchmark along with several sim2real methods.
Acknowledgment
This project is supported by The National Natural Science Foundation of China (No. 62376006), the National Youth Talent Support Program (8200800081) and the National Natural Science Foundation of China (No. 62 62136001).
10

---

References
[1] Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, and Ken Goldberg. Speedfolding: Learning efficient bimanual folding garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8. IEEE, 2022.
[2] Arpit Bahety, Shreeya Jain, Huy Ha, Nathalie Hager, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Bag all you need: Learning a generalizable bagging strategy for heterogeneous objects. IROS, 2023.
[3] Jan Bender, Matthias Müller, and Miles Macklin. Position-based simulation methods in computer graphics. Eurographics, 2015.
[4] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In European Conference on Computer Vision, pages 344-359. Springer, 2020.
[5] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pages 510-517, 2015.
[6] Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Cloth funnels: Canonicalized allimmmen for multi- purpose garment manipulation. In International Conference of Robotics and Automation (ICRA), 2022.
[7] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, and Fisher Yu. Shapenet: information-rich 3d model repository. ArXiv, abs/1512.03301, 2015.
[8] Wei Chen, Dongmyoung Lee, Digby Chappell, and Nicolas Rojas. Learning to grasp clothing structural regions for garment manipulation tasks. arXiv preprint 11XXi:23306145533 2023.
[9] Cheng Chi, Siyuan Feng, Yilun Du Zhenjia Xu, Eric A. Cousincau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. ArXiv, abs/2303.04137, 2023.
[10] Sachin Chitta, Ioan Alexandru Sucan, and Steve B. Cousins. Moveit! [ros topics]. IEEE Robotics Autom, Mag., 19:18-19, 2012.
[11] Philippe G. Ciarlet. The finite element method for elliptic problems. In Classics in applied mathematics, 2002.
[12] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. arXiv, 2016.
[13] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J. Guibas. Vector neurons: general framework for so(3)-equivarian networks. 2021 IEEE/CVE International Conference Computer Vision (ICCV), pages 12180-12189,200
[14] Peter Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. Conference on Robot Learning. 2018.
[15] Haoran Geng, Ziming Li, Yiran Geng, Jiayi Chen, Hao Dong, and He Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. 2023 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 2978 2988, 2023.
[16] Jiayuan Gu. Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei. Yunchao Yao, et al. Maniskil12; unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.0444 04659, 2023.
[17] Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pages 24-33. PMLR, 2022.

---

[18] Minho Heo. Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for Long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
[19] Sebastian Hofer, Kostas E. Bekris, Ankur Handa, Juan Camilo Gamboa, Florian Golemo, Melissa Mozifian, Christopher G. Atkeson, Dieter Fox, Ken Goldberg, John Leonard, C. Karen Liu, Jan Peters, Shuran Song, Peter Welinder, and Martha White. Perspectives on sim2real transfer for robotics: A summary of the r: Ss 2020 workshop. ArXiv, abs/2012.03806, 2020.
[20] Sebastian Höfer, Kostas E. Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Christopher G. Atkeson, Dieter Fox, Ken Goldberg, John Leonard, C. Karen Liu, Jan Peters, Shuran Song, Peter Welinder, and Martha White. Sim2real in robotics and automation:
Applications and challenges. IEEE Trans Autom. Sci. Eng., 18:398-400 400, 2021.
[21] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan A. Carr, Jonathan SeeannKelleyy. and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ArXiv, abs/19 abs/1910.00990 00935, 2019.
[22] Zhiao Huang. Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In ICLR, 2021.
[23] Stephen James, Z. Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5:3019- 3026, 2019.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo. Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision
(ICCV), pages 3992-4003, 2023.
[25] Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint
2019.
[26] Michael J. Landau, Benjamin Y. Choo, and Peter A. Beling. Simulating kinect infrared and depth images. IEEE Transactions on Cybernetics, 46(12):3018-30) 2016.
[27] Anqi Li, Ching-An An Cheng, Muhammad Asif Rana, Mandy Xie, Karl Van Wyk, Nathan D. Ratliff, and Byron Boots. Rmp2: A structured composable policy class for robot learning. ArXiv, abs/2103.05922, 2021.
[28] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- Ik: benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference Robot Learning, pages 80-93. PMLR, 2023.
[29] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019.
[30] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held, Softgym: Benchmarking deep reinforce- ment learning for deformable object manipulation. In Conference on Robot Learning, pages 432-448. PMLR, 2021.
[31] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu. Akb-48: A real-world articulated object knowledge base. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14809-14818, 2022.
[32] Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae-Yong Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG), 33:1 - 12, 2014.
[33] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu based physies simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
12

---

[34] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Roboties and Automation Letters, 8(6):3740-3747, 2023.
[35] Kaichun Mo, Shilin Zhu, Angel X. Chang, L. Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine- fine-grineed and hierarchical part-level 3d object understanding. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 909-918, 2018.
[36] Yoshihiko Nakamura and Hideo Hanafusa. Inverse kinematic solutions with singularity robust- ness for robot manipulator control. Journal of Dynamic Systems Measurement and Control- transactions of The Asme, 108:163-171, 1986.
[37] Open Robotics. Franka emika panda robot. http::///roboot..cccoooooot/////
Emika- Pandal June 2024.
[38] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2017.
[39] Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yoes Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimr Vondru, Thophile Gervet, Vincent-Pierre Berges, John Turner Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Devendra Jitendra Malik, Singh Chaplot, Unnat Jain, Dhruv Batra, AksharaRai, and + RoozbehMottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots. ArXiv, abs/2310. 13724, 2023.
[40] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[41] Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dietor Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. RSS, 2023.
[42] Morgan Quigley. Ros: souncs robot operating system. In IEEE International Conference on Robotics and Automation, 2009.
[43]
Reallusion. Actorcore.
2024.
[44] Shadow Robot. Shadow dexterous hand series.
https//ww..haaoxroootccco
haneeroou--hdd-serebee 2024.
[45] Rockwell, ridgeback-
Ridgeback. plattfor/ 2024.
https//cceerpatttooottccccc
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1 777.06347, 2017.
[47] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, and Andy Zeng. Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[48] Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. arXiv preprini arXiv:2306. 14447, 140.77477..44 2023.
[49] Dave Shreiner, Mason Woo, Jackie Neider, and Tom Davis. Opengl programming guide: the official guide to learning opengl. In Shreiner, 1993.
[50] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se(3aeequivariant object representations for manipulation. 2022 International Conference on Robotics and Automation (ICRA), pages
2021.
13

---

[51] P. Slusallek, Peter Shirley, William R. Mark, Gordon Stoll, and Ingo Wald. Introduction to real-time ray tracing. ACM SIGGRAPH 2005 Courses, 2005.
[52] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. ArXiv, fbss2200.038881, 2023.
[53] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033 IEEE, 2012.
[54] Qianxu Wang, Haotong Zhang, Congyue Deng. Yang You, Hao Dong. Yixin Zhu, and Leonidas J. Guibas. Sparsedff: Sparse-view feature distillation for one-shot dexterous manipulation. ArXiv, Abs/2310.16836 2023.
[55] Yufei Wang, Zhanyi Sun, Zackory Erickson, and David Held. One policy to dress them all: Learning to dress people with diverse poses and garments. In Robotics: Science and Systems (RSS), 2023.
[56] Ruihai Wu, Kai Cheng. Yan Zhao, Chuanruo Ning, Guanqi Zhan, and Hao Dong. Learning environment-awwre affordance for 3d articulated object manipulation under occlusions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[57] Ruihai Wu, Haoran Lu, Yiyan Wang, Yubo Wang, and Dong Hao. Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual correspondence. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[58] Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[59] Ruihai WI, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022.
[60] F. Xia, Amir Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9068 9079, 2018.
[61] Zhou Xian, Bo Zhu. Zhenjia Xu, Hsiao- Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. In The Eleventh International Conference Learning Representations, 2023.
[62] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. Sapien: simulated past-based interactive environment. 2020 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11094-11 11104, 2020.
[63] Han Xue, Yutong Li, Wenqiang Xu, Huanyu Li, Dongzhe Zheng, and Cewu Lu. Unifold- ing: Towards sample-efficient, scalable, and generalizable robotic garment folding. ArXiv, Abs/2311.01220 2023.
[64] Han Xue, Wenqiang Xu, Jieyi Zhang, Tutian Tang, Yutong Li, Wenxin Du, Ruolin Ye, and Cewu Lu. Garmenttracking: Category-level garment pose tracking. 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 21233-21242, 2023.
[65] Han Xue, Wenqiang Xu, Jieyi Zhang, Tutian Tang, Yutong Li, Wenxin Du, Ruolin Ye, and Cewu Lu. Garmentrracking: Category-level garment pose tracking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21 21233-21242, 2023.
[66] Kevin Zakka, Laura M. Smith, Nimrod Gileadi, Taylor A. Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Peter R. Florence, Andy Zeng, and Abbeel. Robopianist: A benchmark for high-dimensional robot control. ArXiv, abs/2304.04150, 2023.
14

---

[67] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. ArXiv, abs/2403.03954 2024.
[68] Fan Zhang and Yiannis Demiris. Learning grasping points for garment manipulation in robot- assisted dressing. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 9114-9220. IEEE, 2020.
[69] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang. Qingman Fan, Kaichun Mo, and Hao Dong. Dualafford: Leariing collaborative visual affordance for dual-gru duul-gripper object manipulation. International Conference on Learning Representations (ICLR), 2023.
[70] Bingyang Zhou, Haoyu Zhou, Tianhai Liang, Qiaojun Yu, Siheng Zhao, Yuwei Zeng, Jun Lv, Siyuan Luo, Qiancai Wang. Xinyuan Yu, Haonan Chen, Cewu Lu, and Lin Shao. Clothesnet: An information-rich 3d garment model repository with simulated clothes environment. ICCV, 2023.
[71] Zhehua Zhou, Jiayang Song, Xuan Xie, Zhan Shu, Lei Ma, Dikai Liu, Jianxiong Yin, and Simon See. Towards building ai-cps with nvidia isaac sim: An industrial benchmark and case study for robotics manipulation. arXiv preprint urXiv:2308.00055 2023.
[72] Berk Çalli, Aaron Walsman, Arjun Singh, Siddhartha S. Srinivasa, Abbeel, and Aaron M. Dol- lar. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. ArXiv, hbb//1502.0344. 2015.
15

---

Appendix Overview
A GarmentLab Asset
GarmentLab Lab asset contains vast range of objects which commonly seen in our daily life, from rigid object, articulated object, garment with various materials to all kinds of robots and human models.
B Experiment
Details about how the experiments described in the main paper is conducted, such as partition of training and testing sets and evaluation metrics definition.
C Related Work
The main reference work under the topic of traditional embodied and robotic simulator, deformable objects and cloth Benchmark, downstream tasks and algorithms of garment manipulation.
D Physics Simulation
Details about the simulation methodology of different types of physies simulation, from garment, deformable body, rigid body to wind and fluid. Discussion of how the changing of physics parameters leads to different simulation performance.
E Sim2Real
More detailed description of sim-to-real visual alignment and motion generation methods with specific algorithm process and quantitative mathematical formulas.
F R Real-World Benchmark
Discussion about the principle and consideration of object selection. Detailed process of scanning real- world objects into simulation models and protocol benchmark guidelines garment manipulation.
G Task
The task categories and detailed settings with multi-physite interactions, including but not limited to Comment-garmen task, garment-avata avattar task. Long- Long-horizon tasks are also discussed here.
H Movelt
How trajectories generated by Movelt are recorded and how to adapt visual models in the simulator the trajectories, aiming narrow sim-to-real gap.
Teleoperation
Detailed process of how the motion of human demonstration is captured and then adapted embodi- ment, bringing robots with action in level intelligence.
Limitation
Limitation of our work.
K
Broader Impact
The potential socictal impacts of our work.
16

---

A
GarmentLab Assets
Rigid Object We mainly import objects from (eapeeNNtt(PPPaaNeeeete and YCB dataset 72). Note that we have filtered out objects that are not suitable for physical simulation and have issues interacting with garments fluid, and then reorganized and reclassified the dataset.
state spaces, articulated objects are, however, generally more difficult to understand and interact with compared to 3d rigid objects. We mainly import articulated objects from PartNet-Mobilig dataset [62] including Chair, Box, Bucket, Washing machine and Storage Furniture etc, to establish the comprehensive tasks for indoor robots such as folding clothes and putting them into the wardrobe.
Articulated Object Having much higher Segrre-of-freedom//0
Garment and Cloth We select garments from ClothesNet [70], a large-scale dataset of 3D clothes objects with information-rich annotations. We select garments from 11 categories including Hat, Tie, Mask, Gloves and Socks and use two physical simulation method to simulate them. We also include standard square cloths, like dishcloths and tablecloths, cover indoor task needs. Note that there are still gaps between meshes and ready-to-sinulate object, do post-processing of garments including giving correct physical parameters to simulate them.
Robot. We deploy a variety of specialized robots for diverse tasks, including a 7-DoF Franka manipulator (37] with a parallel gripper, URS with suction for manipulation tasks, and a RiddebackFFranaa455 with wheels for mobility and navigation. For dexterous tasks, we use ShadowHands 44 mounted on a UR1
Human Model. We incorporate human model to construct long- oonghhorizon tasks, such as dressing up. Utilizing avatars selected from actorcore 43), we assign specific motions to each avatar facilitate collaboration with robots. Each avatar comprises articulated joints, surface skin mesh and clothing, enabling realistic simulation of human structure and motion.
Materials. Materials are crucial components of virtual relightable assets, defining the interaction of light at the surface of geometries. We carefully choose materials from the Omniverse Base Material library to attain optimal rendering outcomes, a critical aspect for visual-based algorithms. Moreover, diverse textures can aid algorithms in understanding the relationship between an object's appearance and its physical behavior.
B Experiment
B.1 Overview
Generalization ability. As a novel environment, GarmentLab especially focus on evaluating and improving the generalization abilities of algorithm. We evaluate the generalization ability from the following aspects. Novel Object Thanks to rich GarmentLab Asset, we split garment and other object dataset into Train/Var/Test proportion 70%//1%//15% to test algorithms generalization ability on object level. Moreover, as garment and deformable object have nearly infinite self-deform state, we introduce Novel State. For example, we disturb garment initial state and test model's ability on handling wrinkled and folded clothes. Moreover, in order to improve algorithm ability of planning and collision avoidance, we also involve Novel Scene, for task like make up table, the shadow of irrelevant object can also influence navigation.
Metrics. We primarily use the success rate the evaluation metric. It is important note that because garments and fluids can easily change state due to gravity or friction, we consider task successful if it meets the success criteria and maintains this state for least five seconds. For tasks that appeared in previous work, we adopted the widely accepted success criteria and tolerance thresholds from the goal state. For example, we use ceeereection-ver--nnee and the folded garments to evaluate folding task|57 6 and use coverage area to evaluate unfolding [7]. For novel tasks such as washing or blowing, the goal states and tolerances are derived according to human behaviors.
(IOU) between the target
17

---

B.2 Experiment Task Setting
For large garments like tops, dresses, and trousers, we chose folding, hanging, and unfolding tasks. And for small items like hats and gloves, we selected hanging and placing tasks to evaluate visual and RL algorithms. The detailed experiment settings for the tasks listed above are shown below:
Garment-hanging task requires robots to hang garments success is that the garment can be hanged steadily (five seconds in real experiment) on the couple. Then to ensure that the garment is hanged in the right pose (not hanged at sleeve or other strange cases), we compare the manipulation result with a standard human demonstration by computing the sum particle-to- teleettt---ttlleeee distance between two states. The total distance within predefined value will be regarded as success. The initial states of garment to hang is obtained by dropping from random initial poses over the ground.
fixed couple. The first criterion of
Unfolding task requires robots to unfold garments at random deformations to be flat. Follow ClothFunnel]66] unfolding task success when the garment ground-truth vertices within reasonable range of the initial state, i.c., the flat state before the vertices were disrupted. This is because we found that using coverage area defined by FlingBot may not reasonably reflect success as the garment structure becomes more complex. The initial states of garment to unfold is obtained by dropping from random initial poses over the ground.
Folding task requires robots to fold garments from a flat states. After manipulation, we calculate the particle-to-p to-particle distance between a.the final state of garment after manipulation trial and b.the garment state obtained by human demonstration. The manipulation whose total distance is lower than predefined value can be regarded success. The initial states of garments fold are obtained by placing flatly on random position with small disturbance.
Hat- Hat-Han task requires robots to hang hats to fixed couple. The criterion of success is that the hat can be hanged steadily on the couple and would not fall on the ground. The initial states of hat to hang is obtained by dropping from random initial poses over the ground.
Placing task requires robots to get the hats/gloves which are previous hanged at the couple. The placing task succeeds when the hats/gloves are fetched and placed on the right position without falling on the ground. The initial states of hats/gloves to fetch and place are obtained by random dropping from a random poses over the couple, where only the successfully hanged cases will be used for training and testing.
B.3 Detailed Analysis
Vision-Based Algorithm. Comparing the three vision models, we found that Maina (UGM) have the best performance on Large-piece of garments. We conjecture that this stems from (1) The consistence of representation on self-deformations. As model have the understanding of deformation on garments, it is casier to detect keypoints required in Folding tasks and Fling tasks in diverse garment states. (2) The explicit design of cross-object representation consistency. Maina
skeleton(a graph of keypoints) as the shared bridge for different garments with similar structures, which makes model have the ability to understand the topology of the 3D object in the same category. However, we found that this result is not as good as the original paper using Pyllex. This could be because we introduced the robot and the scene here, so some invalid selection points, such as those beyond the robot's reach causing collisions, were considered unsuccessful. Additionally, while the original paper used only T-shirts, our study included jackets and other garments with front openings, this wider variety of clothes also increased the complexity of our task. For Affordance, we found that it performs well in Hanging tasks possibly because of its task-specific designed which makes it chooses the grasp points more accurately. In contrast, DIFT has poor performance on these three tasks especially on unfolding tasks due to the unawareness of garment deformations on 2D pretrained correspondence. This is reasonable because most objects in world for training do not have garment-level deformations. However, DIFT perfors better with small-piece garments like hats and gloves due to their minimal deformation, Besides, the pretrain model based on large diffusion model are more robust to rotation, which is crucial for handling small clothes. For UniGarmentManii and Affordance, they are not 3D-equivariant models, so they are more sensitive to rotations, resulting in poorer performance with small clothes compared to DIFT.
18

---

RL-Based Algorithm. We modified the traditional PPO by directly replacing the value net of PPO with the success information from the simulation ground-trutt -ruth information. This is because in robotics tasks, the value of the policy can be easily obtained from the ground-truth information. At the same time, following (l and FlingBot[17] we primarily used RI. for selecting points and adopted scripted policy for the trajectory, with simple adaptations based on the selected points. This is because directly using RL to train the trajectory is particularly unstable. We will elaborate on this point in more detail below.
We found that PPO's performance on state-based tasks was significantly worse than visual algorithms. After analyzing the training videos, we identified several reasons for this: (1) Abnormal trajectories of the robotic arm caused collisions and pushed the clothes far away. (2) Large reward fluctuations in long- long-horizon tasks led to training instabiliity, as the robotic arm's 's random folding actions in the early stages caused significant reward variations. (3) Completing long-horizon tasks was difficult due to the robotic arm's 's abnormal trajectories disrupting previous steps, such interfering with the sleeves during the folding task. (4) Wide-ranging movements of the robotic arm caused clothes to wrap around it, particularly in the hanging task, leading to failure. The visual algorithm avoided these issues because its execution trajectories were mostly predefined, such as pick-and-place or fling trajectories.
We also found that the performance of visual-based PPO is significantly inferior to based PPO, due to the following reasons: (1) The higher dimensionality of visual input makes training more difficult. (2) The visual input consists of partial point clouds, which can confuse the model, especially for thin objects like clothing. (3) Detecting the object position is more challenging for visual input, resulting algorithmic failures during the grasping stage. These findings are consistent with those of SoftGrym 30
B.4 Training Details of Main Algorithms
B.4.1
UniGaarmentMannnuaation (UGM)
For Hyper-parameter parameters selection, we set batch size be 32. In each batch, we sample garment pairs. For each garment pair, we sample 20 positive and 150 negative point pairs for each positive point pair. Therefore, in each batch, 32 x 32 x 20 data will be used to update the model. During the Correspondence training stage, we train the model for 40,000 batches. During Coarse-to- to-fie Refinement, we train the model for 100 batches. During Few-shot Adaptation, we slightly refine the model using 5 demonstration data. Besides, we set the number of skeleton pairs to be 50.
For computational resource, we use PyTorch our Deep Learning framework. Each experiment is conducted on an RTX 3090 GPU, and consumes about 22 GB GPU Memory for training. It takes about 12 hours train the Coarse Stage, with 1-2 hours of Coarseto-fine Refinement and 0.5 hour's Few-shot Adaptation.
B.4.2 Affordance
For Hyper-parameters selection, we set batch size be 128, where each pair contains one positive manipulation point and one negative manipulation point the same garment, automatically balancing the training data, "Positive" means manipulating on that point can lead to the success the whole task while "negative" means failure. In each batch, 128 x 2 data will be used to update the model. During the Affordance training stage, we train the model for 36,000 batches. The model designed make binary classification with cross-entropy as loss function. The output of affordance model reflects the success rate when manipulating on that point, which ranges from 0 to During manipulation, we just select the point with the highest score to manipulate.
For computational resource, we use PyTorch as our Deep Learning framework. Each experiment is conducted on an RTX 3090 GPU, and consumes about 16 GB GPU Memory for training. It takes about 18 hours to train the model for a task with a specific category of garment.
B.4.3 DIFT
As pretrained model, DIFT use stable-diffusion as backbone. For Hyper- selection and prompt enginering We use the default parameters DIFT. We crop the image size to 762 762 and set timestep for diffusion to 261. The ensemble size was set to 8. We use official network architecture
19

---

pipeline followed by our own designed robot excution pipeline. The robot execution pipeline is similar to UniGarmentManip. We only substitute the query model DIFT. For computational resource, we use PyTorch as our Deep Learning framework. Each experiment is conducted on an RTX 3090 GPU, and consumes about 20 GB GPU Memory for inferencing.
C Related Work
Traditional Embodied and Robotic Simulator The simulator plays an indispensable role in robotics development as it allows for the rapid and safe acquisition of vast amounts of interaction data, facilitating the implementation of various algorithms. However, the majority of mainstream robot 33 62 primarily support rigid object simulation including the collision and friction between them. Besides, most of robot simulators are CPU-based 39 66 [4], severely limiting their parallel capabilities and resulting in slow training speeds. Additionally, these simulations exhibit significant sim2real gap due to the absence of comprehensive sim2real algorithm designs (2) (3
simulators 53 12
19 Nevertheless, based on Isaac Sim [7], our benchmark not only supports parallel data collection but also incorporates comprehensive sim2real designs, including RL-based and vision-based algorithms. Deformable and Cloth Benchmark In recent years, there has been a surge in deformable and garment simulation environments 30 22 61. However, the most server problem of these kinds of simulation or benchmarks is that they can only simulate certain kinds of objects as they only support one simulation method, which makes it impossible to explore the physical interaction between multiple kinds of objects. Moreover, these benchmarks are lack of diversity as they are built directly on the underlying simulation architecture and have not integrated with mature platforms, thereby limiting the range of simulated objects and scenes. For instance, softgym 30), built on NVIDIA Flex (32,, is confined to simulating tops and trousers while fluidlab(61). built Taichi (1], can only simulate fluid and performs poorly on rigid objects simulation. Additionally, many benchmarks(66) 30 lack the ability to import robots and establish real grasps, posing significant challenges for joint control and vision-based algorithms. By contrast, GarmentLab provides sophisticated 3D meshes and facilitates various simulation techniques, enabling the modeling of garments, fluids, flow dynamics, avatars, rigid and articulated objects, and their interactions. This inclusive and adaptable platform offers a more comprehensive solution for research and development. The full detailed comparison of out benchmark between others can be found in Appendix A.
Garment and cloth manipulation Manipulating a single garment or cloth is a well-studied area, with previous works focusing on learning policies for specific tasks such as folding [ 63. unfolding [77, grasping {8 68 and dressing-up [55]. However, as many daily tasks involve interactions between various physical media, current algorithms often fall short in solving real-life tasks. Although many proposed algorithms have full potential to solve these problems (59 29] they are hindered by the lack of a mature simulation platform capable of supporting such simulations. Furthermore, while current research predominantly emphasizes gripper manipulation tasks, we introduce tasks utilizing suction, dexterous hands, and mobile robots. We believe that GarmentLab will make unique and valuable contribution to the robotics community by providing a new platform for developing garment manipulation algorithms and significantly expanding the scope of existing methods.
D Physics Simulation
D.1 Modeling methodology
D.1.1 Position-Based Dynamics (PBD) for Garment
-assed Dynamics (PBD) is an efficient and stable method for simulating cloth, particularly suitable for complex garments like dresses. PBD models deformable objects as systems of inter- connected particles governed by constraints that dictate their physical interactions and behaviors. In PBD, a dress is represented as a triangular mesh where particles serve as discrete points on the cloth surface with attributes such as position 24, velocity 0; and inverse mass Ws. The method operates by directly manipulating particle positions to satisfy series of constraints, achieving stable simulations of deformable materials. These constraints include stretching constraints, which enforce distance maintenance between neighboring particles to prevent excessive elongation, mathematically defined as C(r,, 23)) = ||2 - l - d, where d the rest distance between particles x; and æzj- Bending constraints maintain angles between adjacent triangles in the mesh to simulate resistance to
20

---

bending, formulated as Cz2, 2j, Ik), where the constraint function depends on the angle between particle triplets. Collision constraints detect and resolve collisions between particles and other objects. ensuring realistic interactions within the environment. The PBD algorithm involves initializing particles and constraints based on the dress's geometry and material properties, applying external forces such gravity, predicting new particle positions p. P; At o where At is the time step, iteratively adjusting particle positions to satisfy constraints, updating particle velocities, and determining final positions for each time step.
D.1.2 Position-Basee Dynamics (PBD) for Fluid Simulation
Position-Base Based Dynamics (PBD) is a powerful method for simulating fluids due to its computational efficiency and stability. PBD treats fluids as collections of particles, where each particle represents small volume of the fluid. Constraints are applied to ensure physical properties such as incompress- ibility and realistic fluid behavior. In fluid simulation, particles are characterized by attributes such as position velocity vs, and inverse mass We.
A key constraint type in fluid simulation is the density constraint, which ensures that the fluid maintains a constant density. The density constraint for a particle can be defined as:
C2(x)= = Σ my =51l Po
where W the smoothing kernel function, h is the smoothing length, my is the mass of particle j. and po is the rest density of the fluid. Collision constraints handle interactions between fluid particles and solid boundaries, ensuring particles do not penetrate solid objects.
The PBD algorithm steps for fluid simulation include initializing fluid particles with positions, velocities, and masses, applying externa forces such as gravity, computing predicted positions D; p4 + At. 14, adjusting particle positions to satisfy density and collision constraints, updating particle velocities based on the corrected positions, and integrating the updated positions and velocities for the current time step.
D.1.3 Finite Element Method (FEM) for Simulating Deformable Objects
The Finite Element Method (FEM) is a robust numerical technique for simulating the mechanical behavior of deformable objects, ideal for intricate geometries and diverse material properties, such as a toy bear. FEM discretizes the object into mesh of finite elements and solves the equations of motion accurately capture realistic deformations under various forces.
In FEM, the deformable object is represented by mesh consisting of nodes and elements. Nodes points where the equations of motion are solved, and elements polyhedral shapes, such as tetrahedrons, that connect these nodes. Material properties, including elasticity, density, and damping, determine the response of the object applied forces. Modeling deformable body involves several key steps. First, a deformable body component is added to the mesh which generates collision and simulation tetrahedral (tet) meshes from the source mesh. The mesh is then separated into visualization, collision, and simulation tetmeshes, each serving distinct purposes in rendering, collision resolution, and simulation. Configuring the material properties involves defining characteristics such as stiffness and dynamic friction by creating and binding new deformable body material.
D.1.4 Flow Models for Simulating Wind Effects
Flow models are essential for simulating wind effects, capturing the interactions between fluid (air) and objects. These models represent phenomena such as airflow, turbulence, and aerodynamic forces. The typotypoequation form the core of flow models and include the continuity equation for mass conservation 0g DE + V (pu) 0, where p is the fluid density and is the velocity vector; the
momentum equation for force balance @(pu) + V (puu) = - Vp + V + + pg. where is the Øt
pressure, is the stress tensor, and g is the gravitational acceleration; and the energy equation for
thermal effects 8(FF) + (pFu)= = - q + r: Vu + p(g u), where E the total energy per St
unit mass and q is the heat flux vector.
21

---

Wind effects are modeled by defining wind sources, preparing high-resolution meshes, setting boundary conditions, configuring the appropriate flow model (e.g., LES or RANS), and running the simulation to compute wind interactions iteratively. This approach ensures accurate and dynamic representations of wind effects in various environments.
D.1.5 Rigid Body Simulation
Rigid body models are essential for simulating solid objects that move and interact based physical laws without deforming. These simulations accurately represent the dynamics of solid objects under various forces. Key components include rigid body component, which provides properties like linear and angular velocity, and a collision component, which defines how the body collides with other objects. The dynamics of rigid bodies governed by solvers such as Temporal Gauss-Seide (TGS) and Projected Gauss-Seidel (PGS), which ensure stability and efficiency. TGS improves convergence by considering temporal aspects of the simulation, while PGS iteratively projects velocities to satisfy constraints.Rin Rigid bodies interact through collisions defined by collision shapes, which can be approximated using convex hulls, bounding shapes, or signed distance fields (SDFs). These approximations balance accuracy and computational performance. Mass properties of rigid bodies are derived from the volume and density of their collision geometries. For more precise control, explicit mass or density values can be set using a Mass component. This allows for accurate simulation of complex interactions and dynamic behaviors.
D.2 Multi-Physics Simulation Parameters Table
To maximize the value of different simulation methods, we assigned different parameters to various objects. In Table we list all the adjustable parameters.
D.3 Parameter Effects Physical Properties
In most cases, changes in parameters do not significantly alter the physical properties. For PBD simulations involving garment, the Particle Contact Offset parameter affects the thickness of the fabric; as its value increases, the fabric becomes progressively thicker. The Rest Offset parameter influences the distance between the dress and the ground upon landing, with an increase in this value resulting in greater distance between the dress and the ground after it lands.
For PBD simulations involving fluid, the Velocity parameter affects the flow rate of the liquid; as its value increases, the liquid flows faster. The Cohesion parameter affects both the shape and flow rate of the liquid; lower values, the liquid falls quickly and splashes out. As the value increases, the liquid flow slows down and splashing decreases, eventually leading to smooth flow. The Particle Contact Offset parameter affects the form of the liquid falls; as its value increases, the liquid transitions from continuous stream to a segmented, chunk-like flow.
In simulations involving deformable bodies, the Vertex Velocity Damping parameter affects the fall speed of objects such as hats; as the value increases, the fall speed decreases gradually. The Settling Threshold parameter also influences the fall speed of hats; increasing its value results in slower fall speed, but once the value exceeds 1, the fall speed stabilizes. The Elasticity Damping parameter impacts the shape of the hat; as the value increases, the hat gradually collapses from firm structure to a flat plane. The Young's Modulus parameter also affects the shape of the hat; at lower values (around Ie3), the hat collapses into a smaller height. As the value increases, the hat becomes firmer, and when the value reaches around 1e4, the hat initially stays firm and then gradually collapses. At value of 15000, the hat remains completely firm.
For rigid body simulations, the Max Linear Velocity parameter affects the fall speed of rigid bodies such hats; as the value increases, the fall speed decreases. When the value exceeds 50, the object practically stops falling.
In the context of flow simulations, the X-Component, Y-Component, and Z- Z-Component parameters together determine the direction of the wind vector, while the Magnitude parameter determines the strength of the wind.
22

---

Table 7: Multi-physics parameters
Type
Parameiers
Function
Range
Particle Contact Offset
Di stannce at which particles start interacting
0.03 0.03-0.72
Contact Offset
Distance at which collisions detected
0- 16384
Rest Ofliet
Distance al which particles are in ressing contact
0. 0:05
Solid Rest Oflset
Distance for particle-solid interactions
Default
Fluid Rest Offsct
Distance for particle-fuid intoractions
Detault
Solver Position Iteration Counnt
Numher of lterations for solver to sanssy constraints
6 255
Depenetration Velocity
Maximum speed which particles are soparated when overlapping
Max Neighborhood
Maximam number of neighhoring particles for interactions
36-512
Garment
Densily
Mass per unit volume of the material
detanlt
Friction
Resislance to shiling motion
delault
Dampiing
Reduction of motion or oscillations
detault
Viscosity
Internal Friction within the fluid material
default
Cohesion
Attructive force between particles
default
Surface Tension
Elastic lenlency
matienal's surface
detauli
Drag
Resistance experieaced when moving through dldd or air
detauli
Lift
Forez acting perpendicular fluid flow around the matoril
dafault
Partiele Contact Offset
Distance at which particles start interacting
0.17 1 0,3
Contact Offsct
Distance which collisions
dotected
dafault
Rest Offset
Distance at which particles are in resting contact
0.03 128-00 0.2
Solid Rest Ollset
Distarcce purticle sobi interactions
0.1.0.2
Fluid Rest Ollset
Distance for particle interactions
0.1.0.13
Solver Position lierstion Count
Number of Reriiionn Tor solver to satisty constraints
6-255
Mix Depenetration Velocity
Maximum speed at which particless are separated when overlapping
Fluid
Max Neighborhoood
Maximum number of neighboring particles for interactions
36 512
Densily
Mass per unit volume the material
0- 1e10
Friction
Resistance to sliding motion
0.2
Damping
Reduciton motion or ovcillations
U- 10
Viscosity
Internal Friction within the Hldid miterial
103 1e6
Colossion
Aftractive foros between particles
0- 100
Surface Tension
Flastic tendency of material's surface
100
Drag
Ressstance experienced when moving through dudd air
0- 78
Liit
Force acting perpendicular lluid
around the materal
Te10
Vertex Velocity Dumping
Rale ol reduction of vertex velocities
0 10
Simulation Mesh Resolution
Granulaniy of the simolation mesh
Solver Position lecrations
Number of flerations for solver to satiaty positional constraints
8- 235
Sleep Threshold
Velocity below which the body considered to be
lo?
Settling Threshold
Velocity below which the bodly is considered to have setiled
O. le?
Sleep Danging
Additional damping as the body approuches The sleep ihreshold
U le7
Contact Offset
Distance at which collisions are detected
-In
Rest Offsct
Distance at which particles ars in resting contact
-inf
Collisio Filter Distance
Minimum distance to avoid celf.collision
Deformable Body
Remeshing Resolution
Resolation for remeshing the impul mesh
Detauli
Target Triangle Count
Target resolution for quadric simplification
Default inf Default
Max Depanetration Velocity
Maximum speed at which vertiees can be separated when overlapping
Density
Mass per unit volume
Dynammic Friction
Resisiance to shiding motion
D- 2048
Young's 's Modulus
Stiffness the material
1e3 1e10
Poisson's 5 Ratio
Rabo transverse to axial strain
0.499
Musiicit Damping
Redoction of oscillations and vibrations
U- 0.08
Damping Scale
Adjusis the overall damping effect
U- 1.0
Max Lincar Velacity
The rale of change of position of the rigid body.
0-30
Max Angular Velocity
rate change of rotation
rigid body.
U-Ic10
Collision Shape
Defines the shape used for collision detection.
default
Contact Offset
Distance from the surface where collisions detected.
inf
Rest Offist
Ellective contact thstance from the surface.
inl inl
Convex Hull
Approsimsiion mellood For collision shape.
0-64
Rigid Body
SDF (Signed Distance Field)
Approximation method using signod distance hold.
default
Mass
Defines the mass of the ripid hody.
0 to Inf
Densily
Delines the density of the rigid body material.
to 1000
Priction
Resistance to sliding motion
U
Restittion (Bounciness)
Degree of clasticity of collisions.
Material Density
Density of the material applicd to the rigid body.
to 1000
X Component
Flow the direction
inf inl
Flow
Z-C Component
Y-Compooent
Flow rale the y-direction
Flow rale the 2-direction
-iaf-iaf
inl-ial
Magnitide
Overall magnitude flow
0-inf
E
Sim2Real
Transferring models trained in simulator to reality is challenging and become a critical issue for robotic eesearch.Howevee most Sim2Real techniques are not yet fully automated and require careful human oversight. In this work, we present three visual sim2real methods which are fully automated and self-supervised. We mainly conduct experiment follow [57], learning dense visual representation for garment before and after our alignment method. The results are shown in Figure
23

---

Rallpriiee
Ead othe
1000
solid- solid
solid- fluid
fluid- fluid
Figure 8: Different Types of Particle-Particle Interaction
E.1 Sim-Real Vision Alignment
Noisy Observation. Although previous do dedicated exploration on how to add noise to point 226 62). they need capture IR picture and do many calculation which time-consuming, However, we have found that simply adding salt-and-pepper noise and Gaussian noise can already yield very good results. We directly add noise depth picture and generate noised point cloud in the training data.
D(x,y) y) + -N(0, o2) with probability Pgaussion
D(r, y) + salt
with probability Psalt
Dacised (x, v)
D(x, y) pepper
with probability Ppepper
D(r,y)
otherwise
where D(r, y) represents the depth value pixel (a, y) in the original depth picture, /N0 o²) represents Gaussian noise with mean and variance o², and salt and pepper noise added with probabilities Pgaussian- Psalt, and Ppepper respectively.
For experiment, we add noise to the training data during the training process of dense visual correspondence(5)) As shown in figure before we do data argumentation for training data, the query results show many spots, indicating errors. This is due discontinuities of dense representation caused by differences between the real-world point cloud and the simulator. After we add noise, the model become more robust to noise thus the representation become more smooth and accurate.
Point Cloud Alignment. Dense object descriptors [4] that learn point- or pixel-level object representations proposed by and for robotic manipulation. The key idea of these works 57] 54 50Jis to represent an object as function that maps a 3D coordinate x to a spatial descriptor point cloud
R - R". f may further be conditioned P E R N and usually parameterized by neural network. However, f are not always SE(3)- equivariant, which means
f(r) of that 3D coordinate:f
rigid transform (R, E FF3), we NOT guarantee that [xx f(Rx t/RP + t). However, real world experiment, as the height and angle of the camera may be different from that in the simulator, the distributions of point cloud collected in simulation and real world are different. This will lead to wrong query result especially for garment as it highly rely on thickness detect folding relationships.
network 13 the training of it is hard and time- cone-consuming. Thus, propose direct way to align the point cloud in simulation and the realworld. As all rigid transform (R, t) E SE(3) can be represented by affine matrix, we directly use gradient descent to optimize the affine matrix that simulation point cloud. We chose the chamfer distance as the loss function because it is both robust the various deform and shape of the garment and effectively aligns the positions. Equation 7 show our optimization objective and 2 show our loss function.
Although we can choose S0(3)-equivare)
align the distribution between realworld point cloud and
a
b
Transforms R,t)
d
e
f
y
(1)
9
h
0
0
0
1/

---

Chamfer Loomsfer_Lss(AAee A, B)
min Σ REA
B a
bil
1
min
||a aEA
bl
(2)
|A|
B
BEB
As shown in figure 6 before we align the point cloud, the model predict wrong result even the flat case. After alignment, the model successfully predict the correct result.
Keypoint Embedding alignment. As the model learn point level representation, a direct way is to align the representation of corresponding point between realworld garment and simulation. In this part, we first attach marker to garment on the skeleton points(shoulder, end the sleeve and bottom corner etc.) Then, we do self-play which enable franka to randomly choose the pick-and-place point to create garment deformation status. Then we use SAM[24] to detect key point and align correspondence key point with the simulation result. After we get ground-truth key point pair between simulation and realworld,w, we employ InfoNCE[25 widely-used loss function in pree-pssitve-miee multi- negative-pas pair contrastive representation learning. pull close the representation corresponding points, while push away representation of them and other point representations. The loss function is shown in Equation 3
LCD -log( explyp Sp//T) Σ. exp(fp fx//T)
where fp is the skeleton point in realworld garment, for is the corresponding point in simulation point cloud and fyt is other point in simulation point cloud.
As shown in figure [ after model finetune, the performance of the model on realworld garment improve significantly. This is mainly because model is more adapted to the distribution of real world point cloud.
F Real-Worke Benchmark
Benchmarking and performance evaluation in robotic manipulation encounter challenges owing to the diverse range of applications and tasks, prompting research groups to select representative tasks and objects that are frequently inadequately specified and inaccessible to others, thereby impeding the ability to compare experimental results and interpret performance quantitatively, particularly in real-world scenarios. To address this issue, the implementation of a real-world benchmark is erucial, as it can not only narrow sim2real Gap but also provide platform for researchers directly compare algorithm performance. Although previous work has introduced real-world benchmarks, such as YCB (5) and furniture benchmark [18] they primarily focus rigid bodies, lacking benchmarks specifically designed for deformable objects. In this study, we introduce the first real-world benchmark for deformable objects and garments, facilitating the widespread usage of a standardized set of objects and tasks to enable easy comparison of results among research groups worldwide.
F. Object and Data Set: Object Selection
Principle
We aimed to select objects that are frequently used in daily life, and we also reviewed the literature to consider objects that frequently used in simulations and experiments. Several additional practical factors must be considered when formulating the proposed set goals and tasks.
Variety
The objects included are small in number, but ensure a great richness. Judging from the category of items, we roughly include tops, pants, skirts, socks, gloves, dolls, etc. Considering size, generally speaking, clothes occupy larger area, followed by dolls, and then small items such as socks and gloves. Considering deformability, large items of clothing are the softest, can be stacked into various shapes, and have the highest deformability, followed by small items, which can only undergo simple changes because they are relatively small, while dolls are elastic but lack deformability. Grasping and manipulation difficulty was also criterion: for instance, toys are well approximated by simple geometric shapes and relatively easy for grasp synthesis and execution, while garments have higher shape complexity and are more challenging for grasp synthesis and
25

---

execution. In addition, since we are doing benchmark about garments, we have to carefully consider their characteristics: they are diverse and highly deformable, but the same type of garment often only differs in texture or color, and is very similar in structure and key points. This allows us to use few objects to represent category of garments, thereby ensuring the variety of our benchmark.
Use
We included objects that are only interesting for grasping but that also have a wide range of manipulation uses. Soft and highly deformable clothing is also suitable for many complex operations: such as hanging, folding, etc. The introduction interaction between some objects and fluids, such as washing and air-drying. In addition, we also included people, which allowed us simulate the interaction between some objects and people, such as putting scarf someone. As mentioned above, these tasks are intended to span wide range difficulty, from relatively easy to very difficult.
fluid allows us to simulate the
Durability
We aimed for objects that can be useful in the long term, and, therefore, avoid objects that are fragile or perishable. In addition, to mcreasee the longevity of the object set, we chose objects that are likely to remain in circulation and change relatively little in the near future.
Cost
We aimed keep the cost of the object low as possible to broaden accessibility. We, therefore, objects, and tests.
selected standard consumer products, rather than, for instance, We buy all our clothes from Uniglo and all our dolls from Disney.
After these considerations, the final objects were selected. You can see our object set in Table and the corresponding suggested tasks in Table 9
Table 9: Suggested Manipulation Tasks for Different Garments.
Object Category
Suggested Tasks
Washing and Drying
Togs/PPats/SSonn//ettt/Siit
Folding
Stacking and Grabbing of clothes
a. Washing and Drying
b. Folding
Dresses/Coats
c. Stacking and Grabbing of clothes
d. Hanging
Washing and Drying
Folding
Stacking and Grabbing of clothes
Hats/Scarf's
Hanging
Interacting with People: wearing
corresponding objects on people
Deformable Objects
Grabbing and Placing
a. Grabbing and Placing
Small Items (Gloves/Stocks)
Washing and Drying
F.2 Object Scans
Our scanning process is roughly divided into four stages: model wearing clothes, scanner scanning to obtain raw data, post-proces post-processing, and manual annotation of key points. The whole process can be referred to Figure 9
26

---

Table &: Real-Worid Objects and Properties. PBD stands for Position Base Dynamies, while FEM stands for Finite Element Method.
Number
Class
Name
Uhyec
Picture
Mahhel
Branch
Number
Class
Dijjet
Name
Picture
Sindliin Mathal
Branch
Dincy
Vniglo
Bear
Olgeetss
Shorts
Shems
Duney
Unigio
Coats
laclkct
PRD
Uniiglo
Vonte
Conton Vest
PRD
Unigle
Couss
Ginzza Jacket
PBD
Uniglo
15
Shorts
W White Camisole
FSD
Uaigio
Unigho
Unigho
Whitee loog-
White bllne
Tops
PHD
Unigle
Shorts
Unigilo
Whitee eloowaal
Topss
Uniggo
Objects
Unigho
Unigho
Unigio
10
Pazis
Black
PED
Uniglo
FBM Objects
Yellow
FEM
Unigio
27

---

Model Wearing Clothes
Post- Processing
Manual Annotation of Key Points
Scan
Processing Remaining loles
Adding Texture Files
Down-Sampling
Figure 9: Objects Scanning Process.
Model wearing clothes
There are two main ways to scan clothes: scanning them flat and scanning them while the model is wearing them. After careful consideration and constant experimentation, we chose the latter. This is because
large number of points simulate clothing, the wrinkles formed when the clothes are laid flat will cause the points to be unevenly distributed, thus forming many "holes". While when worn the model, the appearance of wrinkles will be reduced, thus try to avoid this situation as much as possible.
Scanning
We a
the object from multiple angles, and obtain copy of the original point cloud data and grid data through the combination of the depth camera and the RGB camera. The general process can be referred Figure
Post- Processccss
During the post-processing process, we mainly did three things: down-sampling, adding texture files, and processing remaining holes. The point set obtained by the initial sean has too many points and is difficult to support with ordinary computing power, so we performed down-sampling to generate file that can retain the main features and have a moderate number of points. We use Meshlab for down-sampling faces. After down-sampling, it can reach about 10,000 points and 30,000 faces. We use MTL files (Material Library File) to add material attribute information. MTL is material library file used describe the material information of objects. It is usually used in conjunction with an OBJ file apply material properties such texture and color to the OBJ model. In this step, implemented some visual textures, such as patterns, colors, etc. The physical texture is achieved through different simulation methods. In addition, there still some "holes" in the processed point set, which we repaired manually.
The original scale is about 100 million points and 300 million
mainly
Manual annotation of key points
Since we use large number of points to simulate objects, it is necessary to mark some key points and edges to indicate important features. For example, for tops, we will mark the sleeves, neckline, hem, etc. These locations are often the key parts for clothing operations. For dolls, we will mark arms, legs and other parts that are easy to grasp. Figure 10 gives some examples for reference.
Manual Annotation of Key Points
Pooh
Notc: The red points marked points
Figure 10: Manual annotation of key points: examples
28

---

F.3 Protocol Benchmark Guidelines
We use the protocol and benchmark templates mentioned in [72]. To both provide more concrete samples of the types of task definitions that can be put forward as well as specific and useful benchmarks for actually quantifying performance, we have developed some example protocols: clothes-hanging protocol, scarf- wearing protocol.
Clothes-hangine protocol and benchmark
When dealing with flexible clothing, hanging clothing is always popular task. The protocol uses the hanger, clothes suitable for hanging of our model set. The clothes are initially laid flat on the platform, and the robot is expected to grab the key points of the clothes, pick them up, and finally hang them on the hangers. The benchmark scores the performance of the robot by evaluating whether the hanging point is reasonable and the stability of the hanging clothes (whether they are easy to fall off). We applied this benchmark to Franka.
Scarf-wearing protocol
Dressing people with robots has always been a difficult subject. In this example, we chose the casier and more manageable task: wearing a scarf. This protocol uses the scarf from the model set, and person from the Issac Simulator. The scarf is initially laid flat on the platform, and what the robot has to do is to pick it up and wrap it around the person's 's neck, and finally adjust it to suitable state. The benchmark scores the performance of the robot by evaluating whether the final state of the scarf is stable (we test it by adding wind to see if the scarf will blow off quickly), whether the remaining length of the scarf on both sides is similar, and whether the scarf fits the person's 's neck (rather than loosely packed). We applied this benchmark Franka.
G Task
G.1 Task category
Carment-Garment This category focuses on fundamental garment manipulation, including tasks like folding and unfolding single garments, as well interactions between multiple garments such as retrieving items from clothes piles. Tasks this category include folding, unfolding (pick and place), and unfolding (fling).
Garment-Fhuid Tasks in this group concentrate on the interaction between garments and fluids as well as flow, where trajectory dynamics play crucial role. This category of tasks includes washing clothes in basin, rinsing clothes under running water, and drying clothes with a hairdryer. this category, specifically introduced the interaction between the robot manipulating objects and fluid flow, including both water flow and air flow.
Garment-FEMObjjer FEMObjects. We mainly focus on the exploration of tasks involving deformable interac- tions, such as using a sponge to clean dirt off clothes packing hats and tops together. Some simple tasks involving the manipulation of deformable objects are also included, such as using dexterous hand or gripper to grab plush toys.
Garment- Rigid. Common interactions between clothing and rigid bodies, such as hanging clothes or putting them into washing machine, require precise grasp point selection and trajectory planning. We also introduced articulated objects such as cabinet drawers and clips to perform garment-related tasks, such taking clothes of wardrobe.
Garment-Avatar Dressing tasks pose the greatest challenge, as they require understanding of human intention and safe collaboration with humans. Some representative tasks include putting scarf person and placing a hat on their head. More advanced tasks involve dressing a person in a jacket or T-shirt.
G.2 Long-horizon task
Organizing clothes. This task comprises several stages, including retrieving tops or trousers from a clothes pile, unfolding them using fling, folding the clothes, and placing them in the wardrobe.
Wash elothes. This task involves several stages: retrieving hat from the cabinet, washing the hat in the basin, using a hairdryer to dry the hat, and placing the hat hanger.
29

---

Make up table. The task of setting the table involves several steps: firstly, retrieving the tablecloth from the box, laying flat, spreading it onto the table, smoothing out, and finally adjusting its position. Note here we need the use of mobile robot like mobile franka.
Dress up. This task involves putting scarf on someone, placing hat on someone's head, and dressing someone in a T-shirt.
H
MoveIt
We adopt Movelt, an open-souree state-of-the-arr robotic manipulation framework, to provide support for real-world trajectories planning and obstacle avoidance. To record the trajectories generated by Movelt and adapt visual models in the simulator the trajectories could bridge the sim2real gap to some extent. In this section, we introduce smooth, lightweight and responsive signal pipeline implementation to transfer real-world joint parameters the simulator.
As robotic manipulation platform, Movelt is built on top of ROS(Robot Operating System) and integrates with various ROS components. Movelt provides a series of comprehensive manipulation interfaces, including collision-free motion planning, kinematics computation, collision detection, etc. Moreover, real- world data collected from sensors like cameras and lidars can be fed into Movelt, allowing for dynamic obstacle avoidance. Once the trajectories are generated by Movelt, we publish the computed joint states through ROS, which transfers the trajectory to the Franka controller, FrankaPy. FrankaPy is a modular control stack that provides a customizable and accessible interface the Franka robot. Utilizing Movelt and FrankaPy, this pipeline enables Franka to devise collision- free path and guide the gripper to the target posisition using vision detectors, while publishing the joint parameters to the ROS server. The simulator then subscribes the joint states and moves the Franka model accordingly.
I Teleoperation
Teleoperation serves as a direct method to acquire human demonstrations for model training. To accurately and smoothly track human hand motions has been proved advantagcous in related works recently. However, the proliferated fine-grained tracking requirements, along with sparse and diverse dexterous hand models and environment settings, have posed challenge towards teleoperation systems. Compared to controller-basee models, we utilize the vision-based motion detection module, Leap Motion, efficiently record human hand poses and then retarget hand poses to the dexterous robot hand. More Formally, our teleopertaion systems can be described as below:
(i) Leap Motion hand pose detection module, which predicts the wrist position, the hand spatial direction and finger poses from the infrared camera stream.
(ii) retargeting module, which converts the wrist position and finger poses recorded by Leap Motion the arm end effector position and dexterous hand parameters.
iiii) motion generating module, which produces accurate, responsive and signals for the robot model that in the simulator and in the real world simultaneously.
In our work, we implement telcoperation for Universal Robot mounted with Shadow Hand and Franka. One control the posture of the robot by adjusting attitude and position of the hand over the detector. In particular, the open state of the gripper of Franka can be controlled by the opening and closing of the thumb and index finger.
L.1 Leap Motion Detection Module
The Leap Motion Controller is small USB device that can be placed on the desktop. Utilizing two 640x240 near-infrared cameras, it captures roughly hemishperical area in the distance of approximately 60 cm, typically at 120Hz. The internal algorithms then translate the received raw spatial data to 27 distinct hand elements, which includes the palm normal vector, the hand direction, the wrist position and 24 finger joint positions. The detailed hand elements are shown in the figure
12
30

---

finger poses
oplimization
Leap Motion detector
relargeling
hand joint params
wrist position palm directions
inverse kinematics
simulator real world move simultancously
trajectorics
Figure 11: Details of Our Teleoperation System.
Distal phalanges
Intermediate phalanges
Proximal phalanges
Metacarpals
O-length thumb metacarpal
Figure 12: The Hand Model Used in Leap Motion, each knuckle joints' spatial position is computed by retargeting algorithm and broadcasted though ROS Message.
1.2 Hand Pose Retargeting
The procedure of hand pose retargeting is two-fold: first, map knuckle positions to hand joint parameters; second, we compute trajectory to smoothly move the robot arm to the recorded wrist position and direction. The finger knuckle positions captured by Leap Motion cannot be directly fed into robot models due to the discrepancy between the dexterous hand joint angular parameters and the knuckle positions. The mapping algorithm that converts knuckle positions joint parameters is often formulated as an optimization problem, which can be described as
min Σ auu - f((1:)l2 + + 11q. q--1l
where de represents joint parameters of the dexterous robot hand time step f. fi the i-th forward kinematic function which takes the joint angles input and computes the knuckle positions. a is scaling factor to alleviate the size discrepancy between different human operators and the robot hand model. Additionally, observe adjacent N frames and add hyperparameter 3 improve temporal smoothness and consistency.
To compute the arm trajectory, we adopt slightly tweaked inverse kinematics approach, which is popular determine the joint parameters the trajectory, given the URDF file (Unified Robotics Description Format) of the robot and the desired configuration. URDF a file format that describes the physical properties and model of the robot, including joints, motors, articulation configurations, etc. In empirical experiments, we find that even slight hand movement or vibration can trigger significant and prolonged changes arm posture. To address this problem, we implement rate limit on target changing in neighboring frames.
31

---

It is notable that this workflow is applicable to a vast range of grasp- robots. Particularly, to control the open state of the gripper Franka, we measure the distance of the thumb and index finger and establish distance threshold in implementation.
I.3 Motion Generation
We integrate ROS for communication. ROS open-source framework that provides series of library which are designed for multi-robots scenarios. In our implementation, after the generation of hand pose and wrist position, the computed joint parameters are transferred through the ROS the simulator and the non-virtual robot simultaneously.
J Limitation
While this work adopts state-of-the-art simulation for different garments, there still exists the gap between the dynamics and kinematics of garments in simulation and the real world.
K
Broader Impact
This work paves solid way to future home-assistant robots in diverse garment tasks for industry. We haven't observed negative potential impacts.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
no theoretical result
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
As we claimed in abstract, we will release our code as soon as possible
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 5 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section initial settings.
We caleulate of the variance of scores of the policy in different
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Section
B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarr...c//bbbbb//hccccee
Question:
Answer: [Yes]
Justification: Section
K
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: no data or models that have high risk for misuse
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: A
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
no crowdsourcing experiments and research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: no experiments with potential risks for study participants
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "v1nfqEZ5KH", "reviewer": "Reviewer_wEM9", "summary": "This paper introduces GarmentLab, a new set of garment manipulation environments, assets, annotations and tasks based on IssacSim. The differences compared to existing benchmarks is the support of a broader range of tasks (such as tasks involving interaction with other objects such as hanger, fluid, and human avatar), more robot types, more assets, and other features such as motion planning, teleoperation, and sim2real transfer. A real-world garment manipulation benchmark is also proposed with the aim for evaluating real-world performances. Several existing algorithms are tested on the proposed tasks, and their performances are analyzed.", "questions": "Please see the weakness sections.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The overall system looks very comprehensive and cover most of the needs for garment manipulation, including a broad range of tasks, assets, and robot control methods. \n- The sim2real analysis and support for real-world data collection are nice features. \n- Overall the whole system looks impressive and can be a good contribution to the deformable object manipulation community.", "weaknesses": "I mainly have the following questions:\n- Regarding the real-world benchmarks: I appreciate the authors provide the 3d scans of the real objects and provide semantic annotations. However, I think the proposed real-world benchmark still lacks details such that every lab can easily replicate the set up in the real world. For example, it would be ideal to include purchase links for the chose objects. The protocols presented in F.3 seems to be just describing the task in a very high-level way, without details such as how the robot / objects should exactly be initialized, e.g., their relative position, and how to exactly measure the success / performance. Without such details it would not be possible for other labs to perfectly replicate this real-world setups and to form a fair comparison for different algorithms. I would encourage the authors to provide more details on such protocols to ensure that this can be well replicated in any lab, otherwise this proposed real-world benchmark would seem infeasible. \n- For some readers not familiar with the context, it is hard to understand the right part of figure 6. How can one interpret this figure to understand that the performance becomes better after applying the sim2real method such as point cloud align? E.g., what does the color scheme mean, and why is the color scheme after better than before? Also what is the figure in the dashed bounding box supposed to show? I would suggest to revise this figure to make these more clear.\n- Since this is a benchmarking and dataset paper, it is a little bit hard to access its quality and potential contribution to the community without actually using/seeing the code. For example, numerous robotics benchmarking has been proposed in the past, but only a few of them will be widely adopted by the community, where factor such as easy-to-use APIs, intuitive UI design, informative debugging tools, are all critical for the survival of the benchmarking. As the code of this paper is not released, it's hard to access the above qualities. I would encourage the author to release the code as soon as possible, and take into consideration the above factors, especially given the proposed system seems complicated and involve many different components. \n- An error in table 1: PyBullet actually support garments. See Antonova et al., Dynamic Environments with Deformable Objects, NeurIPS Datasets and Benchmarks Track, 2021. This paper is also highly relevant and should probably be discussed in related work. \n- There are some typos in the paper, e.g., line 6 in the abstract: \"which exhibit offer limited diversity\" -> \"which offer limited diversity\". Please have a thorough check over the whole paper for typos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We have addressed them below, and hope to hear back from you \nif you have further questions!\n\n# Setup of Real-World Evaluation\n\nThanks for the appreciation of our proposed real-world assets,\nand pointing out that the real-world evaluation setup is not detailed enough.\nWe have attached detailed descriptions (task, setup, hardware requirement, execution procedure, evaluation methods, etc.) about the real-world implementation protocols\nin the released document of GarmentLab. Specifically, the Protocol sub-section in the Real-World Benchmark section of the released document.\n\nThanks again for this valuable comment!\nWe hope that the real-world benchmark can really help the community to evaluate the real-world performance of garment manipulation methods.\n\n# Clarifications on Figure 6\n\nFor the **color scheme** of Figure 6, it denotes the per-point correspondence between garments.\nAccording to [1], when the learned representations of two points in two garments are similar, the point colors would be similar.\nSuch per-point correspondence is a good way to both (1) evaluate and visualize the learned representation of garments in an intuitive way,\nand (2) further guide the manipulation of novel garments within few-shot demonstrations in previous garments, by selecting the manipulation points on the novel garment, which are the most similar to those in the demonstrations garments.\nMore details of such representation can be found in [1].\nWhen adding sim-to-real methods (Noise Injection for Domain Randomization, Corresponding Keypoint Representation Alignment, Point Cloud Alignment between Simulation Garments and Real-World Scans),\nthe learned representations, reflected by the color scheme, can be more continuous and smooth across different point on the different garments.\n\nWe have revised Figure 6 (as Figure 1 in the \"Attached Rebuttal PDF\"), \nshowing the procedures of three sim-to-real methods on the left of the green block, and there corresponding effects on the learned representations of garments (making the learned representations more continuous and smooth, with better correspondence across different garments) on the right of the green block.\n\nThanks again for this valuable comment! We are sparing no effort to make our paper more clear.\n\n## Details of Code and Usage\n\nAs described in the \"Global Response\", we have made GarmentLab public, with the documents of installation and usage. Welcome to have a try!\n\nWe are very grateful for your tips, and we have taken the details you mentioned into account when designing the benchmark. \nFor **UI design**, we adopted the Isaac Sim UI interface. As a mature, industrial-grade simulator, Isaac Sim's UI is easy to operate, as verified by practical tests, making it much more user-friendly than most other simulators' basic interfaces. \nMoreover, it supports moving with the mouse during operation, which is something most simulators [2, 3] cannot do.\nAdditionally,\nthe functions that can be achieved in the UI interface can easily be extended to **code APIs**,\nand we have provided detailed instructions on how to use the APIs in the code and documentation,\nwith many example tasks to control the robot and achieve the goal.\nFor **debugging tools**, Isaac Sim and Omniverse already have a very comprehensive warning and error reporting system. \nOn this basis, we have added numerous assertions at key points in our code to ensure that user input meets the requirements and to ensure that users correctly understand how to use the API.\nIt is worth noting that, as Isaac Sim is still under active development,\nit sometimes encounters segmentation faults without detailed descriptions. \nWe have included more detailed error or warning statements in our high-level code. \nThis way, users can quickly locate the problem when an error occurs.\n\n\n# Descriptions on PyBullet\n\nThank you for pointing out that [4] based on PyBullet also supports the simulation of garment manipulation tasks.\nCompared to Pybullet based methods, our proposed GarmentLab has the following advantages:\n\n1. GarmentLab supports the coupling of garments simulated by different simulation method (PBD and FEB), and the interactions between garments with different materials (e.g., hat, gloves and shirts). \nOn the contrary, previous works with garments simulated in PyBullet could only support one kind of simulation, without the simulation of multi-material interactions.\n2. GarmentLab includes GPU acceleration for simulation in the large scale, which supports collecting large scale data and training in an efficient manner.\n3. Equipped with high fidelity GPU based PhysX engine, GarmentLab is capable of supporting multi-sensor RTX rendering at an industrial scale.\n4. While [4] only demonstrated 4 representative tasks on only a few categories of garments, we present and benchmark 11 garment categories with much more diverse tasks. \n\nWe will include the above descriptions in the final version of the paper.\n\n\n# Typos\nThanks a lot for pointing out this. We have thoroughly checked our paper, and corrected them.\n\n\n\n[1] Wu, et al. UniGarmentManip: A Unified Framework for Category-Level GarmentManipulation via Dense Visual Correspondence. CVPR 2024.\n\n[2] Lin, et al. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. CoRL, 2021.\n\n[3] Zhou, et al. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. ICLR, 2023.\n\n[4] Antonova, et al. Dynamic Environments with Deformable Objects, NeurIPS Datasets and Benchmarks Track, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuhSIDfJXL", "reviewer": "Reviewer_E1Nw", "summary": "This paper proposes GarmentLab, a content-rich benchmark and realistic simulation specifically designed for deformable objects and garment operations. It encompasses a diverse range of garment types, robotic systems, and manipulators.", "questions": "See weaknesses.", "limitations": "The paper mentions that the limitation is the sim2real gap, perhaps with the development of simulation engines, this problem will be alleviated.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The GarmentLab Environment offers a realistic and comprehensive setting for garment manipulation. Additionally, the GarmentLab Benchmark covers a wide range of garment manipulation tasks and serves as the first real-world benchmark for these activities.\n2. Although this job is primarily focused on engineering, it makes a certain contribution to the development of the garment manipulation community.", "weaknesses": "1. The paper mentions a characteristic of  GarmentLab is efficiency (Line 59), however, there are no quantifiable comparison with other enviroment in the speed and GPU memory during training/testing.\n2. The modal of the GarmentLab environment is limited to visual (RGB-D), and other modalities such as touch and hearing can also be helpful when operating garment objects. Supporting this will help further development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We have addressed them below, and hope to hear back from you \nif you have further questions!\n\n# Efficiency of the Benchmark\nTo the best of our knowledge, GarmentLab is the first to support all of ray tracing, robot control simulation, and diverse deformable object simulation methods in a whole environment.\nTherefore, it is a little bit difficult to directly compare the speed and memory of our environment with those of other environments in a fair setting.\nFor example, SoftGym [1] and PlasticineLab [2] do not support real robot control (they only support pickers or manipulators as robots), which requires solving inverse kinematics and controlling the robot.\nFluidLab [3] does not support real time tracing, which is important for realistic visual effects.\nFurthermore, the above environments only support a single type of deformable object simulation method.\n\nWhile our environment supports all of the above features,\nwe also take the speed and efficiency of the environment into account.\nGarmentLab is GPU-based, placing all simulations of deformable objects and real-time tracing on the GPU, while handling robot-related inverse kinematics on the CPU. This improves the simulator's speed and avoids the bottleneck of CPU-GPU data transfer.\nMoreover, to improve data collection performance, we support importing multiple robot-object interaction scenarios in the same simulation environment for parallel data collection.\nPlease refer to 2:37 to 2:41 in the **Supplementary Video** for the demonstrations of simultaneous multi-scenario data collection.\n\nWe have designed two settings to compare the speed of GarmentLab (with and without the multi-scenario data collection method respectively denoted as GarmentLab-Multiple and GarmentLab-Single) and SoftGym.\nFirst, we evaluate the speed of the robot reaching the target garment and then grasping the garment.\nSecond, we evaluate the speed of the robot directly grasping the garment in the near front without other movement.\nThe following tables respectively show the speed comparisons in these two settings,\ndemonstrating that GarmentLab, with the designed multi-scenario data collection method, is comparable and a little more efficient than SoftGym in terms of simulation for data collection.\n\n| Environment | Data Collection Speed |\n|---------------------|-----------------------|\n| GarmentLab-Single            | 30 samples/hour        |\n| GarmentLab-Multiple             | 160 samples/hour       |\n| SoftGym             | 140 samples/hour     |\n\n\n\n| Environment | Data Collection Speed |\n|---------------------|-----------------------|\n| GarmentLab-Single            | 180 samples/hour      |\n| GarmentLab-Multiple             | 1200 samples/hour     |\n| SoftGym             | 900 samples/hour     |\n\nThanks again for this valuable comment! This is really significant for a robotic manipulation environment.\nWe will add the above descriptions, comparisons and tables in the final version of our paper.\n\n\n[1] Lin, et al. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. CoRL, 2021.\n\n[2] Zhou, et al. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. ICLR, 2023.\n\n[3] Huang, et al. PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics, ICLR, 2021.\n\n\n\n\n# More Input Modalities\nThanks for this valuable comment!\nThe current engine supports tactile detection for rigid-rigid interaction.\nAs demonstrated in Figure 4 (Right) in the \"Attached Rebuttal PDF\",\nwe replaced the Franka gripper with the Trossen's X300 gripper and installed tactile sensors on its inner side,\nand can detect the tactile feedback including normal and friction points and forces between the robot manipulator and the rigid object using SDF collision.\nWith this feature, we are able to design interesting tasks like using a rigid stick to nudge and adjust the posture of garments. The clothing provides force feedback to the stick, which is then perceived by the tactile sensors attached to the manipulator. \nFurthermore, we are actively extending and developing this feature into directly sensing the tactile feedback of garments,\nwhich could largely support more interesting and fine-grained manipulation tasks,\nand we are willing to present such features in our future work, to continuously make contributions to the community.\n\nFor other modalities, the engine also supports IMU, Lidar, Radar, Ultrasonic and Generic Range sensors, while these sensors are rarely used in garment manipulation tasks.\nIf some tasks require such modalities, we are super willing to extend such sensory signals to our proposed envrionment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0VR3rWU95o", "reviewer": "Reviewer_1QRi", "summary": "This paper proposes a content-rich benchmark and realistic simulation for deformation object and garment manipulation. The GarmentLab consists of the GarmentLab Engine, Assets, and Benchmark, supporting various research topics.", "questions": "1. Will all the 3 components, i.e. the Engine, Assets and Benchmark, be publicly available?\n2. As for the physics engine, is it possible to simulate the interactions among different layers of garments? And will different layers of garments really affect each other instead of generating dynamics layer by layer? For example, a human wearing multiple layers of garments doing customized actions.\n3. What about the interactions between garments and fluid? Will the garment dynamics also be affected through the interactions with fluid? Some simulation engines do not support the real interactions between garments and fluid. They may generate the garment dynamics first, fix the dynamic sequence, and then simulate the fluid. Can the proposed engine solve this issue?\n4. Is it possible to generate customised dynamics using the proposed physics engine? For example, using customised garments and interacting with other deformable objects.", "limitations": "Please refer to the questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The GarmentLab provides a realistic and rich environment for research topics related to garments.\n* The engine supports different interactions between garments and other objects.\n* The author further benchmarks various garment manipulation tasks based on this environment.\n* Different sim2real methods are integrated into the GarmentLab.", "weaknesses": "Please refer to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We have addressed them below, and hope to hear back from you \nif you have further questions!\n\n## Code Release\n\nThese mentioned components with the guidelines for installation and usage have been already made publicly available. Please refer to the **Global Response** for more details. Welcome to have a try!\n\n## Interactions among Multiple Garment Layers\nGarmentLab supports simulating the interactions among different layers of garments, and these layers really affect each other.\nWe use position-based dynamics (PBD) [1] to simulate garments with multiple layers as a whole to achieve this feature.\nAs demonstrated in Figure 2 in the **Attached Rebuttal PDF**, \nthe garment consists of multiple layers,\nand when the robot is trying to grasp and lift the front layer (from the left sub-figure to the right sub-figure),\nother layers will have the effects resulted from the robot's operation on the grasped front layer.\n\n\n## Interactions between Garments and Fluid\nGarmentLab supports simulating interactions between garments and fluid, and generating the dynamic sequences at the same time. \nThe engine simulates garments and fluid using the same simulation method, position-based dynamics (PBD) [1], and thus supports this feature.\nAs demonstrated in Figure 3 in the **Attached Rebuttal PDF**,\nin the first four sub-figures, \nwe adjust the weight of the garment particles to make the garment respectively floats on the water, suspends in the water, and sinks to the bottom. \nThis interaction requires importing both garment particles and fluid particles into the same particle system, and then generating dynamic sequences simultaneously.\nIn the first four images, we adjust the weight of the clothing particles to make the clothes float on the water, suspend in the water, and sink to the bottom.\nBesides, we can also simulate clothing and fluid in different particle systems separately. As shown in the last sub-figure, the garment and fluid do not have interactions in this case.\nAdditionally, we can adjust simulation parameters, such as particle contact offset and fluid rest offset, to simulate different interaction effects between garments and fluid. You can refer to Section D in **Appendix** for more details.\n\n\n## Customized Dynamics\nGarmentLab supports customized dynamics. We can customize the dynamics in two ways. The **first** way to use customized meshes of garments or deformable objects. For example, in the left of Figure 4 in the **Attached Rebuttal PDF**, we import\na garment mesh from our simulation assets and a deformable object mesh from our real-world benchmark. These two objects use different simulation methods: the garment uses Position-Based Dynamics (PBD), while the deformable object uses the Finite Element Method (FEM)[2].\nWe can simulate the interactions between these two objects. This demonstrates that we can combine simulation objects and scanned objects from the real world.\nThe **second** way is to adjust the parameters of the physical simulation to change the effect of object simulation. You can refer to Figure 3 in the **Main Paper** to see the effects of physics and Section D in **Appendix** for the physical parameters that can be adjusted.\nIn summary, our GarmentLab is very flexible. It supports customized meshes of deformable objects and garments, and also allows for adjusting object parameters to modify the physical simulation effects.\n\n\n[1] Bender, et al. Position-Based Simulation Methods in Computer Graphics. Eurographics (tutorials). 2015.\n\n[2] Ciarlet, et al. The finite element method for elliptic problems. Society for Industrial and Applied Mathematics, 2002.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5SLDqjrsl", "reviewer": "Reviewer_6kGg", "summary": "The authors present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Multiple tasks such as interactions between garments, deformable objects, rigid bodies, fluids, and human body are explored. A first real-world deformable benchmark along with several sim2real methods is also presented in this paper. The integration of ROS and MoveIt is also supported.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework is multi-functional supporting simulation of various interactions and coupling for garment manipulation scenarios. In addition to simulation, several sim-2-real algorithms such as Teleoperation, Visual Sim-Real Alignment are provided. The integration with ROS and MoveIt also help the framework to be a off-the-shelf tool for the community to use.\n- Extensive evaluations and experiments are presented.\n- The paper is well-written and organized.", "weaknesses": "The main contributions of this paper lie in the engineering and benchmarking (or dataset) aspects. Although I acknowledge the substantial contribution to the community, the novelty of this work remains a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We have addressed your comments below. \n\n\n\n## Novelty of Our Paper\n\nWe are grateful to your review on our **contribution**: \"the substantial contribution to the community\".\n\nAs for the **novelty** of our paper,\nas our paper proposes the environment and benchmark for robotic manipulation,\nthe novelties mostly lie in the comparisons with other environments and benchmarks.\n\nFirst, garment manipulation is a novel research direction. \nWhile previous studies mostly provide environments for rigid objects, or relatively simple deformable objects (e.g., cables, rings and fabrics), our work first comprehensively provides the simulation and benchmark for the novel garment manipulation direction.\n\nSecond, the diversity of objects, interactions and tasks is our novelty.\nWhile previous environments mostly only support the simulation of a certain type of object or interaction,\nour work not only supports simulation for rigid and articulated objects,\nbut also supports different simulation methods for different deformable objects.\nFor example, Position-Based Dynamics (PBD) [1] for shirts, Finite Element Method (FEM) [2] for hats and gloves, and even the simulation of fluid and flow.\nFor these diverse object simulation methods, we further support the interaction coupling among them.\nThe simulation of these diverse objects and their interactions provides a rich environment for various tasks,\nwhich has not been achieved in previous benchmarks.\n\nAdditionally,\nwhile previous benchmarks mostly focus on evaluation in simulation,\nour work provides a systematic sim-to-real pipeline and the real-world benchmark, paving way to the real-world deployment, comparison and benchmarks.\n\n\n[1] Bender, et al. \"Position-Based Simulation Methods in Computer Graphics.\" Eurographics (tutorials). 2015.\n\n[2] Ciarlet, et al. The finite element method for elliptic problems. Society for Industrial and Applied Mathematics, 2002.\n\n\nWe hope that we have clarified our novelty well. We are happy to take further questions from you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our gratitude to reviewers for their careful reading, meticulous feedback and valuable insights! 

We are glad that reviewers unanimously agree that our work:

(1) proposes a comprehensive and realistic environment for garment manipulation (wEM9: "very comprehensive, cover most of the needs", E1Nw: "comprehensive, realistic, content-rich; a wide range of tasks", 1QRi:"realistic and rich", 6kGg: "multi-functional, various interactions and coupling");

(2) designs well for sim-to-real transfer (wEM9: "sim2real analysis and support for real-world data collection are nice features", E1Nw: "first real-world benchmark", 1QRi: "different sim2real methods are integrated", 6kGg: "several sim-2-real methods are provided");

(3) makes good contributions to the community (wEM9: "a good contribution to the deformable object manipulation community", E1NW: "makes a certain contribution to the development of the garment manipulation community", 6kGg: "substantial contribution to the community").

In this **Global Response TEXT**, we clarify some common concerns raised by reviewers. Thanks again for valuable comments! Any further questions are welcomed!

## Code Release (Reviewer wEM9 and 1QRi)

We have made GarmentLab publicly available, and sent the corresponding anonymous link to the Area Chair.
The open-sourced components include：
+ Instructions for installation of the environment and benchmark based on the Engine
+ All simulation assets (scenes, robots, garments, scenes and other objects)
+ Real-World Benchmark: scans and purchase links of real-world garments, setup of real-world manipulation
+ Code for loading robots, garments and scenes
+ Code for robot control
+ Code for more than 10 benchmark tasks
+ Code for teleoperation
+ Code for customizing the physics of the simulated garments
+ Code for data collection and learning baselines (point-level correspondence and affordance for manipulation)
+ Documents (User Manual) of how use the environment, including detailed descriptions of different components and corresponding demos

Welcome to have a try on GarmentLab! We are making every effort to create a user-friendly, high-quality and powerful benchmark.

Furthermore, we are also actively developing the environment to support more robots, tasks, sensors and modalities. We hope to do our best to contribute high-quality, easy-to-use garment manipulation environment and benchmark to the community.

For the remaining questions, we will respond to each reviewer individually. We appreciate the valuable and insightful comments from reviewers! If you have any further questions or suggestions, please feel free to contact us. We are more than happy to discuss with you!

Meta-review: This submission introduces GarmentLab, a comprehensive benchmark and realistic simulation environment for garment manipulation, featuring diverse tasks, multiple simulation methods, and sim-to-real algorithms, designed to advance research in deformable object manipulation for home-assistant robots. The system is based on NVIDIA IssacSim using FEM and PBD methods. Codes are submitted and planned to be released. All the four reviews vote to accept the submission. The AC sees the submission as a clear acceptance.
Decision: Accept (poster)