DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy
Authors: Yuran Wang
Ruihai Wu
Yue Chen
Jiarui Wang
Jiaqi Liang
Ziyu Zhu
Haoran Geng
Jitendra Malik
Pieter Abbeel
Hao Dong
Venue: NeurIPS 2025 spotlight ()
Keywords: Garment Manipulation; Datasets and Benckmarks; Dexterous Hands Manipulation; Automated Data Collection; Imitation Learning;
Primary area: applications

Abstract: Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural correspondence to automatically generate a dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO). It first identifies transferable affordance points to accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis of our method and baseline, we demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: https://wayrise.github.io/DexGarmentLab/.

Dex GeGmmnnttta:: Dexterous Garment Manipulation Environment with Generalizable Policy
Yuran Wang""
Ruihai Wu
Yue Chen
Jiarui Wang'
Jiaqi Liang'
Ziyu Zhu'
Haoran Geng²
Jitendra Malik?
Pieter Abbeel? Hao Dong
1 Peking University
Universityy of California, Berkeley
Abstract
Garment manipulation is critical challenge due to the diversity in garment cate- gories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLas the first environment specifically designed for dexterous (espe- cially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural corre- spondence to automatically generate dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose Hierarchical gArment- manipul-ation pOlicy (HAL.O). first identifies transferable affordance points accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis our method and baseline, demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: hwtpaaawwssgiiibbbboooeoaeeemm.m
1 Introduction
The ability to manipulate various objects is critical for general robots. Despite advancements in the manipulation of rigid [12) and articulated 33 objects, deformable objects, and garments in due to their highly variable geometries and intricate deformations. Despite this, humans handle garments with remarkable ease using dexterous hands, thanks to their superior adaptability, larger manipulation area, and coordinated finger control, highlighting the importance of equipping robots with similar capabilities. Dexterous (especially bimanual) hands enable stable and precise actions (such as catching, cradling, pinching, and smoothening, shown in Fig. D. and excel in complex tasks like tie knotting and assisted dressing, where multi-finger coordination ensures accurate and adaptive manipulation.
particular, continue to pose substantial challenges 34
36
35
Dexterous garment manipulation faces three key challenges: (i) Data: The high-dimensional action space of dexterous hands and the complex nature of garments make policy learning data-intensive 11 6). What's 's more, different garment manipulation tasks require different hand grasp poses to ensure smooth manipulation and make garments maintain the desired condition. Directly collecting real-world data is impractical due to high cost. Thus, researchers have pursued simulators for garment manipulation
2 26 16 25 but these simulators often rely on labor-intensive teleoperation
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DEXCARMENTEE
ENVIRONMENT
GARMENT STRUCTURE
POLICT
15 Task Seenes 2500+ Garment
Generale Goscidialle Alloodanee
(eteration
Lavd Generilictton
Gureet Sazee Stncase
AUTOMATED
Affedino Medd
DATA COLLECTION
Gereraic Camuraltohi Trijesrriee
Beidetl
Eetiake Polky
2011
Digil
Posttios
Play
Generalize
Actos
Slape
Deformaticn
Gaemso
Eev Cord
Interaction
Diverse Deformations
Catch
Cradle
RMM
Divente Guamrnat Slapes
Figure 1: Overview. DexGarmentl.ab includes three major components: Environment, Automated Data Collection and Generalizable Policy. Firstly, we propose Dexterous Garment Manipulation Environment with 15 different task scenes (especially for bimanual coordination) based on 2500+ garments. Because of the same structure of category-level garment, category-level generalization accessible, which empowers our proposed automated data collection pipeline to handle different position, deformation and shapes of garment with task config (including grasp position and task sequence) and grasp hand pose provided by single expert demonstration. With diverse collected demonstration data, introduce Hierarchical gArment manipuLation pOlicy (HALO), combining affordance points and trajectories to generalize across different attributes in different tasks.
or expert RL policies to collect demonstrations, which are inefficient. (ii) Environment: Effective garment manipulation involves interactions not only with garments but also with rigid and articulated objects (such as hangers, the human body, etc.). Current simulators [23, 39 lack the necessary level of realism and physical accuracy for complex interactions, particularly with dexterous hands. (iii) Algorithm: Garment manipulation requires understanding diverse geometries, complex states, and difficult goals. Existing reinforcement learning (RL)
44 23 imitation learning [ 6
approaches often require intricate task task-specific reward designs or extensive demonstrations, limiting their scalability to real-world applications.
To address these challenges, we introduce DexGarmentLab (Fig. an environment built upon Isaac Sim and specially designed for dexterous (especially bimanual) garment manipulation, featuring: (1) Diverse and Realistic Environments: A large-scale dataset of more than 2,500 garments categories from ClothesNet [45), high-quality 3D assets for 15 task scenarios, paired with advanced simulation techniques to reduce sim-to-real gap. (2) Automated Data Collection Pipeline: An automated pipeline that generates diverse demonstrations based on garment structural correspon- dence with the help of a single expert demonstration, facilitating the generation of large and varied datasets without requiring manual intervention. (3) Generalizable Policy: a Hierarchical gArment- manipuLation pOlicy (HALO) which leverages affordance (for locating garment manipulation areas) and diffusion method (for generating trajectories based on garment and scene), achieving better generalization in garment manipulation than previous imitation learning algorithms.
In summary, contributions include:
We introduce DexGarmentLab Environment, the first simulation environment for dexter- ous (especially bimanual) garment manipulation, featuring a wide range of task scenarios, high-quality assets, and realistic physical interactions.
We propose pipeline for automated data collection, generating diverse demonstrations in various task scenarios and reducing the need for manual intervention.
We propose Hierarchical grrmenttmanipulaate Lation pOlicy (HALO), a novel hierarchical framework that leverages affordance and diffusion method to enable generalizable manipu- lation of diverse garments.
Extensive experiments and detailed analysis of our approach and baseline in both simu- lation and real-world settings, demonstrating its data efficiency and generalization ability significantly outperforming baseline methods.

---

2 Related Work
2.1 Deformable Object Simulation
39 21 limited in the types of objects they support and lack realistic physical interactions, which hinders dexterous garment manipulation research. For instance, softgym [33 is confined to simulating tops and trousers while fluidlab [39 can only simulate fluids. Although Lu et al. [25] extends to various object types, it relies on attaching invisible cubes to garments, failing simulate realistic physical interactions. In contrast, we introduce DaxaarmentLab, which incorporates extra physical parameters (adhesion, friction, htwpccd-ddddesioooiiitto---sssse to guarantee stable and realistic interaction between garments and robots. Additionally, simulators [23] 26 39 16 25] often rely on labor-intensive telcoperation or expert RL policies to collect demonstrations, which are inefficient. Thus, we propose an automated pipeline that exploits structural correspondence across garment categories to collect demonstrations, eliminating manual effort. Table shows comparisons between DexGarmentLab and other environments.
Current deformable simulations [8 23
etc.)
Table 1:
Comparisons with Other Deformable Object Environments
Simulator
Seene
Garment
Sim on
GPU
Multiple Dexhands
Garment Task
DexHand
Data Collection
Method
Demonstration
Data
Robot- Interaction
Physically Plausible
Softgym
Manval
PyBulket
Manual
Fluidlab
Manual
Desterous Gym
Manual
x
Sapien
Manual
GameenLab
Manual
DexGarmentlas
Automated
2.2
Dexterous Manipulation
Dexterous manipulation has emerged as a critical research frontier, with applications tasks like in- hand manipulation [40 15
19 9 42 and deformable object manipulation [2 38 43 However, most researches focus on single-handed manipulation, overlooking bimanual dexterity, which is essential for tasks like cradling garments and pinching gloves (Shown in Fig. D While recent efforts have demonstrated bimanual capabilities in specialized contexts including lid manipulation (22), dynamic object interception [17], and articulated object manipulation 442, these approaches remain fundamentally limited to rigid-body dynamics. Our work performs the first investigation on the learning- based bimanual dexterous manipulation of garment and build the pioncering environment with diverse scenarios covering different garment categories.
29
37 8 articulated object manipulation [4
2.3 Garment and Deformable Object Manipulation
While much research has focused on manipulating simple deformable objects like square-shaped cloths 34 38 [24] ropes and cables [31 34 38) , and bags 2 7 garment manipulation presents
a substantial challenge. Garment manipulation involve diverse geometries, complex deformations, and fine-grained actions. Many existing studies on dexterous garment manipulation rely on optimize method [3 [3], which struggles with the high freedom of bimanual dexterous hands. Zhaole et al. [4)) Lin al. (23) attempt solve bimanual dexterous manipulation tasks with reinforcement learning (RL). However, they require intricate reward designs tailored to specific manually designed tasks. Avigal al. [ Canberk al. 6 rely on large-scal annotated data, which is labor-inte iebor-intensive and time-consuming, hindering the scalability in the scenarios of real-wor world applications. In this paper, we introduce Hierarchical ga cemeet-manipul...e diffusion method to facilitate manipulating diverse unseen category-leve garments in multiple tasks with different scene configurations.
pOlicy (HALO) which leverages affordance and
3 DexGarmentLab Environment
In this section, we present the construction of DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation and built upon IsaaeSim 4.5.0.
3.1
DexGarmentLab Environment Setup
Observation Space. The observation space includes both proprioceptive and visual data, Propriocep- tive data comprises robot joint positions, end-effector 6D poses, and other kinematic information.
3

---

TRANSFER
COMPARE
Grasp with Attaclancatlloc
Grasp with AttachmentBlock
Grasp with Our Precise Simulation
from Garment ab)
(DexGarmentl ab)
(DexGarmentLab)
Red Objects in Pictures Meum Atiachmen::Bock
Figure 2: Comparison of Garment-Robot Interactions between GarmentLab and DexGarment- Lab. Left: In GarmentLab, Franka grasp garment with red block attached. Middle: We transfer Attachementlookk method to dexterours hands and set red block at the tip of each finger (ten blocks totally). The performance is not good, described in 3.2 Right: Our method (DexGarmentLaab) can make the interactions between dexterous hands and garment more natural.
COMPARE
GarmentLab
More Realistic Physical Simulation in DexGarmentLab
Tops
Trousers
Dress
Tops
Figure 3: Compare Garment Properties between GarmentLab and DexGarmentLab. Left: After folding in GarmentLab, the garment struggles to maintain a stable folded state and easily becomes disorganized. Right: With realistic physical simulation in DexGarmentLab folded garments can stably maintain their folded states, which means Garment Properties is more mature and natural.
Visual data includes point clouds captured by depth camera and RGB images. The point cloud is cropped to the robot's 's workspace and down-sampled for efficiency.
Action Space. The action space is a 60-dimensional vector: 6 DoF for each arm and 24 DoF for each Shadow Hand. The UR10e arms be controlled using both given end-effector 6D poses and inportional-Derivaan (PD) Controller given joint positions, while Shadow Hands are controlled using PD controller based on joint positions.
(IK) Controller
3.2 DexGarmentLab Physcial Simulation
Simulation Method. To achieve realistic simulation, employ methods tailored the physical properties of garments. Large garments (e.g., tops, dresses, trousers, etc.) are simulated using Position-Based Dynamies (PBD) (27), while small, elastic items (e.g., gloves, hats) modeled via the Finite Flement Method (FEM) [5] We provide detailed introduction and selection reason about PBD and FEM in Appendix [.] Human avatars are represented by articulated skeletons with rotational joints and a skinned mesh for lifelike rendering.
Key Design for Physical Garment Simulation. PBD is widely used for simulating most garments, but its loosely connected particles often allow grippers to penetrate the garment without achieving effective lifting. GarmentLab introduces attach blocks to address this, enabling garment- -eripee--rrpperr attachment (Fig. 2 left). However, this approach fails to capture realistic interactions, resulting in unnatural sagging when applied dexterous hands (Fig. 2 middle). Moreover, minimal contact-such as a single finger block touching the garment- can establish attachment and lift the garment, which is clearly unreasonable.
Therefore, we introduce adhesion (between particle and rigid), friction (between particle and rigid) and particle-scale (between particles) parameters enhance realism. Benefiting from friction and adhesion, dexhands can grasp and lift garments based on physical force without attach blocks (Fig. 2 right), while particle-adhesion (or friction)-scall stabilizes the particle system, preventing excessive self-collisions between particles which cause garments to become disorganized (Fig. 3). We provide more details in Appendix A

---

SIMULATION ENVIRONMENT
Fling Tops
Fling Dress
Fling Trousers
Fold Tops
Fold Dress
Hang Dress
Wear Baseball Cap
Fold Trousers
Hang Tops
Hang Trousers
Hang Coat
Wear Scarf
Wear Bowl Hat
Store Tops
Wear Glove
Figure DexGarmentLaal Simulation Environment. We introduce 15 garment manipulation tasks across categories, encompassing both sermeen-self-inteaacctn (e.g., Fling, Fold) and garment- cnvironment-interace (e.g.. Hang. Wear, Store) scenarios. In garment-self -en-ttrrctttttnnatiiiinn tasks, key variables include garment position, orientation, and shape. In seemeetteniiiromme---tteee environment-interactii assets positions (e.g., hangers, pothooks, humans, etc.) are also considered.
tasks,
3.3 Asset Selection and Annotation
We garment models from the ClothesNet dataset 455 which contains over 2,500 garments across 8 categories (e.g., tops, coats, trousers, dresses, etc.). and build environment-interaciooo assets (such as hangers, pothooks, humans, etc.). We provide plain meshes customizable with colors and textures for garments to support both realistic and controlled experimental setups. Controlled randomness in placement for both garments and enviromment-interaan translations- cuusllatons-maiitainns task feasibility while enhancing generalization in policy learning.
thhouuh limited rotations and
3.4 DexGarmentLab Tasks
Dexterous (especially bimanual) garment manipulation is vital for domestic applications, yet has not been thoroughly explored in existing research. To address this, introduce 15 tasks across garment categories (Fig. D. Further details on these tasks are available Appendix K
4
Automated Data Collection
Collecting data through teleoperation or RL is highly labor-intensive, especially for dexterous garment manipulation tasks, due to the diverse shapes and deformations of garments and the high- dimensional action space of dexterous hands. This makes automated data collection essential, with the key challenges being: 1) identifying appropriate manipulation points across different garment configurations, and 2) generating task-specific hand poses accordingly.
In our proposed automated data collection pipeline, for given task, we begin with a single expert demonstration to extract key information: hand grasp poses, task sequences, and demo grasp points on the garment. Leveraging the Garment Affordance Model (Refer Sec. 4.I). we use affordance to identify target grasp points on novel garments with diverse deformations corresponding to demo grasp points. Then, the pipeline executes the task sequence based on inferred points and hand grasp poses, thereby enabling efficient and scalable data collection. Sec. 4.2 explains the whole procedure.
4.1 Garment Affordance Model (GAM)
Built upon the UniGarmentManip (35) framework, GAM leverages structural and correspondence consistency across category-level garments, enabling the identification of target grasp points on category-level novel garments. For training process, as shown in Fig. 5 (Red Part), we employ Skeleton Merger 322 network architecture obtain the skeleton point correspondences between flat garments while adopting the point tracing method in simulation to establish correspondences between flat garment and its deformed version. Using InfoNCE (00 loss function, we train PointNet++ (8 to pull features of positive corresponding point correspondences closer while pushing apart negative corresponding pairs, which enhances dense visual correspondence by enabling alignment across different garments in various states. Please refer
[35] for more details.

---

Inference Process
Points
Actions with
Demo Task Configuration
Demo Points
Features
Target Manipulation Points
Demo
Garment Alfordance
Single Expert
Model
aue Shot Transfer
Target Points
Fentures
111
Operaned Point Cloud
Mouch
Demo Peints
Dema Tack
Sequenee
pemonstration
Training Process
Execution
GarmentAffordamee Affordance
STEP
Grasp Posc
Model
LeepMotioon
Shicsten Meggs
Donn Prse
Clas
TCfONCE
Paskec
Negeins
Loss
Corresponding Pairs
Open
Figure 5: Automated Data Collection Pipeline. Given a single expert demonstration, can get demo points, demo task sequences and demo grasp poses for the specific task. Category-Ievel garment (w/ or w/o deformation) has almost the same structure, base on which we train Garment Affordance Model (GAM) with category-level generalization. With GAM (refer Sec. 4.1 we match demo points from the demo garment point cloud O robot to execute the specific task based on the demo task sequences (through trajectory retargeting) with dexhands' movement guided by demo hand grasp poses (through PD controller based on joint positions). FFlld Tops' task is shown as example in this figure.
garment point cloud O' and control
It's worth mentioning that, enable GAM to handle point clouds with translation and scale invariance, we pre-normalize the input demo/operated point cloud into a canonical space. This ensures that GAM maintains generalization ability when faced with garments undergoing different translations and scales. As for the issue of rotational invariance, we consider it as the garment's deformation state. By generating sufficient samples with varying rotations, enable GAM to effectively learn the correspondences of garments under different rotations.
For inference process, shown in Fig. 5 (Green Part), with pretrained GAM, given demo garment point cloud O, demo grasp points (P1, P2. P2...) ...) and one operated garment point cloud O', we can obtain the demo grasp points features (pp. fp2.... ...) and operated garment observation features. By dotting demonstration grasp points features and operated garment observation features to get similarities and selecting features with biggest similarity scores, points (p/, P/2 ...) on O'.
get corresponding grasp
4.2
Automated Data Collection Pipeline
Here we give a detailed description about our automated data collection pipeline shown in Fig. 5
Firstly. We obtain demo grasp points, demo task sequences, and demo hand grasp poses from single expert demonstration. In our actual operation process, while grasp points and task sequences are manually defined, hand poses generated using the LeapMotion solution (see Appendix (B).
Once target grasp points are identified on the operated garment using GAM, with demo task sequences, we control the robotic arms via inverse kinematics (IK) to execute sequential operations while controlling dexterous hands using PD controller based on the joint positions from demo hand grasp poses. It is important to note that the task trajectories are not fixed but are adapted based on the garment's s shape and length. For example, in serrreentt-eff-iterreeee lifting height is adjusted according to the sleeves and overall garment length to ensure proper folding. In garment-enviineeeee envirommettitittttteeeee -inereenntinteeactionn tasks, such as hanging, both the lifting height and placement position are adapted to align the garment's center with the hanger, preventing slippage. These adjustments reflect common and reasonable actions in real-world scenarios, and introducing such variations increases the task difficulty. The details about tasks can be found in Appendix K
tasks, such as folding, the
During task execution, we can simultaneously record various information (such as images, point clouds, robot joint states, etc.) within the simulation environment. These serve expert demonstra- tion data for subsequent offline training of the policy. Details about recorded information can be found in Appendix D

---

5 Generalizable Policy
When dealing with garments, which exhibit highly complex deformation states, current mainstream imitation learning (IL) algorithms (e.g. Diffusion Policy [00], Diffusion Policy 3D 411 show relatively poor generalization (as evidenced by our experimental results shown in Tab. 2 The main
issue is that IL-based trajectories fail to accurately reach the target manipulation points on garments with new shapes and deformations, while also being unable to generate suitable trajectories based on the garment's 's own shape and structure, ultimately leading to manipulation failures.
To address this, we propose Hierarchical gArment manipuLation pOlicy (HALO), generalizable policy to solve the manipulation of garments with complex deformations and uncertain states. HALO decomposed into two major stages, as shown in Fig. 6
STAGE I
STAGE II Generalizable Trajectories
Structure-Aware Diffusion Policy
Generalizable A Affordance Points
CALCULTTE
Environennt
Tran
Expert
EXECUTE
Operatied
Demo Point Cloud
INPUT
Perception
Decision
Potnf Clond
Garment Affordance
Model
PERCEPTION
Concuc
DECESION
Ferure
POINTNET
Timeriice--pee Port Clod
crncakrole
Naael beike
Targe
Gerlio
Gra
Bonking
POINTNET
Dope
labt Sezz
Target Point eft)
Target Poier (Right) Affordance Heatmap
Opermed GETTNTE
That
Affordance Heatmap
Figure 6: Generalizable Policy. We adopt hierarchical structure to implement Generalizable Policy, Firstly, we use GAM to generate generalizable affordance points, which will be used for robots to locate and move to target area. Secondly, we introduce Structure-Aware Diffusion Policy (SADP), which extracts features from garment point cloud (with left and right point affordances as binding features), interaction-object point cloud, environment point cloud and robot joint states as condition to generate joint actions (including 24 DOF for each hand and 6 DOF for each arm, totally 60 DOF).
In the first stage, we use GAM to accurately locate the manipulation area of the garment, addressing the limitation of previous IL grasping brand-new clothes at the correct position. Refer Sec. [.1 for GAM's details. We next focus on the design of the Structure-Aware Diffusion Policy (SADP).
Due the poor generalization ability of current mainstream methods such as DP and DP3 for complex and variable garment manipulation scenarios, we propose Structure-Aware Diffusion Policy (SADP), narment-environmenn generalizable diffusion policy that improves the generalization for different garment shapes and scene configurations, thereby enabling the smooth generation of subsequent trajectories after moving to the target manipulation area guided by GAM.
SADP fundamentally follows the framework of Diffusion Policy 10), with the primary distinction lying in its observation representation, denoted as s, which is elaborated below.
With operated garment point cloud and left / right target point affordances generated by GAM, we concatenate them together and use PointNet++ 28 to extract garment feature F Egarment, while using MLP-based Feature Extractors to extract interaction-objed fcature Fobject- Fgarment and Fobject are concatenated into compact scene feature Fscene- At each timestep, the full environment point cloud O Oenwiromment and the robot state Ostate are encoded using MLP and fused with Faceme to form the denoising condition s for SADP. As for Garment-San Interacttnttn tasks without interaction-ojject point cloud, we only use Fgarment to be Fscene. which means interaction- point cloud is optional. Here, Fgarment captures current garment state (position, shapes, structure, etc.), while Fotject reflects current interaction-ojject state (position, etc.).
Through experimental validation, SADP exhibits better generalization capabilities. We will further illustrate this advantage with experimental results in Sec. 6.2 Training details can be found in Appendix H
7

---

Experiment
6.1 Environment Setup
Tasks and Environments. We evaluate method 14 garment manipulation tasks with varying deformation characteristics. Detailed environment specifications including scene randomization parameters, success metrics, and train/test configurations provided in the Appendix [K]
we acquired 100 demonstrations per task, with 30- 100 environment steps in each demonstration. The time required to collect a demonstration varies between approximately 30-80 seconds depending on the task, which is significantly faster than data collection through teleoperation. We make comparison between
Demonstration Collection. Using our automated data collection pipeline (Sec.J), 4)
autonomously collected data and teleoperation data Appendix F
Baselines, We compare against two state-of-lt offte-of-the-arr diffusion- approaches across all the garment manipulation tasks: 1) Diffusion Policy (DP) [TO]. which utilizes images as observations to generate actions via diffusion. 2) 3D Diffusion Policy (DP3) [4], which replaces image inputs in the Diffusion Policy with point clouds. What's more, we have also additionally included four new baselines for comparison: ACT (IL), pio (VLA), RDT (VLA), and Eureka (RL+VLM) and select representative tasks for evaluation, including Fling Dress, Fold Trousers, Hang Coat, and Wear Bowlhat in simulation, as well as Fold Tops in real world, which can be found in Appendix G
Ablations. To analyze components, we evaluate two ablated variants: 1) w/o GAM: Excludes the Garment Affordance Model (GAM), using SADP for trajectory execution. 2) w/o SADP: Removes Stracture-Aware Diffusion Policy (SADP), using GAM DP3 for trajectory execution.
Metrics. Each task is evaluated over 50 episodes with three different seeds. We report success rates as MM ean + Std across all trials. Table 2: Quantitative Results in Simulation. HALO outperforms baselines and ablations.
Methal
Diow
Togn
Trveact
Tign
Trenacts
Tign
City
Rop
BF
1.35
3.1
8.14
DJS 1u33
0.3 3
CAT tuk
1651 4:23
0.48 301
C.56 (00
1152 14004
0.65 DAS
641 24155
0.57 can
Buo tuc
Oas wil GAM
D006 003
0.68 = 06
Dst enos
065 00
077 2003
064 9:0
6 em
0.2-00 -020
0464 6/0
045 -000
6 0:00=001
Oha SADP
2:0
CA8 tub
0.71 a:033
EX11
034
0:8 1u77
0.66
a
8.77
0.55
8 RMM
0.
a.5
6.2 Results Analysis
Tab. (Row 1, 2, 5) quantifies HALO's performance against baseline methods simulation. Our method achieves superior success rates across dexterous garment manipulation tasks, demonstrating statistically significant improvements over existing approaches. We also provide more baselines comparison in Appendix G
Row 3, 4, 5) quantifies the impact of ablating two core components in HALO: the Garment Affordance Model (GAM) and the Structure-Aware Diffusion Policy (SADP). provides visual evidence of these effects.
Our ablation study (Tab. 2
Fig. 7
Through Tab. 2 we find that the performance of HALO markedly decreases when GAM is excluded. The integration of GAM leverages dense visual correspondence, thereby enhancing the model's performance to locate precise manipulation area (shown in Fig. 7 particularly for tasks with significant variability in garment shapes and deformations.
Sunckac-benr:
Stacter-Anae Diffiaiea
Fusl Carch
Hong Treaurs
Bryond Hargiea
Ting Dress
Artendaaoe
Hang Thraes
Uncomperiasy
Foldd Toer
Figure 7: Effects brought by GAM SADP. Left: Without GAM, robots fail to catch the right manipulation points. Right: Without SADP, robots fail to adjust the trajectories based on garment shapes and structures.
Moreover, SADP substantially boosts the performance of HALO. When the model encounters garments that differ considerably from those in the training set, SADP can assist in adjusting the trajectories according to the garments' own shapes and structures, thereby better accomplishing
8

---

Govenizzble Kovennee
Geternle Gzsculizkle
inose to larget
procedurae
state
move to target unscipulation
lift procedure
Ginal state
Alfredasee Points Hoomop
Figure 8: HALO's whole procedure. Using "Hang Coat" as an example. GAM first infers target manipulation points for robot's movement, enabling generalization across different garments. Then, SADP generates trajectories based on the garment and scene configurations. Despite variations in garment shape, length, and pothook positions across scenes, SADP adapts accordingly, moving to accurate positions and lifting coats to appropriate heights to successfully hang them the pothook.
the tasks. For instance, as shown in Fig. 7 in the "Hang Trousers" task, the shape of the trousers determines the appropriate hanging height and forward-moving distance; the HALO can adjust its trajectory to complete the task. In the "Fold Tops" task, the HALO can adjust the folding position according to the garment structure, making the folds neater.
HALO enables precise manipulation point inference and generates robust policies for diverse garment and scene configurations. Figure [ further illustrates this.
6.3 Real-World Experiments
There are two ways using HALO for garment manipulation tasks:
1) transfer the data collection pipeline from simulation to real world, conduct automated data collection and policy training directly in the real world.
2) train the policy in the simulation and transfer the policy to the real world.
6.3.1 Experiments and Results Analysis for Way
we adopt way to demonstrate the effectiveness of HALO. In the Appendix F Tab. 3 we report the efficiency and success rate of real-world data collection, highlighting the strong sim-to-real performance of GAM.
Test Sample
Initial State
Execution
Initial State
Execution
Initial State
Exccution
Wear Hat Wear Scart' Hang Tops Fold Tops
Typo
Flail
Figure 9: Real-World Experiments (Way 1). We seleet Fold Tops GGarment-Suf--nteeaattoon) Hang Tops, Wear Scarf, Wear Hat GGarment-Environmenn Interaationn as typical tasks for illus- tration. The test samples have different shapes, length, deformations while position of garments and interaction objects are variable. Despite this, GAM of HALO ensures accurate manipulation area location for grasp while SADP of HALO ensures adaptive trajectories base on garment and interaction object.
Setup. Our setup comprises two RealMan RM75-6F (Arms) with Psibot GO-R (Dexterous Hands)
and a RealSense D435 camera (As shown in Appendix EE. We use Segment-Anything-2 (30 to segment the garment and interaction object from the scene and obtain corresponding point clouds.

---

Table 3: Real-World Evaluation different tasks.
Task
Fold Tops
Hang Tops
Wear Scarf
Wear Hat
9/
10/ 15
6/15
10/15
DP3
8/15
8/
7/15
9/15
Ours(HALO) 13/15
13/
11/15
14/15
Evaluation. We evaluate our proposed method on tasks: Fold Tops, Hang Tops, Wear Scarf, and Wear Hat. For each task, we have 3 distinct garments per category, each with 5 initial deformations. Shown in Tab. our method outperforms all basclines. Fig. @demonsttatess the excellent performance of our proposed method.
6.3.2 Experiments and Results Analysis for Way
Sim-to-real transfer from the simulation environment remains an important aspect of our study, To this end, shown in Fig. TO, we aligned the settings of both the simulation and real-world environments by using the same hardware setup-Shadod Hand and UR 10e-and selected two tasks, Hang Trousers and Wear Hat, for policy-level sim-to-real transfer, which means way 2. The evaluation criteria follow those used in the previous real-world experiments.
It is worth noting that sim-to-real performance is more sensitive to point cloud noise. To address the limited precision of the Realsense D435 in this context, we employed a Kinect camera for more accurate point cloud acquisition.
Kinect
Hang Tiusses
URIOe
Shadow Hand
Manipulation Platform
Object Set
Obscroaiio
Observation
Affordance
Figure 10: Real-Woor World Experiments (Way 2). Two URe0e paired with ShadowHands and an Azure Kinect camera are used in our sim-to-real experiment. For each task, we show the scene configuration, affordance generated by GAM and trajectories generated by SADP.
Table 4: Performance Impact of Adding Soal-World Data to Simulation Data
Task Name
Only Simulation Data
Simulation Data + 15 Real-World Data
Hang Trousers Wear Hat
(53.3%)
13 (86.7%)
9 (60.0%)
13 15 (86.7%)
Experimental results in Tab. show that due the gap between simulation and the real world, training the policy solely on simulated data leads to a drop in sim-to-real performance. Incorporating a small amount of real-world data into the training process can effectively enhance the policy's 's generalization ability.
7 Conclusion
In this paper, we introduce DexGarmentLab, the first simulation environment designed to address the challenges of dexterous (especially bimanual) garment manipulation. Our work mainly makes three key contributions, including DexGarmentLab Environment, Automated Data Collection and Generalizable Policy. Through extensive experiments, we demonstrate that our approach can ef- fectively learn complex manipulation tasks with minimal supervision and generalize across wide range of garment shapes and deformation states in both simulation and real-world environments. The limitation of our work is discussed in Appendix
Acknowledgment
This project was supported by National Natural Science Foundation of China (62376006) and National Youth Talent Support Program (820080008 l).
10

---

References
[1] Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, and Ken Goldberg. Speedfolding: Learning efficient bimanual folding of garments, 2022. URL/ittp::/arxvvvoor//////
10552
[2] Arpit Bahety, Shreeya Jain, Huy Ha, Nathalie Hager, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Bag all you need: Learning a generalizable bagging strategy for heterogeneous objects, 2023. URL.https://arxiit http://axiivvorrrras/////// 09997
[3] Yunfei Bai, Wenhao Yu, and C. Karen Liu. Dexterous manipulation of cloth. Computer Graphics Forum, 3522):223-532, 2016. doi: https://doi.org/00..11 l/cgf.12852. 12852. URL https
/onliieelibrary wiley. www/doi////ccccccccccccccc.c..ot aatooo////////a abs/aaasa../////eee 1111/cgf. 12852.
[4] Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipulation with articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21190-21200, June 2023.
[5] Klaus-Jürgen Bathe. Finite element method. Wiley encyclopedia of computer science and engineering, pages 1-12, 2007.
[6] Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Cloth funnels: Canomicalizzddallignment for multi-purpose garment manipulation,
2022. URL CtRL/htps::/// /tarxiv//arxiv..o hwtp://///aaaabaaaaaba htttpp://arxiv..org//b/////111/
09347
[7] Lawrence Yunliang Chen, Baiyu Shi, Daniel Seita, Richard Cheng, Thomas Kollar, David Held, and Ken Goldberg. Autobag: Learning to open plastic bags and insert objects, 2023. URL
attps://arxix>: http://wwxvir../..a////// abs/ 2210. 17217
[8] Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jiechuang Jiang, Zongqing Lu, Hao Dong, and Yaodong Yang. Bi-dexhands: Towards human-level bimanual dexterous manipulation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 46(5):2804-2818, 2024. doi:
10.1 1099TPPAM..22233335555
[9] Yuanpei Chen, Chen Wang, Yaodong Yang, and C. Karen Liu. Object-centric dexterous manipulation from human motion data, 2024. URL btttpp://arxiii www.oogg//aa..eo.o..o berrg/bba/20 abs//aa/.11../////////// //1.//4111....5...
[10] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668. 2023.
[11) Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang. Ruihan Yang, Xiaojuan Qi, and Xiaolong Wang. Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning, arXiv preprint arXiv: 2rXiv:2407.03162. 2024.
[12] Yushi Du, Ruihai Wu, Yan Shen, and Hao Dong. Learning part motion of articulated objects using spatially continuous neural implicit representations, 2023. URL URL /tttps////ax org/ abs/ 2211. 12407
[13] F. Ficuciello, A. Migliozzi, E. Coevoet, A. Petit, and C. Duriez. Fem-based deformation control for dexterous manipulation of 3d soft objects. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4007-4013, 2018. doi: 10. 10.1 1000/RROS
8593512.
[14] Huy Ha and Shuran Song. Flingbot: The unreasonnale offersicseseeeeeeeefeettteeeeee of dynamic manipulation for cloth unfolding. 2021. URL
03655
[15] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, -aa-nocccois Lafleche, Dieter Fox, and Gavriel State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv,, 2022.
[16] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation, 2020. URL https: / aariiiiv. org/ abs/ hwtp:/9110.0000 00935

---

[17] Binghao Huang, Yuanpei Chen, Tiayy Wang, Yuzhe Qin, Yaodong Yang, Nikolay Atanasov, and Xiaolong Wang. Dynamic handover: Throw and catch with bimanual hands, 2023. URL
pettp://arxiv..oo abs /209 05655
[18] Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics, 2021.
UtLpss/:s://axi org/ httg/bb//1000 03311
[19] Taoran Jiang, Liqian Ma, Yixuan Guan, Jiaojiao Meng, Weihang Chen, Zecui Zeng, Lusong Li, Dan Wu, Jing Xu, and Rui Chen. Dexsim2real Building explicit world model for precise articulated object dexterous manipulation, 2024. URL https: http:ss://xxvvorr////// org/ abs/
00750
[20] Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint arXiv: 19X4001904.01555, 2019.
[21] Sizhe Li, Zhiao Huang, Tao Chen, Tao Du Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics, 2023. URL http:///xaxxwv..ooaaaaao 2304. 03223
[22] Toru Lin, Zhao-Heng Yin. Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Twisting lids off with two hands, 2024. URL https:://arxvv.org//bba/a/ wttp:///r.aab///....../ 02338
[23] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforce- ment learning for deformable object manipulation, 2021. URL wtt.tttss:///aaaattt httptbtt/axxpi..oooo. abs/ 2011.07215
[24] Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing, 2022. URL http:t//////ttt org/ aat/a/as//////// 10389
[25] Haoran Lu, Ruihai Wu, Yitong Li, Sijie Li, Ziyu Zhu, Chuanruo Ning, Yan Shen, Longzan Luo, Yuanpei Chen, and Hao Dong. Garmentlab: A unified simulation and benchmark for garment manipulation, 2024. URL https://s/// www.arwwww....ooooooooooo abs 2411. 01200
[26] Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Yong Kim. Unified particle physics for real-time applications. ACM Trans. Graph 33(4), July 2014. ISSN 0730-0301. doi: 10.1144/2601097.26000 200007.6611122
URL /tt.//ooiiooooo org/oo..//1.
[27] Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff Position based dynamics. Journal of Visual Communication and Image Representation, 18(2):109-118, 2007.
[28] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[29] Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-Hand Object Rotation via Rapid Motor Adaptation, In Conference on Robot Learning (CoRI.), 2022.
[30] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick. Piotr Dollár, and Christoph Feichtenhofer, Sam 2: Segment anything in images and videos, 2024. URL https //arxiv.org/ea wttp::///abbb/2....//// 00714
[31] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks, 2023. URL https://axxtx.......// http://wwwvvv.r/r///oo// ab//abaa///00 2012 03388
[32] Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu. Skeleton merger: unsupervised aligned keypoint detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 43-52, 2021.
[33] Ruihai WI, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. ICLR, 2022.
12

---

[34] Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightfll dense visual affordance for deformable object manipulation, 2023. UKL.ttpss//ar22...0/////
11057
[35] Ruihai Wu, Haoran Lu, Yiyan Wang, Yubo Wang, and Hao Dong. Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16340-16350, 2024.
[36] Ruihai Wu, Ziyu Zhu, Yuran Wang, Yue Chen, Jiarui Wang, and Hao Dong. Garmentpile: Point- level visual affordance guided retrieval and adaptation for cluttered garments manipulation. arXiv preprini arXiv:2503.09243 2025.
[37] Tianhao Wu, Jinzhou Li, Jiyao Zhang, Mingdong Wu, and Hao Dong. Canonical representation and force-based pretraining of 3d tactile for dexterous visuo-tactile policy learning, 2024. URL https::/ /arxiv org/ abs 17549
[38] Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020. URL https: /tttps://rxxiv..o org/ abs/1910. 13439
[39] Zhou Xian, Bo Zhu. Zhenjia Xu, Hsiao- Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation, 2023. URL httptttsss://iii..oooo
[40] Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without sceing: Towards in-hand dexterity through touch. arXiv preprint 108800 2023.
[41] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. arXiv preprint 1eXiv:2403.039555 2024.
[42] Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In 2024 International Conference on 3D Vision (3DV), pages 235-246, 2024. doi: wttp:///DDD22222222..00000
[43] Sun Zhaole, Jihong Zhu, and Robert B. Fisher. Dexdlo: Learning goal-conditioned dexterous policy for dynamic manipulation of deformable linear objects. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16009-16015, 2024.
doi: 10. 1199/
ICRAS7147.2024, 211.44722222111111177 10610754.
[44] Sun Zhaole, Jihong Zhu, and Robert B Fisher. Dexdlo: Learning goal goal-conditioned dexterous policy for dynamic manipulation of deformable linear objects. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 16009-16015. IEEE, 2024.
[45] Bingyang Zhou, Haoyu Zhou, Tianhai Liang, Qiaojun Yu, Siheng Zhao, Yuwei Zeng, Jun Lv, Siyuan Luo, Qiancai Wang, Xinyuan Yu, Haonan Chen, Cewu Lu, and Lin Shao. Clothesnet: An information-rich 3d garment model repository with simulated clothes environment, 2023.
URL httthttps:// //tp:i/.////i///// org/ ab// 2308 09987
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Section
5 Appendix Appendix
K
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Please refer to our code repo https//wwayyigggghhhoo.oooooaaar
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/pips.cccppullc//guui /
Soa.eeees//ooSSeemiisssee www.SSubbissioooooioiioooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
we provide detailed evaluation of our method in both simulation and the real world, reporting various scores such as average success rate, standard deviation, and others.
Justification: As shown in Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix L
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No experiments with potential risks for study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix Overview
A Details about Key Parameters in PBD for Improved Realism
Comprehensive explanation about how the aateenteersdaeesoonnnn friction, porticle-adhesion-scall scale-contribute particle-frictio-ss scale- contribute to improving the realism garment in the simulation environment.
B LeapMottio for Teleoperation
Detailed overview of the LeapMotion workflow and its operational performance.
C A dvantages of Dexterous Hands over Parallel Grippers
Detailed explanation advantages of dexterous hands over parallel grippers.
D Recorded Information in Automated Data Collection
Comprehensive introduction to the various types of environmental information collected in the simulation during the automated data collection phase, along with the techniques employed for efficient data acquisition.
E Real- World Experiment Scene
Schematic Diagram of the Real-World Experimental Setup.
F Comparison with Teleoperation Data
Detailed comparison between autonomously collected data and telcoperation data to demonstrate the efficiency and high quality of automated data collection pipeline.
Additional Baseline Comparison
Additional baseline comparison for demonstrating the excellent performance of HALO.
H Training Details of Main Algorithm
Training details for the GAM and SADP algorithms, including hyperparameter settings, computational resources, and other relevant specifications.
I Limitation
The limitation of our work, including simulation-method method limitation, task limitation, data collectiooo limitation.
DarGmennet Assets
detailed description the various assets used in DexGarmentLab, including garment assets, environment-interactiooo assets, robots assets, and material assets.
K Detailed Task Description
A detailed description of all garment manipulation tasks involved in DexGarmentL.at including task environment initialization and randomization, task sequence, task success metrics, and garment assets used for each task.
Broader Impact
The potential societal impacts of our work.
21

---

A Details about Key Parameters in PBD for Improved Realism
Position-Baset Dynamies (PBD) is widely adopted method for simulating deformable objects such as garment. It operates by enforcing geometric constraints directly on particle positions, offering both numerical stability and computational efficiency. In this section, we elaborate on several key parameters - adhesion, friction, particle-adhession--ceee and particle-friction-sall which significantly influence the realism and stability of garment-object interactions, particularly in dexterous manipulation scenarios.
A.1 Adhesion
Adhesion introduces artificial attractive forces between particles between particles and surfaces (e.g., a robotic hand). is used to simulate surface stickiness, enabling persistent contact during manipulation. Although adhesion is not physical force in classical mechanics, it can be heuristically modeled as:
fath = Kadh (x; x,);
||x: - xyyl < Yadh:
where x; and xj are the positions of two particles (or a particle and a surface proxy), kidh is the adhesion coefficient, and Fadh is the adhesion radius threshold.
Effect: Adhesion enables garments to maintain contact with the fingers of a dexterous hand without requiring continuous high-pressure gripping, facilitating reliable grasping and lifting.
A.2 Friction
Friction resists relative tangential motion and is critical for grasp stability. The classical Coulomb friction model is expressed as:
Ve |vell'
frie = - ||fa||
where y is the friction coefficient, is the contact normal force, and v, is the tangential relative velocity. In PBD, friction often approximated as a positional correction that opposes sliding during constraint projection steps.
Effeet: Friction allows garment to resist sliding off hand surfaces, enhancing control during manipu- lation.
A.3 Particle-Adheesinn-Salll
This parameter scales the adhesion forces between internal garment particles. helps prevent excessive separation or instability during self-collisions or folding. High values increase inter- inter-particl attraction, leading to more cohesive motion.
Effect: Particl-adhesion-scall improves garment stability by preventing explosive separations or chaotic folding, especially when complex self-contact occurs.
A.4 Particle-Priction-Scal
Particle-friction scale controls the internal friction between garment particles during relative motion. It is applied during internal constraint solving (e.g., stretch, shear, or collision resolution) and acts as a damping mechanism:
This parameter suppresses excessive internal sliding, preserving folds and wrinkles, and enabling more physically plausible draping and manipulation behaviors.
Effect:
A.5 Summary
Together, these parameters significantly enhance the realism of garment manipulation simulation. Benefiting from adhesion and friction, dexterous hands can grasp and lift garments based on contact
22

---

forces. Meanwhile. the particle-adheseee and friction scales stabilize internal garment dynamics, reducing self-collision artifacts and preserving structural coherence.
B LeapMotion for Teleoperation
We employ the Leap Motion Controller as a teleoperation device to control the Shadow Hand and generate task-specific hand poses, thereby facilitating the automated data collection pipeline. The Leap Motion Controller is a compact USB device designed for desktop use. It leverages two 640x240- pixel near-infrared cameras capture hand motion within roughly hemispherical interaction volume extending to approximately 60 cm, typically operating at 120 Hz. Its internal algorithms process the raw spatial data to extract 27 distinct hand features, including the palm normal vector, hand direction, wrist position, and 24 finger joint positions.
Here just utilize the 24 finger joint positions to control Shadow Hand's movement. The telcopera- tion system and visualization performance are shown in Fig IT
We have provided relevant tutorial about how to use LeapMotion in our released code. We also explain details on how LeapMotion generates grasping poses below.
For single specific task category (e.g., Hang Coat), the hand poses required to perform specific actions in specific regions, can generally be treated as consistent. Moreover, the deformable nature of garments allows them to adapt naturally to the dexterous hand's pose during manipulation. Therefore, for each specific task, we first define a set of task-specific hand poses (the number depends on the task; for example, Hang Coat requires both closed grasping pose for the collar and an open pose for release). These poses are generated via teleoperation using LeapMotion, with physical feasibility considered during generation. In this way, the pose definition process inherently includes a manual filtering step to select the most suitable poses.
These task-specific hand poses are then transferred directly into the data collection process for that task category. That is, during data collection across different garments within the same task, the same predefined hand poses are reused.
retargeting with optimization
Finger Poses
Shadow Hand joints
Wrist Position
Palm Direction
Lcap.Motion Detector
Closc
Open
Denal swarge
Real-World Visualization
Performance
the
oleggh
Hand Model
OPEN
CLOSE
PINCH
Figure 11: LeapMotion Workflow and Real-World Visualization Performance.
C Advantages of Dexterous Hands over Parallel Grippers
Due to current restrictions in simulation technology, certain complex tasks cannot yet be effectively implemented in simulated environments. However, across the existing tasks, dexterous hands still demonstrate clear advantages over parallel grippers in several aspects.
Our experimental results, particularly those real-world results, reveal significant limitations in two- finger grippers when handling thin flat regions of garments. In such cases, the gripper frequently
23

---

fails to achieve a stable grasp, with garments either slipping during manipulation or not being grasped all. Many existing demonstrations involve garment folding with two-finger grippers rely on grasping the garment's s boundary, but these strategies also often leads to repeated failed attempts. In contrast, our system employing five-finger dexterous hand demonstrates clear advantages. Owing to the hand's 's larger operation range, greater surface contact, and coordinated multi-finger control, it can stably grasp various regions garment-nol just its edges-aid maintain firm hold throughout the manipulation process. This capability greatly enhances the flexibility and robustness of robotic garment handling, freeing the system from constraints on graspable areas predefined strategies.
Furthermore, the diverse hand postures offered by the dexterous hand allow us tailor the manipulation strategy to the specific requirements of each task.
to accomplish a fine-grained task such as putting on a glove, the hand can adopt a pinch posture precisely grasp the glove edge.
For stable garment back-and-place operations, an open/close posture can be used to maximize contact and grip.
When placing folded clothes, a cradle posture helps preserve the folded structure.
To smooth wrinkles, the hand can switch to a smoothening pose that maximizes surface coverage.
These example poses are shown in the bottom-left corner of Figure (main paper).
The dexterous hand not only replicates the functionality of two-finger grippers but also enables finer and more robust manipulation of garments.
3. Although current simulation still faces significant challenges in accurately modeling interac- tions between deformable objects and dexterous hands linitingmiting its ability to simulate complex these limitations of computer graphics and robotics technologies. As a result, the advantages dexterous hands will become even more prominent in complex tasks such as assistive dressing or knot tying, which require flexible coordination among multiple fingers. We will continue to extend tasks and optimize the simulation framework based on DexGarmentl ab, aiming support wider range of more complex dextcrous-ham garment interaction tasks.
expected to be gradually addressed with the continued advancement
D Recorded Information in Automated Data Collection
In this section, we provide detailed overview of the various types of data collected during our automated data collection process. These include:
joint state: The joint values of the dual-arm dexterous system, including the left arm and right arm (6 joints each), left hand and right hand (24 joints each). The shape is (60, ).
image: RGB images of the workspace. The shape is (480, 640, 3).
env enint cloud: Point cloud of the workspace with color, downsampled 2048. The shape is (2048, 6).
lttptttoooooottt:ttttetteeeeeeet The shape is (2048, 3).
prrmentt_ponnt cloud: Point cloud of the garment without color, downsampled to 2048.
points affordance feature: The left / right point affordance feature generated by GAM, which can be seen similarity score (normalized to [0, The shape is (2048, 2).
point_ccood:: Point cloud of the ieapoommennn-nterattiiii object without color, downsampled to 2048, only exist in (2048, 3).
bhject_pointt cloud:
Interaationnn tasks. The shape is
It is worth noting that during data collection, we record all the aforementioned information every five time steps. Empirical validation shows that this approach not only reduces data density and accelerates policy training, but also does not compromise overall performance during validation.

---

E Real-World Experiment Scene
Our real-world experiment scene comprises RealMan RM75-6F (Arms) with Psibot GO-R (Dexterous Hands), a RealSense D435 camer and few garments across different categories, which is shown in Fig. 12
Head Camera (RealSense D435)
6-DoF Robotic Hand (PsiBot GO-R)
7-DoF Robotic Arm (RealMan RM75-6F)
Object Set &
Manipulation Platform
Figure 12: Real-World Experiment Scene
Comparison with Teleoperation Data
Table 5: Automated Data Collection Time and Success Rate Across All Tasks
Task Name
Collection Success Rate
Single Collection Time
Task Name
Collection Success Rate
Single Collection
Time
Fling Dress
92.6% (100/108)
56
Hang Trousers
99.0% (100/101)
min
Fling Tops
90.1% (100/111)
0 min 55
Store Tops
98.0% (100/102)
min 05
Fling Trousers
90.1% (100/111)
O min 48
Wear Basehall Cap
85.5% (100/117)
0 min
Dress
84.7% (100/118)
Scarf
(100/120)
min
Fold Tops
82.0% (100/122)
I min 08
Wear Bowl Hat
95.2% (100/105)
0 min 53
Fold Trousers
82.6%
(100/121)
0 min 56
Fold Tops Worldd
90.9% (50/55)
0 min 50
Hang Coats
92.6% (100/108)
min 41
Hang Tops (oold
96.2% (50/51)
min
Hang Dress
90.9% (100/110)
48
Wear Scarf (Rcal-World)
83.3% (50/60)
min 15
Hang Tops
91.7% (100/109)
0 min 46
Wear Hat (Real World)
93.8% (50/53)
0 min 36
Tab. 5 presents the data collection time and success rate across all tasks, which validates the effective- ness and efficiency of the automated data collection pipeline. To further verify the quality of data acquired via automation, three representative tasks were selected for data collection, model training. and evaluation, using both teleoperation data and autonomously collected data. Specifically, teleoper- ation data in simulation was gathered using LeapMotion, while an exoskeleton device was employed for werldworld teleoperation data collection. The performance of policies trained teleoperation data was then compared with that of policies trained on autonomously collected data, with the results summarized in Tab. 6
Experimental results indicate that policies trained on teleoperated and autonomously collected data achieve comparable performance during evaluation. However, from the perspective of human
25

---

Table 6: Performance Comparison of HALO with Different Data Sources
Method (Data Source)
Hang Tops (Simulation)
Bowthat (Simulation) Fold Tops (Real-World)
HALO (Automated Collection Data)
0.92:0.04
0.72:0.04
13/15 15
HALO (Teleoperation Data)
0.88+0.03
0.70+0.06
13/15
and time cost, teleoperation-bant based data collection significantly more labor-intensive and time- consuming than automated data collection. is also important note that our data is not synthetically generated, but collected through real executions in either simulation or the real world, which means the demonstrations are physically realistic. The entire data collection pipeline is enabled by model inference and one-shot demonstrations.
G Additional Baseline Comparison
We have additionally included four new baselines for comparison: ACT (IL), pio (VLA), RDT (VLA), and Eureka (RL+VLM). We selected representative tasks for evaluation, including Fling Dress, Fold Trousers, Hang Coat, and Wear Bowlhat in simulation, well as Fold Tops in real world. The evaluation protocol remains consistent with the original paper:
For simulation tasks, each task is evaluated over 50 episodes using three different random seeds. We report the success rates as Mean + Std across all trials.
For real-world task, we evaluate using distinct garments per category, each tested under different initial deformations. We report the success rates as successful_trials all_trials.
Table Performance Comparison of More Baseline Methods on Typical Tasks
Method
Fling Dress (Simulation)
Fold Trousers (Simulation)
Hang Coat (Simulation)
Wear Bowlhat
Fold Tops
(Simulation)
(Real World)
DP (IL)
0.59=0.05
0.47+0.04
0.52+0.04
0.41
9/15
DP3 (II.)
0.51+0.03
0.54+0.07
0.58+0.04
0.55+0.04
8/15
ACT (IL.)
0.35+0.02
0.49=0.06
0.43+0.04
0.51=0.03
7/15
piO (VLA)
0.69+0.01
0.52+0.06
0.72+0.01
0.59+0.02
10/15
RDT (VLA)
0.60#0.02
0.58+0.02
0.62+0.02
0.48+0.01
9/15
Eureka (RL+VLM)
/
/
0.16+0.03
0.08+0.02
/
HALO (Ours)
0.82=0.06
0.77+0.02
0.90=0.01
0.72+0.04
13/15
Analysis For Emitation-Learming--
Methods:
We collect 100 demonstrations for each simulation task and collect demonstrations for each real-world task, which are used for policy training (DP/DP3/ACT). A notable limitation of their performance lies in the inability to accurately grasp the target region of the garment, as well as the lack of fine-grane fine-grained control during placement based the garment's shape(/state).
Analysis for VLA Methods:
We combine all the demonstrations of simulation and rorl-world tasks to fine-tune piO and RDT model. In terms of final performance, pretrained models such as piO and RDT outperform from- scratch approaches like DP, DP3, and ACT. However, compared to HALO, VLA-based models still exhibit clear limitations in accurately perceiving garment shape and state, and in executing precise grasp-and-place actions, resulting in noticeable performance gap relative to HALO.
Analysis for RL-Based Methods:
We selected Eureka the representative RL- VLM-combined method and conducted preliminary experiments several tasks in simulation. While we observed some initial successful cases on sim- pler tasks such as Hang Coat and Wear Bowlhat, the overall success rate remained low. Additionally, due to the lack of robust parallelization support in the current Isaac Sim environment and the limited effectiveness of reward functions in handling long-horizon tasks like Fling and Fold, this baseline did not yield valid results across all tasks. Nevertheless, we are actively developing multi-parallel deformable object manipulation environment based on Isaac Lab, and RI-based approaches for deformable manipulation will be a key focus of our future work.
26

---

H Training Details of Main Algorithm
H.1 Garment Affordance Model (GAM)
Hlpee-Parameeers Selection. The training of GAM follows the hyperparameter settings used in the UniGarmentManip framework. we set the number of skeleton pairs to be 50 and batch size to be 32. In each batch, sample 32 garment pairs. For each garment pair, we sample 20 positive and 150 negative point pairs for each positive point pair. Therefore, in each batch, 32 x 32 x 20 data will be used update the model. During the Correspondence training stage, we train the model for 40,000 batches. "Coarse-to-fine Refinement' and 'Few-shot Adaptation' mentioned in UniGarmentManip are also adopted for improving GAM's performance.
Computational Resource. we use PyTorch as our Deep Learning framework. Each experiment is conducted RTX 4090 GPU, and consumes about 22 GB GPU Memory for training. It takes about 20 hours to train the Coarse Stage, with 1-2 hours of Coarse-to-Fine Refinement and 0.5 hour's Few-Shot Adaptation.
We trained GAM checkpoints for eight types of garments, including: Bascball Capp Dress LongSleeve, Glove, Scarf, Tops FrontOpen, Tops LongSleeve, Jops_NooSees and Trousers. We release all these checkpoints in our code repo
H.2 Structure-Aware Diffusion Policy (SADP)
Hyper-Parameters Selection. We set the horizon, observation _tepssttepssssss__sss and action ssspssstpppsppppppp to 8, 3, and 4. respectively. The number of denoising steps in the diffusion process is set 10. The model is trained for total of 3000 epochs, with validation performed every 25 epochs and checkpoints saved every 100 epochs. We use the AdamW optimizer with initial learning rate of I x 10-4 and adopt a cosine learning rate scheduler with 500 warm-up steps. During dataset loading, the test set is configured comprise 2% of the entire training dataset.
Computational Resource. We use PyTorch as our deep learning framework. All experiments are conducted NVIDIA A800 GPU, with approximately 75 GB of GPU memory consumption when training with a batch size of 200. The training process takes around 16 hours to complete 3000 epochs. However, checkpoints from earlier epochs be selected for validation if desired.
We release two variants of our method: SADP and SADP G. SADP is designed for Garment- cntiiiommen--nteractiii Tasks, where the encoder incorporates the point cloud of environment- interaction objects. contrast, SADP G is tailored for Garment-SSe Sef-rt--Se----tratttiin Tasks, where the encoder excludes the point cloud of environment-interaan interaction objects. We provide detailed tutorials for both SADP and SADP G in the released codebase facilitate easy usage and integration.
I Limitation
The manipulation of deformable objects (garments) is a highly challenging field. DexGarmentl provides an initial environment, along with data collection and policy training methods, to facilitate advancements in this domain. However, we must acknowledge that DexGarmentLab still has several limitations, which will be analyzed detail as follows.
I.1 Garment Simulation Method Limitation
In DexGarmentLa.. We employ Sirtion--Bsed--ynaaaa (PBD) and to simulate garment. However, both PBD and FEM have different limitations to accurately simulate the real-performance of fabric.
(FEM)
L.11 Position-Basee Based--ynamidee (PBD)
In the simulator, garments modeled using Position-Basse Dynamics (PBD) are represented as collection of discrete particles, as shown in Fig. 13 This approach effectively captures the softness of fabric, facilitating operations such as folding and deformation. For this reason, we employ PBD to simulate larger and highly deformable garments, such as tops, trousers, dresses, etc.
27

---

Position-Bascd Dynamics (PBD)
Draf Effect PBD
Finite-Element Method (FEM)
Drag Offect FEM
Figure 13: Garment Simulation Method
However, since the particles not tightly connected and contain numerous gaps (shown in Fig. and self-collision interactions between particles can easily lead to instability in their states (This insta- bility manifests continuous jittering of the garment.), when manipulating the garment, penetration artifacts may occur, and maintaining a stable configuration can challenging.
In DexGarmentLab, we designed set of carefully tuned parameters for PBD simulation tailored to the provided assets, aiming mitigate these issues. While some unrealistic garment behaviors may still persist, our approach enables relatively reasonable approximation of garment performance.
L.12 Finite-Element-Methedd (FEM)
In the simulator, garments modeled using Finite Element Method (FEM) are represented as a collection of discrete blocks, shown in Figure [3] This approach effectively simulates deformable and elastic objects but does not accurately capture garment deformations. When garments undergo deformation in the simulator, they tend to revert to their original shape easily, phenomenon that is particularly pronounced for larger garments. As shown in Fig. 13 when garments modeled using FEM dragged, the discrete blocks used for simulation remain tightly connected and do separate. Besides, the garment tends to preserve its original shape throughout the dragging process.
In Da, employ FEM to simulate garments such as gloves and hats, which do not exhibit significant deformations in the real world and typically maintain relatively stable shapes. Within the corresponding tasks, these garments demonstrate highly realistic physical properties.
1.2 Environment Task Limitation
Although we propose 15 different tasks, including garmnet-self-inneraatoo tasks and garment- environment-interaciin tasks, these tasks all belong single-garment tasks, while multi-garments tasks are not introduced.
What's more, in simulation environment, utilize two independent robotic arms equipped with dexterous hands (UR10e + ShadowHand). The tasks in our scenarios do not involve the movement of robotic system. However, for real-world household applications, more suitable robotic platform would be a dual-arm mobile robot with a wheeled base. Nevertheless, our proposed method is applicable to various types of dual-arm robotic systems.
The aforementioned environment task limitations are fundamentally challenging issues the field of deformable object manipulation and even in robotics whole. We look forward to future researchers building upon DexGarmentl.an to continuously address these challenges and advance the development of deformable object manipulation.
28

---

1.3 Generalizable Policy Limitation
Based on both simulation and real-world experimental results, we summarize the following points regarding policy limitation:
1. For garments with occluded or highly deformed regions, the Garment Affordance Model (GAM) does not always predict manipulation points accurately. This can lead to failed or suboptimal grasps by the dexterous hand, impacting the final performance. Nonetheless, GAM performs reliably most garments. For challenging cases, an initial unfold action is recommended to expose key regions.
2. The policy struggles with garments that deviate from standard geometries, such asymmetrical designs (e.g., single-sleeved tops) or heavily adorned costumes, which distort the point cloud and affect both GAM and SADP performance.
J DexGarmentLall Assets
Garment Assets. We select garments from ClothesNet 451, a large-scale dataset of 3D clothes objects with information-rict annotations. We select garments from categories (including Tops, Dress, Trousers, Hat, Scarf, etc.) and use two physical simulation method (FEM and PBD) to simulate them.
Robots Assets. The dual-arm robot used in DexGarmentLab consists of two ShadowHands mounted on UR10e robotic arms. Leveraging the URDF files provided by [. we integrated the ShadowHand and UR10e URDFs and converted the combined URDF into USD file using NVIDIA Isaac Sim. Users can customize the robot configuration through the provided URDF file to suit their specific tasks or simulation setups.
Environment- Inntrrattint Assets. The anvironmenntinteractioo assets used in DexGarment- Lab mainly includes: hanger, pothook, placement platform, human. The hanger, pothook and placement platform are created using basic components (for example, cube, capsule, etc.) supplied by Isaac Sim, while human model are obtained from Onniverse Base Asset library.
Material Assets. Materials are crucial components of virtual relightable assets, defining the interaction of light at the surface of geometries. Our materials are primarily sourced from two repositories: a selection from Omniverse Base Material Library and additional assets obtained from https://mmbienteg.com garments, grounds, and other scene elements.
These materials are mainly used for simulating
K Detailed Task Description
As mentioned in the paper, divide all the tasks into two categories: Task and Garment-Emin EenvrooneentInneraeee Task.
Garment- Self--eeee---etiinn Tasks include:
Fold Tops (Sec.
K.T Fold Dress (Sec. K.2 2
Fold Trousers (Sec. K.3 Fling Tops (Sec. K.4 .4
Fling
Dress (Sec. K.5 Fling Trousers (Sec. K.6
Garment-Envineg www.ree--niiooom----neeerraaeeee
Tasks include:
Hang Coat (Sec. K.7 Hang Tops (Sec. K.8 Hang Dress (Sec. K.9
Hang Trousers (Sec. K.10
Store Tops (Sec. 1. Wear Baseball Cap (Sec. K.12), Wear Bowl Hat (Sec. K.13), Wear Scarf
(Sec. K.14). Wear Glove (Sec. K.15).
In this section, we will provide detailed description each selected task, covering task initialization and randomization, task sequences, task success metrics, garment assets for task and other related aspects. These details will be introduced in separate subsections for clarity.
K.1
Fold Tops
K.1.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 14 (a).
29

---

Sage Gareaa Cesler Peslner Nanalenncaiie
stage
stuge 2
stage 3
BAT
task initialization randomization
sequence
(c)
Figure 14: Fold Tops Task.
As for initialization, the garment is initialized in a flat state, which means that the initial orientation of the garment is set to (0.0, 0.0, 0.0). The positions of the left and right robots are set to (-0.8, 0.0, 0.5) and (0.8, 0.0, 0.5) respectively, with both orientations initialized to (0.0, 0.0, 0.0).
During data collection and policy evaluation, randomly selected the position of the garment within a certain range. For this task, the initial position of the tops is randomized within a rectangular area defined by -0.10<x<0..1 10 and 0.70<y<0.90.
should be noted that the randomization range for the position refers to the location of garment's s center, and the unit distance is meter. Additionally, the initial orientation is set as culer angles. All descriptions in the following sections follow this convention.
K.1.2 Task Sequence
As shown in Fig. [4 (b), the sequence of Fold Tops consists of three stages. First, fold the left sleeve the right, then fold the right sleeve to the left, and finally grab the corners of the garment and fold them upward to complete the fold.
K.1.3 Task Success Metrics
The garment is initially in flat state. First, Garment Affordance Model (GAM) is used to locate four key points: the left and right sleeve points, as well as the collar and bottom of garment. Then, the left and right boundaries are determined by the points on the left and right sleeves, and the central boundary is determined by the points on the collar and bottom. The collar and center are then used as the upper and lower boundaries, respectively, forming rectangular area, shown in Fig. 14 (c). The success of the fold is determined by checking whether the final state lies within this If the garment region within the covers than 85% of the total garment area, the 'fold' manipulation is considered successful. To ensure data validity, images of the final state were also recorded and manually checked.
K.1.4 Garment Assets for Task
For this task, the garments used are long-sleeved tops, a total of 247 pieces. During data collection, we first selected 100 garments and then randomly chose from these 100 garments for data collection. During policy randomization, we randomly select from all 247 garments ensure that the validation set contains data that was not seen during training.
K.2
Fold Dress
K.2.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 15 (a).
The initialization configuration of Fold Dress task is the same with those of Fold Tops. As for randomization, the initial position of the dress is randomized within rectangular area defined by
-0. 00.00x<0.10 and 0.65<y<0.90.
30

---

Carmen Caster Pollia Randemaatten
stago
stage 2
Di (.3
stage 3
initialization aadomization
(b) sequence
(c) metric
Figure 15: Fold Dress Task.
K.2.2 Task Sequence
As shown in Fig. (b), the sequence of Fold Dress also consists of three stages. First, fold the left sleeve the right, then fold the right sleeve to the left, and finally, since dress is usually longer, we choose to grab the waist of the dress (instead of the skirt hem) and fold upward.
K.2.3 Task Success Metrics
The success criteria for this task are the same as for Tops. Four points are selected form rectangular area, as shown in Fig. (c), and the folded garment is checked to see if it lies within this area. the garment region within the covers more than 80% of the total garment area, the 'fold' manipulation is considered successful.
K.2.4 Garment Assets for Task
The garments used in this task are long-sleeved dresses, with total of 38 pieces. During data collection, we first randomly select 18 dresses and then randomly choose from these 26 for data collection. the validation experiments, dresses are randomly selected from all 38 pieces for the validation process.
K.3 Fold Trousers
inttalzzaiinn modocizzion
talk
Figure 16: Fold Trousers Task.
K.3.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 16 (a).
For the initialization of Fold Trousers, the trousers need to be laid horizontally on the ground, so the orientation of the garment is set to (0.0, 0.0, 90.0). As for randomization. initial position of trousers is randomized within rectangular area defined by -0.10xx<0..1 10 and 0.70cy<0.90.
K.3.2 Task Sequence
The sequence of Fold Trousers consists two stages, as shown in Fig. 16 (b). First, fold the trousers along the center axis, then fold the pant legs towards the waisthand to complete the task.
K.3.3 Task Success Metrics
Similar to the previous two folding tasks, three points are selected, as shown Fig. T6 (c). The boundaries of the folded garment calculated based on the positions of these three points, and
31

---

the success of the task is determined by checking whether the folded garment lies within this area. If the garment region within the area covers more than 85% of the total garment area, the 'fold' manipulation is considered successful.
K.3.4 Garment Assets for Task
The Fold Trousers task uses total of 317 trousers. During data collection, 100 pieces are randomly selected from the 317, and then randomly chosen from these 100 for data collection. In validation experiments, garments are randomly selected from all 317 pieces for validation.
K.4
Fling Tops
Sape Caper Tudlxo Xebmicai
Lat 1444
Z
65"--00"
(a) lask initialization and randomization
lask metric
stage 2
(b) task sequence
Figure 17: Fling Tops Task.
K.4.1 Task Initialization and Randomization
Task Configuration is shown in Fig. 17 (a).
For the Fling task, the garment should initially be in wrinkled state. Therefore, inclination angle and certain height are given at the start to ensure that the garment lands in a stacked state. For this task, the inclination angle is set to 65-80 degrees, i.e., the initial orientation is (65.0-80.0, 0.0, 0.0), as shown in Fig. (a).
As for position randomization, the initial position of the tops is randomized within rectangular area defined by 10<x<0. 10 and 0.50<y<0.70.
K.4.2 Task Sequence
As shown in Fig. 17 (b), the Fling Tops task consists of two stages. First, grab the shoulders of the top, lift up, and extend hands forward and downward to flatten the body part of the top. Then, grab both sleeves and pull them outward to flatten the sleeves.
K.4.3 Task Success Metrics
For the Fling task, we use an area-based judgment method. A matching garment placed different location in the environment in flat, non-tilted state. In the final judgment, images
32

---

captured by cameras positioned at the same height directly above both the Fling garment and the garment used for comparison, as shown Fig. (c). Then, the proportion of the area occupied by the garments in the images is calculated. If the difference in area proportions is smaller than certain threshold (here we set 0.2), indicates that the area of the Fling garment close to that of the flat garment, and the Fling is considered successful.
K.4.4 Garment Assets for Task
The assets and partitioning method used in this task are the same as those in the Fold Tops task mentioned earlier.
K.S
Fling Dress
Garaerr Tosiisx Kukeninai
stage
en rrietatiin stange
(a) task initialization and randomization
(b) task sequence
Figure 18: Fling Dress Task.
K.5.1 Task Initialization and Randomization
Task Configuration is shown in Fig. 18 (a).
In this task, the orientation randomization of the dress is also set (65.0-80.0, 0.0, 0.0). As for position randomization, the initial position of the dress is randomized within the range -0. 10xxxxxx0010 and 0.50<y<0.70.
K.5.2 Task Sequence
The sequence of Fling Dress is essentially the same as that of Fling Tops, as shown in Fig. 18 (b).
K.5.3
Task Success Metrics
The same area-based judgment method as described in the Fling Tops task is also applied in this task.
K.5.4
Garment Assets for Task
The assets and partitioning method used in this task are the same as those in the Fold Dress task mentioned earlier.
K.6 Fling Trousers
[n) tekk Initaalosrion
(b) tsk sequenze
randemization
Figure 19: Fling Trousers Task.
33

---

K.6.1 Task Initialization and Randomization
Task Configuration is shown in Fig. 19 (a).
In this task, the orientation randomization of the trousers is also set (65.0-80.0, 0.0, 0.0). As for position randomization, the initial position of the trousers is randomized within the range- 0. 0.10<x<0.10 10 and 0.50<y<0.70.
K.6.2 Task Sequence
The Fling Trousers task consists of only one stage: grab the waistband and lift it up, then extend hands forward and downward to flatten the trousers, as shown in Fig. 19 (b).
K.6.3 Task Success Metries
The assets and partitioning method used this task are the same those the Fold Trousers task mentioned earlier.
K.6.4 Garment Assets for Task
The same area-based judgment method as described in the Fling Tops task also applied in this task.
K.7
Hang Coat
(x) takk tritializaiiu and
mndomisstion
Figure 20: Hang Coat Task.
K.7.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 20 (a).
This task involves randomization not only in the position of garments but also in the position of anvironment-interaciin objects. The initial position of the coats is randomized within the range -0.10<x<0..1 10 and 0.50<y<<0.70 The initial position of the pothook is randomized within the range -0.25<xx0.25 and 0.685<y<00933..
K.7.2
Task Sequence
The Hang Coat task consists of stage: grab the coat's lapels and lift the garment up the pothook, as shown in Fig. (b). The lifting height and placement position vary depending on the shape of the garment and the location of the pothook.
K.7.3 Task Success Metrics
The ideal final state of this task is that the garment is stably hung the pothook without falling off. Therefore, we determine task success based on the center position the garment. Specifically, if the z-coordinate of the garment's center is greater than 0.5 (to exclude cases where it has fallen to the ground) and less than 2.0 (to exclude abnormal simulation states), the task is considered successful.
K.7.4 Garment Assets for Task
The garments used in the Open Coats task are front opening tops, with a total of 78 garments. During data collection, 40 pieces of the garments are randomly selected, and then random subset of these 40 garments chosen for data collection. In the validation experiments, garments are randomly selected from all 78 pieces for validation.
34

---

K.8
Hang Tops
(a] task Initiallaiion
mindmmsssiio
(b) usk sequence
Figure 21: Hang Tops Task.
K.8.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 21
This task involves randomization not only the position of garments but also in the position of environmenttinnerratioon objects. The initial position of the tops is randomized within the range -0.10<x<0. and 0.50<y<0.80. The initial position the hanger randomized within the range -0.15<x<0.1 15 and 0.80<y<0.90.
K.8.2
Task Sequence
The sequence of Hang Tops shown Figure (b). First, grab the shoulders of the top and lift the garment, then move it forward to drape the top over the hanger. The lifting height and placement position vary depending on the shape of the garment and the location the hanger.
K.8.3 Task Success Metrics
For the Hang task, the garment should ultimately be hanging on the hanger, with all parts of the garment above a certain height. Therefore, we determine the success of the Hang task by checking whether all points of the garment's 's point cloud are above this height. Additionally, we also check whether the distance between the center of the garment and the center of the hanger falls within certain range, in order to exclude cases where the garment is not hung near the central region of the rack.
K.8.4 Garment Assets for Task
The assets and partitioning method used in this task are the same as those in the Fold Tops task mentioned earlier.
K.9 Hang Dress
(e) tade Initalization
mandomisaiion
(b) usk sequence
Figure 22: Hang Dress Task.
K.9.1 Task Initialization and Randomization
Task Configuration is shown in Fig. (a).
This task involves randomization not only the position of garments but also in the position of environmenttinteracinn objects. The initial position of the dress is randomized within the range -0.10<x<0010 10 and 0.50<y<0.800 The initial position of the hanger randomized within the range -0.15<x<0.15 and 0.80<y<0.90.
35

---

K.9.2
Task Sequence
The sequence of Hang Dress is the same that of Hang Tops, shown in Figure 22 (b). The lifting height and placement position vary depending on the shape the garment and the location of the hanger.
K.9.3
Task Success Metrics
The metric for Hang Dress is the same as that for Hang Tops.
K.9.4
Garment Assets for Task
The assets and partitioning method used in this task are the same as those in the Fold Dress task mentioned earlier.
K.10 Hang Trousers
[n] takk initallosiion and
mndominntion
(b) tank scquanece
Figure 23; Hang Trousers Task.
K.10.1 Task Initialization and Randomization
Task Configuration is shown in Fig. (a).
This task involves randomization not only the position of garments but also in the position of muvironment-interaae
objects. The initial position of the dress is randomized within the range -0.10<x<0.10 10 and 0.50<y<0.80 The initial position of the hanger randomized within the range -0.15<x<0.11 15 and 0.80<y<0.90.
K.10.2 Task Sequence
The process of Hang Trousers is shown in Fig. 23 (b). Grab the waistband and then follow the same steps as for Hang Tops and Dress. The lifting height and placement position vary depending on the shape of the garment and the location of the hanger.
K.10.3 Task Success Metrics
The metric for Hang Trousers the same as that for Hang Tops.
K.10.4
Garment Assets for Task
The assets and partitioning method used this task are the same those the Fold Trousers task mentioned earlier.
K.11
Store Tops
K.11.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 24 (a).
This task involves randomization not only the position of garments but also in the position of environment-interactenn objects. The initial position of the tops is randomized within the range -0.05<x<0005 and 0.65<y<00755. The initial position the placement platform randomized within the range -0.30<x<0.30 and 1.00<y<1.20.
36

---

stage Siperazion
stage
Stape Object Centee Poslion Rendvaization
aa) task initialization and randomization
(b) task sequence
Figure 24: Store Tops Task.
K.11.2 Task Sequence
The sequence of Store Tops is divided into tongepprepartton and stage_1, shown in Figure 24 (b). In prge__prparation, grab the shoulders and fold the garment towards the corners in order to make garment ready. Then in stage I, use both hands grab the corners and place the garment the corner of placement platform.
K.11.3 Task Success Metrics
The goal of this task is to place the folded garment at the exact center of the placement platform. Therefore, after the task is completed, we load camera above the garment capture point cloud of its final state. The center position of the garment is then computed from the point cloud coordinates and compared with the center position of the placement platform, which is accessible the simulator. If the distance between the two centers is smaller than predefined threshold (set to 0.1 in this case), the task is considered successful.
K.11.4 Garment Assets for Task
The garments used for the Store task are sleeveless tops, with a total of 217 pieces. During data collection, 100 pieces are randomly selected from these 217 garments. In validation experiments, garments are randomly selected from all 217 pieces.
K.12 Wear Baseball Cap
stage
(n) tak initallosrion and madomisaaiion
(b) takk sequence
Figure 25: Wear Baseball Cap Task.
K.12.1 Task Initialization and Randomization
Task Configuration is shown in Fig. 25 (a).
This task involves randomization not only the position of garments but also in the position of environmenttinteractioon objects. The initial position of the baseball cap is randomized within the range 10xxx00x01000 and 0.70<y<0.90. The initial position of the human is randomized within the range -0. 15 and 1.10<y<1.20.
37

---

K.12.2 Task Sequence
The sequence of Wear Baseball Cap is shown in Figure 25 (b). First, use the right hand to grab the brim of the cap, then position the cap and place it on the top the head. The lifting height and placement position vary depending on the shape of the garment and the location of the human.
K.12.3 Task Success Metrics
The final goal of wearing the cap is for it to be placed on top of the head. Therefore, the success criterion is based on whether the distance between the final center of the cap and the top of the head is within a certain threshold.
K.12.4 Garment Assets for Task
There are 12 baseball caps in total. During data collection and policy validation, we all use 12 pieces of caps for random selection.
K.13 Wear Bowl Hat
(o] tak teliatiation and mnnermossooon
(6) tak suparcce
Figure 26: Wear Bowl Hat Task.
K.13.1 Task Initialization and Randomization
Task Configuration is shown in Fig. 26 (a).
This task involves randomization not only in the position of garments but also in the position of unvironment-interaciioo objects. The initial position of the bowl hat is randomized within the range -0.10<x<0,, 10 and 0. 0.70ky<0.90. The initial position the human randomized within the range -0.15<xx0.11 15 and 1.10<y<1.20.
K.13.2
Task Sequence
The sequence of Wear Bowl Hat is done with both hands. First, use both hands to grab the left and right sides of the hat from below, then align it with the head, and gently place on the head. The entire process is shown in Figure (b). The lifting height and placement position vary depending on shape of the garment and the location of the human.
K.13.3 Task Success Metrics
The same the criterion for wearing baseball cap.
K.13.4 Garment Assets for Task
There are 8 bowl hats in total. During data collection and policy validation, we all use 8 pieces of hat for random selection.
K.14
Wear Scarf
K.14.1
Task Initialization and Randomization
Task Configuration is shown in Fig. 27 (a).
This task involves randomization not only the position of garments but also in the position of anvionmeent-ineraacinn objects. The initial position of the scarf is randomized within the range
38

---

Soapee Garaant Center Fosiis Kandvnizuine
Sage Ofgee Canrr Polin Kenimttoiin
(a) task initialization and randomization
task sequence
Figure 27: Wear Scarf Task.
-0.05<x<0.05 and 0.30<y<0.40, The initial position of the human is randomized within the range -0.05<x<0.05 and 0.85<y<1.00.
K.14.2 Task Sequence
The sequence of Wear Searf is shown in Figure 27 (b). First, grab both ends of the scarf and drape it around the neck. Then, use the right hand to grab the right end and wrap it around the neck to complete the task. The manipulation position and placement position vary depending on the shape of the garment and the location of the human.
K.14.3 Task Success Metries
For the scarf-wearing task, the scarf is not successfully placed around the person's 's neck, will droop to the ground. To detect this, place camera in front of and behind the human model to capture point clouds the scarf. the number of points below a certain height threshold (set to 0.2 meters) exceeds predefined count threshold (set 20) in both front and rear point clouds, consider the searf to be drooping on the ground and thus regard the task as a failure. Otherwise, the task is considered successful.
K.14.4 Garment Assets for Task
We use 8 types of scarf for data collection and randomization, which have different length (0.35m, 0.36m, 0.37m, 0.38m, 0.39m, 0.40m, 0.41m, 0.42m, respectively).
K.15 Wear Glove
Figure 28:
Wear Glove Task.
The sequence of the Wear Gloves task is shown in Figure 28 First, the index fingers of both hands to insert into the gloves and pull them apart. Then, move the gloves forward and put them on the hands.
Due the small size of the glove and the requirement for precise insertion of the index finger, followed by pinching with the thumb to grasp the side of the glove, achieving this effect is challenging with the current FEM simulation and dexterous hand manipulation. Therefore, we employed the attachment block method to accomplish the wear gloves task, and did not implement randomization of the initial position for this task, nor did we validate through policy evaluation. We look forward future work enabling dexterous hands to better handle such extremely fine-gra ane-grained tasks.
39

---

K.16 Additional Explanation for Success Metric
Although we have designed specific success metrics for each task, and our validation shows that these metrics can reliably determine task success or failure in most cases, the inherently complex nature of garment states makes it difficult to judge success purely based on the final state in certain situations. Therefore, during both data collection and policy validation, we record the final state of the garment as well as a full video of each episode. During data collection, users can review the final state images to identify potentially abnormal episodes and use the corresponding videos to decide whether keep the data. Similarly, during policy validation, users can apply the same approach to avoid misjudgments in result evaluation.
L Broader Impact
Our work presents dexterous garment manipulation environment, with particular focus on coordi- nated dual-arm dexterous hands. This effort advances the field of deformable object manipulation and lays solid foundation both in simulation and algorithmic development- for future progress in general-purpose home-assistant robotics involving deformable object handling. We haven't 't observed negative potential impacts.
40


Reviews:
[{"review_id": "3FrDp9bbJ1", "reviewer": "Reviewer_JjAQ", "summary": "This paper introduces DexGarmentLab, a novel simulation environment specifically designed for dexterous (especially bimanual) garment manipulation. The environment provides large-scale high-quality 3D 8 assets for 15 task scenarios, and refines simulation techniques tailored for garment 9 modeling to reduce the sim-to-real gap. To address the bottleneck of labor-intensive data collection, the authors propose an automated pipeline that leverages a single expert demonstration and a GAM to generate diverse manipulation trajectories for novel garments. Furthermore, they propose HALO, which combines affordance point inference and a structure-aware diffusion policy to Generate Generalizable Trajectories. Extensive experiments show that the methods outperforms baselines, and generalizing well to unseen garments,shape,deformation,env config.", "questions": "1.\tYour method uses five-finger dexterous hands instead of common two-finger grippers. What specific advantages do five fingers provide for garment manipulation tasks in your experiments?\n2.\tAre the grasp points on the garment defined in 2D (on the surface) or 3D coordinates?\n3.\tSince garments are flexible, did you consider grasping force or pressure to avoid slippage or fabric damage?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.\tThe integration of GAM and structure-aware diffusion policy (HALO) is novel and provides a principled way to generalize manipulation across unseen garments and shapes.\n2.\tThe proposed environment and methods are highly relevant for real-world applications in robotics, such as laundry folding, dressing, and assistive robotics.\n3.  The figures and tables are of high quality, with clear and informative visualizations that greatly aid understanding.\n\nWeaknesses\n\n1.\tIn the real-world experiments, particularly on the Fold Tops and Hang Tops tasks, the garments used appear to have relatively simple and uniform colors. It is unclear how well the proposed method would generalize to garments with more complex patterns, prints, or highly varied textures, which are common in real-world scenarios.\n2.\tIn Table 3, please clarify what metric is being reported (e.g., is it the number of successful trials out of total attempts, or another measure?).", "was_revised": true, "final_justification": "Many thanks to authors for their detailed responses. They raised various good points, although I keep my score the same, as it was on the positive side.", "rebuttal": "# Rebuttal for JjAQ\nThanks for your helpful and meaningful reviews! We will provide a detailed response below to the issues outlined in the Weakness and Question sections.\n\n---\n\n## **Weakness1:**\n\n> In the real-world experiments, particularly on the Fold Tops and Hang Tops tasks, the garments used appear to have relatively simple and uniform colors. It is unclear how well the proposed method would generalize to garments with more complex patterns, prints, or highly varied textures, which are common in real-world scenarios.\n\n## **Answer:**\n\nFor our policy training and validation, we **only utilized the XYZ positional coordinates of the point clouds without incorporating color attributes**. This is because, for garments of the same category, color, material, and appearance can vary significantly, while their underlying geometric structure remains largely consistent. Introducing color information could potentially hinder generalization performance by introducing unnecessary variance, as garment colors in the real world are highly diverse and will further be significantly affected by variations in viewpoint and lighting. Since HALO relies solely on positional information of point cloud, it is theoretically robust to variations in color and texture.\n\nTo further validate this, **we conducted real-world experiments (Fold Tops) using three distinct types of garments featuring prints, textures and patterns** : (1) those with highly saturated colors, (2) those with complex textures (e.g., black-and-white patterns, color-blocking, or printed designs), and (3) those featuring raised elements (e.g., embroidery or appliqués). For each type, we performed five folding trials, resulting in a total of 15 experiments. **The overall success rate remained at 85% (13/15), confirming the method's robustness.**\n\nHowever, we acknowledge a limitation: **the policy may underperform when handling garments with atypical structures**, such as asymmetrical designs (e.g., single-sleeved tops) or deviations from conventional garment geometry. While such cases are relatively rare, they represent an important direction for future research.\n\n---\n\n## **Weakness2:**\n\n> In Table 3, please clarify what metric is being reported (e.g., is it the number of successful trials out of total attempts, or another measure?).\n\n## **Answer:**\n\nFor real-world evaluation, in every task, we have 3 distinct garments, each with 5 initial deformations and positions, resulting in a total of 15 experiments. **The metric is the number of successful trials out of total 15 experiments.**\n\n---\n\n## **Question1:**\n\n> Your method uses five-finger dexterous hands instead of common two-finger grippers. What specific advantages do five fingers provide for garment manipulation tasks in your experiments?\n\n## **Answer:**\n\n1.Our experimental results, particularly those real-world results, reveal significant limitations in two-finger grippers when handling thin or flat regions of garments. In such cases, **the gripper frequently fails to achieve a stable grasp, with garments either slipping during manipulation or not being grasped at all**. Many existing demonstrations involve garment folding with two-finger grippers rely on grasping the garment’s boundary, but these strategies also often leads to repeated failed attempts. In contrast, our system employing a five-finger dexterous hand demonstrates clear advantages. **Owing to the hand’s larger operation range, greater surface contact, and coordinated multi-finger control, it can stably grasp various regions of a garment—not just its edges—and maintain a firm hold throughout the manipulation process.** This capability greatly enhances the flexibility and robustness of robotic garment handling, freeing the system from constraints on graspable areas or predefined strategies.\n\n2.Furthermore, **the diverse hand postures offered by the dexterous hand allow us to tailor the manipulation strategy to the specific requirements of each task.** \n- to accomplish a fine-grained task such as putting on a glove, the hand can adopt a **pinch** posture to precisely grasp the glove edge. \n- For stable garment pick-and-place operations, an **open/close** posture can be used to maximize contact and grip. \n- When placing folded clothes, a **cradle** posture helps preserve the folded structure. \n- To smooth wrinkles, the hand can switch to a **smoothening** pose that maximizes surface coverage. \n\nThese example poses are shown in the bottom-left corner of Figure 1 (main paper). \n\nThe dexterous hand not only replicates the functionality of two-finger grippers but also enables finer and more robust manipulation of garments.\n\n3.Although current simulation still faces significant challenges in accurately modeling interactions between deformable objects and dexterous hands—limiting its ability to simulate complex tasks—these limitations are expected to be gradually addressed with the continued advancement of computer graphics and robotics technologies. As a result, **the advantages of dexterous hands will become even more prominent in complex tasks such as assistive dressing or knot tying, which require flexible coordination among multiple fingers.** We will continue to extend tasks and optimize the simulation framework based on DexGarmentLab, aiming to support a wider range of more complex dexterous-hand garment interaction tasks.\n\n---\n\n## **Question2:**\n\n> Are the grasp points on the garment defined in 2D (on the surface) or 3D coordinates?\n\n## **Answer:**\n\n**The grasp points are defined directly in 3D coordinates** .\n\n---\n\n## **Question3:**\n\n> Since garments are flexible, did you consider grasping force or pressure to avoid slippage or fabric damage?\n\n## **Answer:**\n\nThis is a highly promising research direction. Current studies in the field are still exploring how to enable robots to grasp garments both accurately and stably. As a result, the most straightforward strategy has been to apply maximum force during grasping to ensure success. However, beyond grasp success, it is equally important to detect garment slippage in real time and to avoid potential fabric damage. This requires considering grasping force or pressure, and leveraging tactile sensing during manipulation. We view this as an important and promising direction for future work. Relevant developments are currently underway in simulation, and we will continue to update and expand this line of research in subsequent work."}, {"review_id": "npLevgqEUH", "reviewer": "Reviewer_bHU9", "summary": "The authors introduce DexGarmentLab, a novel simulation environment designed for dexterous (especially bimanual) garment manipulation. Built on Isaac Sim, it offers over 2,500 garments across 8 categories and 15 task scenarios with high-quality 3D assets. They also propose an automated data collection pipeline that generates diverse demonstrations using a single expert demonstration, reducing manual intervention. Furthermore, the authors present HALO (Hierarchical gArment manipuLation pOlicy), a generalizable policy that combines affordance points for locating manipulation areas and a diffusion method for generating trajectories. Through extensive experiments, the authors demonstrate that HALO outperforms existing methods in both simulation and real-world settings, showing strong generalization across various garment shapes and deformations.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. DexGarmentLab Environment: Offers a realistic and diverse simulation platform for garment manipulation tasks.\n2. Automated Data Collection: Reduces manual effort by generating diverse demonstrations from a single expert demonstration.\n3. HALO Policy: Combines affordance points and a diffusion method to achieve strong generalization across different garment shapes and deformations.\n4. Experimental Validation: Demonstrates superior performance and generalization ability of HALO in both simulation and real-world experiments.\n\nWeaknesses：\n1. While the authors compare DP and DP3 methods, the study lacks benchmarking against state-of-the-art VLAs (pi0 or RDT) that demonstrate superior performance on folding tasks.\n2. While demonstrating performance gains on this benchmark DexGarmentLab, the claimed generalizability of the proposed method requires validation across diverse simulation environments. Additional cross-platform testing would strengthen the evidence for its universal applicability.\n3. Could the authors clarify whether this is a single-task or multi-task setting?\n4. Please specify the computational costs (time overhead) for both training and inference phases of the algorithm.", "was_revised": true, "final_justification": "My concerns have been addressed, so I will maintain my score to support the acceptance of the paper.", "rebuttal": "# Rebuttal for bHU9\nThanks for your helpful and meaningful reviews! We will provide a detailed response below to the issues outlined in the Weakness and Question sections.\n\n---\n\n## **Weakness1:**\n\n> While the authors compare DP and DP3 methods, the study lacks benchmarking against state-of-the-art VLAs (pi0 or RDT) that demonstrate superior performance on folding tasks.\n\n## **Answer：**\n\nWe have incorporated all reviewer suggestions related to the benchmark and additionally included four new baselines for comparison: **ACT (IL), pi0 (VLA), RDT (VLA), and Eureka (RL+VLM)**. We selected representative tasks for evaluation, including **Fling Dress, Fold Trousers, Hang Coat, and Wear Bowlhat in simulation, as well as Fold Tops in real world**. The evaluation protocol remains consistent with the original paper:\n- For simulation tasks, each task is evaluated over 50 episodes using three different random seeds. We report the success rates as Mean ± Std across all trials.\n- For real-world task, we evaluate using 3 distinct garments per category, each tested under 5 different initial deformations. We report the success rates as successful_trials / all_trials.\n\n| Typical Task Name | Fling Dress (Simulation) | Fold Trousers (Simulation) | Hang Coat (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real World) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\nDP (IL)  |0.59±0.05|0.47±0.04|0.52±0.04|0.41±0.05|*9/15*|\nDP3 (IL) |0.51±0.03|0.54±0.07|0.58±0.04|0.55±0.04|8/15|\nACT (IL) |0.35±0.02|0.49±0.06|0.43±0.04|0.51±0.03|7/15|\npi0 (VLA) |*0.69±0.01*|0.52±0.06|*0.72±0.01*|*0.59±0.02*|10/15|\nRDT (VLA) |0.60±0.02|*0.58±0.02*|0.62±0.02|0.48±0.01|9/15|\nEureka (RL+VLM) | / | / |0.16±0.03|0.08±0.02| / |\nHALO (Ours)    |**0.82±0.06**|**0.77±0.02**|**0.90±0.01**|**0.72±0.04**|**13/15**|\n\nAnalysis For **Imitation-Learning-Based Methods**:\n\nWe collect 100 demonstrations for each simulation task and collect 50 demonstrations for each real-world task, which are used for policy training (DP/DP3/ACT). A notable limitation of their performance lies in the inability to accurately grasp the target region of the garment, as well as the lack of fine-grained control during placement based on the garment’s shape(/state).\n\nAnalysis for **VLA Methods**:\n\nWe combine all the demonstrations of simulation and real-world tasks to fine-tune pi0 and RDT model. In terms of final performance, pretrained models such as pi0 and RDT outperform from-scratch approaches like DP, DP3, and ACT. However, compared to HALO, VLA-based models still exhibit clear limitations in accurately perceiving garment shape and state, and in executing precise grasp-and-place actions, resulting in a noticeable performance gap relative to HALO.\n\nAnalysis for **RL-Based Methods**:\n\nWe selected Eureka as the representative RL-VLM-combined method and conducted preliminary experiments on several tasks in simulation. While we observed some initial successful cases on simpler tasks such as Hang Coat and Wear Bowlhat, the overall success rate remained low. Additionally, due to the lack of robust parallelization support in the current Isaac Sim environment and the limited effectiveness of reward functions in handling long-horizon tasks like Fling and Fold, this baseline did not yield valid results across all tasks. Nevertheless, we are actively developing a multi-parallel deformable object manipulation environment based on Isaac Lab, and RL-based approaches for deformable manipulation will be a key focus of our future work.\n\n---\n\n## **Weakness2：**\n\n> While demonstrating performance gains on this benchmark DexGarmentLab, the claimed generalizability of the proposed method requires validation across diverse simulation environments. Additional cross-platform testing would strengthen the evidence for its universal applicability.\n\n## **Answer:**\n\nGarment simulation in our work is built upon NVIDIA PhysX's Position-Based Dynamics (PBD) and Finite-Element Method(FEM) framework, which currently offers the most realistic and stable performance for garment simulation. **Alternative simulators such as SofGym or PyBullet only support PBD and generally exhibit inferior performance in soft body dynamics. Moreover, they also lack support for fine-grained interactions between dexterous hands and garments.** Therefore, in terms of garment simulation fidelity and interaction capability, Isaac Sim remains the most suitable and advanced platform available to date, and it's not feasible to perform garment manipulation tasks in simulators outside of Isaac. \n\nHowever, regardless of the simulation platform used, real-world validation remains essential. Therefore, we directly consider the real-world environment as the final platform. Our real-world garment manipulation experiments demonstrate the effectiveness of our policy in physical settings, further supporting its general applicability.\n\nWhat's more, **for cross-platform testing, we selected three rigid-object manipulation tasks—covering both single-arm and dual-arm settings—from three different benchmarks: ManiSkill, RLBench, and Robosuite, each built on a distinct simulation backend.** We used DP and pi0 as baselines for comparison, and the success rates were obtained from existing publications.\n\nThe results show that HALO performs on par with, or even better than, pi0. This further supports the universal applicability of the HALO strategy across a wide range of tasks, including both rigid and deformable object manipulation.\n\n| Task Name | stackcube (Maniskill) | put_shoe_in_box (RLBench) | two_arm_lifting (Robosuite) |\n|:---:|:---:|:---:|:---:|\n| Backbone | Sapien | CoppeliaSim | Mujoco |\n| DP    | 53.8 % | 42.5%  | 51.0% |\n| pi0   | 92.0% | 78.6% | 86.5% |\n| HALO  | 90.0% | 85.0% | 90.0% |\n\n---\n\n## **Weakness3:**\n\n> Could the authors clarify whether this is a single-task or multi-task setting?\n\n## **Answer:**\n\nIf you are referring to whether a single model is trained for a single task or multiple tasks, this is a single-task setting. However, there are also some plausible ways to transfer it to multi-task setting, as discussed in [1,2].\n\nIf this is not what you meant, please feel free to let us know and we can have a further discussion.\n\n**ref:**\n\n[1]Hao C, Lin K, Luo S, et al. Language-guided manipulation with diffusion policies and constrained inpainting[J]. arXiv preprint arXiv:2406.09767, 2024.\n\n[2]Ma X, Patidar S, Haughton I, et al. Hierarchical diffusion policy for kinematics-aware multi-task robotic manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 18081-18090.\n\n---\n\n## **Weakness4:**\n\n> Please specify the computational costs (time overhead) for both training and inference phases of the algorithm.\n\n## **Answer:**\n\nThe training of **GAM** is performed on a **single RTX 4090 GPU**, requiring approximately **22 GB of GPU memory** when using a batch size of 64. Depending on the dataset size, the training process takes around **12 hours** to complete. GAM requires only about **1 GB of GPU memory** during the inference stage.\n\nThe **SADP** module is trained on a **single NVIDIA A800 GPU**, with a memory usage of **approximately 75 GB** when using a batch size of 200. The training process spans approximately **16 hours** for 3000 epochs. However, earlier checkpoints can be selected for validation depending on the use case.\n\nHALO, which integrates both GAM and SADP, requires about **3 GB of GPU memory** during the inference stage. When running HALO inference in the Isaac Sim environment, the total GPU memory footprint is approximately **15 GB**."}, {"review_id": "69kSdxNql7", "reviewer": "Reviewer_b8Ya", "summary": "This paper is dedicated to solving the dexterous garment manipulation problem. It introduces a new simulation environment (DexGarmentLab), an effective method for automatically collecting demonstration data from a single expert example, and a hierarchical policy (HALO) that generalizes across diverse garments. Extensive experiments in both simulation and the real world validate the approach, showing it significantly outperforms existing methods.", "questions": "see the weaknesses", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper delivers a complete framework—a novel simulated environment, an automated data pipeline, and a new dexterous garment manipulation policy—creating a valuable resource that can accelerate research in this field.\n- The proposed DexGarmentLab is a novel and large-scale simulation environment, which features over 2,500 garments, 15 task scenarios.\n- The proposed automated data collection method uses garment structure to generate diverse demonstrations from a single example, which effectively solves a major bottleneck for imitation learning.\n- The method's effectiveness is convincingly demonstrated through rigorous experiments on 14 simulated and 4 real-world tasks, with clear ablation studies and significant improvements over some baselines. \n\n\n**Weaknesses**\n\n- The methodological comparison is limited to Diffusion Policy (DP) and 3D Diffusion Policy (DP3). Including other relevant imitation learning or deformable object manipulation methods would provide a more comprehensive performance benchmark.\n- The quality of the automatically generated data is not quantitatively compared against traditional data collection methods (e.g., human teleoperation), which could further validate the pipeline's effectiveness. Moreover, the paper emphasizes the efficiency of the automated data collection pipeline. What is the success rate of the pipeline in generating valid and successful task demonstrations that can be used for policy training?\n- A more in-depth analysis of the sim-to-real transfer, including specific challenges and the impact of perception noise in the real world, would provide greater insight.\n- The paper would benefit from a clearer discussion on the boundaries of the policy's generalization, including an analysis of specific failure cases on more complex or unusual garments.", "was_revised": true, "final_justification": "The paper presents a complete and well-structured framework for dexterous manipulation, featuring a large-scale simulated environment (DexGarmentLab) with diverse tasks, an automated data pipeline that generates varied demonstrations from single examples, and a novel policy. The method is technically sound, with clear ablation studies validating key design choices. Experiments on simulated and real-world tasks demonstrate significant performance improvements over baselines, showcasing the framework's effectiveness and real-world applicability.", "rebuttal": "# Rebuttal for b8Ya\nThanks for your helpful and meaningful reviews! We will provide a detailed response below to the issues outlined in the Weakness and Question sections.\n\n---\n\n## **Weakness1:**\n\n> The methodological comparison is limited to Diffusion Policy (DP) and 3D Diffusion Policy (DP3). Including other relevant imitation learning or deformable object manipulation methods would provide a more comprehensive performance benchmark.\n\n## **Answer：**\n\nWe have incorporated all reviewer suggestions related to the benchmark and additionally included four new baselines for comparison: **ACT (IL), pi0 (VLA), RDT (VLA), and Eureka (RL+VLM)**. We selected representative tasks for evaluation, including **Fling Dress, Fold Trousers, Hang Coat, and Wear Bowlhat in simulation, as well as Fold Tops in real world**. The evaluation protocol remains consistent with the original paper:\n- For simulation tasks, each task is evaluated over 50 episodes using three different random seeds. We report the success rates as Mean ± Std across all trials.\n- For real-world task, we evaluate using 3 distinct garments per category, each tested under 5 different initial deformations. We report the success rates as successful_trials / all_trials.\n\n| Typical Task Name | Fling Dress (Simulation) | Fold Trousers (Simulation) | Hang Coat (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real World) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\nDP (IL)  |0.59±0.05|0.47±0.04|0.52±0.04|0.41±0.05|*9/15*|\nDP3 (IL) |0.51±0.03|0.54±0.07|0.58±0.04|0.55±0.04|8/15|\nACT (IL) |0.35±0.02|0.49±0.06|0.43±0.04|0.51±0.03|7/15|\npi0 (VLA) |*0.69±0.01*|0.52±0.06|*0.72±0.01*|*0.59±0.02*|10/15|\nRDT (VLA) |0.60±0.02|*0.58±0.02*|0.62±0.02|0.48±0.01|9/15|\nEureka (RL+VLM) | / | / |0.16±0.03|0.08±0.02| / |\nHALO (Ours)    |**0.82±0.06**|**0.77±0.02**|**0.90±0.01**|**0.72±0.04**|**13/15**|\n\nAnalysis For **Imitation-Learning-Based Methods**:\n\nWe collect 100 demonstrations for each simulation task and collect 50 demonstrations for each real-world task, which are used for policy training (DP/DP3/ACT). A notable limitation of their performance lies in the inability to accurately grasp the target region of the garment, as well as the lack of fine-grained control during placement based on the garment’s shape(/state).\n\nAnalysis for **VLA Methods**:\n\nWe combine all the demonstrations of simulation and real-world tasks to fine-tune pi0 and RDT model. In terms of final performance, pretrained models such as pi0 and RDT outperform from-scratch approaches like DP, DP3, and ACT. However, compared to HALO, VLA-based models still exhibit clear limitations in accurately perceiving garment shape and state, and in executing precise grasp-and-place actions, resulting in a noticeable performance gap relative to HALO.\n\nAnalysis for **RL-Based Methods**:\n\nWe selected Eureka as the representative RL-VLM-combined method and conducted preliminary experiments on several tasks in simulation. While we observed some initial successful cases on simpler tasks such as Hang Coat and Wear Bowlhat, the overall success rate remained low. Additionally, due to the lack of robust parallelization support in the current Isaac Sim environment and the limited effectiveness of reward functions in handling long-horizon tasks like Fling and Fold, this baseline did not yield valid results across all tasks. Nevertheless, we are actively developing a multi-parallel deformable object manipulation environment based on Isaac Lab, and RL-based approaches for deformable manipulation will be a key focus of our future work.\n\n---\n\n## **Weakness3:**\n\n> A more in-depth analysis of the sim-to-real transfer, including specific challenges and the impact of perception noise in the real world, would provide greater insight.\n\n## **Answer：**\n\nTo prevent potential misunderstandings, we clarify this point in advance: the real-world experiments presented in the submitted paper are not the result of a policy-level sim-to-real transfer. As can be observed, the real-world setup differs from the simulation environment.\n\nActually, there are two ways using HALO for garment manipulation tasks:\n1) **transfer the data collection pipeline from simulation to real world, conduct automated data collection and policy training directly in the real world.**\n2) **train the policy in the simulation and transfer the policy to the real world.**\n\nIn our paper, we adopt **way 1** to demonstrate the effectiveness of HALO. In the section *Weakness2*, we report the efficiency and success rate of real-world data collection, highlighting the strong sim-to-real performance of GAM.\n\nSim-to-real transfer from the simulation environment remains an important aspect of our study. To this end, **we aligned the settings of both the simulation and real-world environments by using the same hardware setup—Shadow Hand and UR10e—and selected two tasks, Hang Trousers and Wear Hat, for policy-level sim-to-real transfer, which means way 2.** The evaluation criteria follow those used in the previous real-world experiments. It is worth noting that sim-to-real performance is more sensitive to point cloud noise. To address the limited precision of the Realsense D435 in this context, we employed a Kinect camera for more accurate point cloud acquisition.\n\n| Task Name | Only Simulation Data | Simulation Data + 15 Real-World Data |\n|:---:|:---:|:---:|\nHang Trousers |  8 / 15 (53.3%) | 13 / 15 (86.7%) |\nWear Hat| 9 / 15 (60.0%) | 13 / 15 (86.7%) |\n\nExperimental results show that due to the gap between simulation and the real world, training the policy solely on simulated data leads to a drop in sim-to-real performance. Incorporating a small amount of real-world data into the training process can effectively enhance the policy’s generalization ability.\n\n---\n\n## **Weakness2:**\n\n> The quality of the automatically generated data is not quantitatively compared against traditional data collection methods (e.g., human teleoperation), which could further validate the pipeline's effectiveness. Moreover, the paper emphasizes the efficiency of the automated data collection pipeline. What is the success rate of the pipeline in generating valid and successful task demonstrations that can be used for policy training?\n\n## **Answer：**\n\n**We use LeapMotion for teleoperation data collection in simulation and an exoskeleton device for teleoperation in the real world.** Three representative tasks are selected for data collection, training, and evaluation. The performance of teleoperated policies is then compared with that of policies trained on autonomously collected data. The results are presented below.\n\n\n| Task Name | Hang Tops (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real_World)\n|:---:|:---:|:---:|:---:|\nHALO (Automated_Collection_Data) | 0.92±0.04 | 0.72±0.04 | 13 / 15 |\nHALO (TeleOperation_Data)| 0.88±0.03 | 0.70±0.06 | 13 / 15 |\n\nExperimental results indicate that **policies trained on teleoperated and automated data achieve comparable performance** during evaluation. However, from the perspective of human and time cost, teleoperation-based data collection is significantly more labor-intensive and time-consuming than automated data collection. It is also important to note that our data is not synthetically generated, but collected through real executions in either simulation or the real world, which means the demonstrations are physically realistic.\nThe entire data collection pipeline is enabled by model inference and one-shot demonstrations.\n\nThe following table reports the data collection time and success rate across all tasks, demonstrating the effectiveness and efficiency of the automated data collection pipeline.\n\n| Task Name | Fling Dress | Fling Tops | Fling Trousers | Fold Dress | Fold Tops | Fold Trousers | Wear BowlHat |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |92.6%（100/108）|90.1%（100/111）|90.1%（100/111）|84.7%（100/118）|82.0%（100/122）|82.6%（100/121）|95.2%（100/105）|\n| Single Collection Time |0min56s|0min55s|0min48s|1min06s|1min08s|0min56s|0min53s|\n\n| Task Name | Hang Coats |Hang Dress | Hang Tops | Hang Trousers | Store Tops |  Wear BaseballCap | Wear Scarf |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |92.6%（100/108）|90.9%（100/110）|91.7%（100/109）|99%（100/101）|98.0%（100/102）|85.5%（100/117）|83.3%（100/120）|\n| Single Collection Time |0min41s|0min48s|0min46s|0min58s|1min05s|0min55s|1min42s|\n\n| Task Name | Fold Tops (Real_World) |Hang Tops (Real_World) | Wear Scarf (Real_World) | Wear_Hat (Real World) |\n|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |90.9%（50/55）|96.2%（50/51）|83.3%（50/60）|93.8%（50/53）|\n| Single Collection Time |0min50s|0min38s|1min15s|0min36s|\n\n---\n\n## **Weakness4：**\n\n> The paper would benefit from a clearer discussion on the boundaries of the policy's generalization, including an analysis of specific failure cases on more complex or unusual garments.\n\n## **Answer：**\n\nBased on both simulation and real-world experimental results, we summarize the following points regarding boundary-related performance:\n\n1.For garments with occluded or highly deformed regions, the Garment Affordance Model (GAM) does not always predict manipulation points accurately. This can lead to failed or suboptimal grasps by the dexterous hand, impacting the final performance. Nonetheless, GAM performs reliably on most garments. For challenging cases, an initial unfold or fling action is recommended to expose key regions.\n\n2.The policy struggles with garments that deviate from standard geometries, such as asymmetrical designs (e.g., single-sleeved tops) or heavily adorned costumes, which distort the point cloud and affect both GAM and SADP performance."}, {"review_id": "9Ezzb8ePuF", "reviewer": "Reviewer_MdvE", "summary": "The paper presents DexGarmentLab, a data generation pipeline for garment manipulation and a generalizable manipulation policy using dexterous hands. To eliminate the extensive human effort typically required in imitation and reinforcement learning, the authors propose an automated data collection strategy that enables the generation of large-scale data from a single expert demonstration. An affordance model is employed to transfer manipulation strategies from a source garment to diverse target garments. Leveraging the generated dataset, the authors introduce a generalizable policy framework capable of performing garment manipulation across different scenarios. Extensive experiments are conducted to validate both the effectiveness of the data generation process and the generalizability of the learned policy. Real-world evaluations further demonstrate the practical applicability of the proposed approach.", "questions": "- Why could the authors claim their GAM as a \"generalizable policy\"? Using which mechanisim could the authors truly guarantee the generalization ability or \"ensure the generalization for xxx\" (line 217)? Except for experiments conducted on cloth with different materials and deformations, the paper does not provide deeper analysis—either from a theoretical perspective or through more fundamental insights—to support the generalization claim. I understand that the method is developed to improve the generalizability. In this case, it would be better not to use such a strong tone since the method cannot fully gaurantee 100% generalization ability.\n- How to generate plausible hand grasping poses in the automated data collection process? The authors mentioned LeapMotion in 4.2. However, after checking the appendix for details, LeapMotion seems to be a teleoperation interface that transfers human hand poses to the shadow hand poses. If the data collection is a teleoperation, then everything is reasonable. But it is claimed as an \"automated\" one.  So what's the automatic grasping pose generation process using LeapMotion? Besides, how to make sure that the grasping pose is physically valid? Would a filtering mechanism be introduced to ensure the plausibility and task completion? If so, what about the success rate of the demonstration generation?\n- Real-world evaluations are conducted on a Psi Bot with dexterous hands, which are different from the shadow hand. So, how to directly transfer the policy trained on shadow hands to the real robot? Would it require retraining the policy using Psi Bot in the simulator? If so, we need to re-collect demonstrations -- and since the LeapMotion is designed for shadow hand -- how to generate grasping poses for Psi Bot?", "limitations": "- Insufficient comparisons. The paper lacks comparisons with reinforcement learning (RL)-based approaches and imitation learning (IL) strategies that use demonstrations collected via teleoperation. Without these baselines, it is difficult to fully assess the effectiveness and advantages of the proposed demonstration generation method and policy training strategy.\n- Presentation quality. (1) Sections 2 and 3 occupy a disproportionate amount of space relative to their content. (2) Some claims throughout the paper are stated too strongly, which may overstate the method's contributions. (3) Several figures could be made more concise to enhance clarity and improve the overall flow.\n- Restricted scientific value. (This is a relatively minor point and has little impact on my overall rating, but I include it for completeness.) The technical innovations and the core message of the paper do not appear to offer fundamentally new insights or contributions to the field.\n\nOverall, DexGarmentLab proposes a promising manipulation environment tailored for a range of tasks involving deformable garments. The thoughtful design of the simulation setup, task environment, garment assets, and the proposed policy training strategy collectively contributes a useful resource to the community. So I would like to give a positive rating. Considering limitations like insufficient experimental comparisons, issues with presentation quality, and a relatively modest level of scientific value, my support would be limited to BA.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Good motivations and valuable insights. The authors propose an efficient and automatic garment manipulation data collection strategy where only one expert demonstration is required, based on the observation that leveraging RL or IL to collect data would be quite time-consuming and human-labor intensive. \n- Well-designed method. The data generation pipeline effectively leverages affordance modeling to transfer manipulation strategies from one object to new objects. The policy designed to achieve generalization across different garments is also reasonably designed. \n\nWeakness\n\n- The task difficulty is restricted. Although the task involves long-horizon cloth folding using dexterous hands. The manipulation challenge is not that high --- such manipulation could also be completed by two grippers. The value of dexterous hands, as well as their superiority over parallel grippers, are not clearly demonstrated or justified in the context of the garment manipulation tasks considered in this work. \n- Experiments lack a comprehensive comparison with RL-based approaches. Although RL requires reward tuning, there are many works that have explored how to leverage LLMs and VLMs to accelerate the reward tuning process. Compared to IL with demonstrations generated by the proposed approach or collected via teleoperation, RL policies can automatically explore diverse solutions to complete the task and would be much more robust. It would be better not to exclude such types of baselines. \n- Section 3 takes too much space -- more than 4 pages, which leaves quite insufficient space for the experiment section. The method section and the related work section could be made more concise than its current version.", "was_revised": true, "final_justification": "DexGarment focuses on dexterous garment manipulation. The method, including a data generation pipeline and the policy design, is technically sound. Thorough experiments conducted in the simulated environment and the real world further validate the effectiveness of the proposed technique. It makes a solid contribution to the community and merits acceptance.", "rebuttal": "# Rebuttal for MdvE\nThanks for your helpful and meaningful reviews! We will provide a detailed response below to the issues in the Weakness and Question sections.\n\n---\n\n## **Weakness1:**\n\n> The tasks are limited in difficulty, and the advantages of dexterous hands over parallel grippers are not clearly demonstrated.\n\n## **Answer:**\n\nDue to current restrictions in simulation technology, certain complex tasks cannot yet be effectively implemented in simulated environments. However, across the existing tasks, dexterous hands still demonstrate clear advantages over parallel grippers in several aspects.\n\n1.Our real-world experiments highlight **the limitations of two-finger grippers in handling thin or flat garment regions(such as), where stable grasps often fail due to slipping or missed contacts**. Many existing demonstrations rely on grasping the garment’s boundary, but these strategies also often leads to repeated failed attempts. In contrast, our system employing a five-finger dexterous hand demonstrates clear advantages. **Owing to the hand’s larger operation range, greater surface contact, and coordinated multi-finger control, it can stably grasp various regions of a garment—not just its edges—and maintain a firm hold throughout the manipulation process.** This capability greatly enhances the flexibility and robustness of robotic garment handling, freeing the system from constraints on graspable areas or predefined strategies.\n\n2.Furthermore, **the diverse hand postures offered by the dexterous hand allow us to tailor the manipulation strategy to the specific requirements of each task.** \n- to accomplish a fine-grained task such as putting on a glove, the hand can adopt a **pinch** posture to precisely grasp the glove edge. \n- For stable garment pick-and-place operations, an **open/close** posture can be used to maximize contact and grip. \n- When placing folded clothes, a **cradle** posture helps preserve the folded structure. \n- To smooth wrinkles, the hand can switch to a **smoothening** pose that maximizes surface coverage. \n\nThese example poses are shown in the bottom-left corner of Figure 1 (main paper). \n\nThe dexterous hand not only replicates the functionality of two-finger grippers but also enables finer and more robust manipulation of garments.\n\n3.Although current simulation still faces restrictions in precisely modeling highly complex interactions and multi contact between deformable objects and dexterous hands, these limitations would be gradually addressed with the continued advancement of computer graphics and robotics technologies, just like Sapien and Isaac Sim. As a result, **the advantages of dexterous hands will become even more prominent in complex tasks such as assistive dressing or knot tying, which require flexible coordination among multiple fingers.** We will continue to extend tasks and optimize the simulation framework based on DexGarmentLab, aiming to support a wider range of more complex dexterous-hand garment interaction tasks.\n\n---\n\n## **Weakness2:**\n\n>  The paper lacks comparisons with RL and IL methods based on teleoperated demonstrations.\n\n## **Answer:**\n\nWe have incorporated all reviewer suggestions related to the benchmark and additionally included four new baselines for comparison: **ACT (IL), pi0 (VLA), RDT (VLA), and Eureka (RL+VLM)**:\n\n| Typical Task Name | Fling Dress (Simulation) | Fold Trousers (Simulation) | Hang Coat (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real World) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\nACT (IL) |0.35±0.02|0.49±0.06|0.43±0.04|0.51±0.03|7/15|\npi0 (VLA) |*0.69±0.01*|0.52±0.06|*0.72±0.01*|*0.59±0.02*|10/15|\nRDT (VLA) |0.60±0.02|*0.58±0.02*|0.62±0.02|0.48±0.01|9/15|\nEureka (RL+VLM) | / | / |0.16±0.03|0.08±0.02| / |\nHALO (Ours)    |**0.82±0.06**|**0.77±0.02**|**0.90±0.01**|**0.72±0.04**|**13/15**|\n\nAnalysis for **RL-Based Methods**:\n\nWe selected Eureka as the representative RL-VLM-combined method and conducted preliminary experiments on several tasks in simulation. While we observed some initial successful cases on simpler tasks such as Hang Coat and Wear Bowlhat, the overall success rate remained low. Additionally, due to the lack of robust parallelization support in the current Isaac Sim environment and the limited effectiveness of reward functions in handling long-horizon tasks like Fling and Fold, this baseline did not yield valid results across all tasks. Nevertheless, we are actively developing a multi-parallel deformable object manipulation environment based on Isaac Lab, and RL-based approaches for deformable manipulation will be a key focus of our future work.\n\n**Comparison with teleoperation data**:\n\n**For teleoperation, We use LeapMotion in simulation and an exoskeleton device in the real world.** The results are presented below.\n\n\n| Task Name | Hang Tops (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real_World)\n|:---:|:---:|:---:|:---:|\nHALO (Automated_Collected) | 0.92±0.04 | 0.72±0.04 | 13 / 15 |\nHALO (TeleOperated)| 0.88±0.03 | 0.70±0.06 | 13 / 15 |\n\nExperimental results indicate that **policies trained on teleoperated and automated data achieve comparable performance** during evaluation.\n\n---\n\n## **Weakness3 & Question1:**\n\n> Section 3 takes too much space......\n\n> Why could the authors claim their GAM as a \"generalizable policy\"?......\n\n## **Answer:**\n\nThanks a lot for your quite helpful advice on paper organization and presentation! **We believe these suggestions are highly valuable for DexGarmentLab and we will address these problems in camera-ready version to further enhance the quality of the paper.**\n\n---\n\n## **Question2:**\n\n> ...... explain details on how LeapMotion generates grasping poses, how physical validity is ensured, whether a filtering mechanism is used, and the success rate of demonstration generation.\n\n## **Answer:**\n\nFor single specific task category (e.g., Hang Coat), the hand poses required to perform specific actions in specific regions, can generally be treated as consistent. Moreover, the deformable nature of garments allows them to adapt naturally to the dexterous hand's pose during manipulation. Therefore, for each specific task, we first define a set of task-specific hand poses (the number depends on the task; for example, Hang Coat requires both a closed grasping pose for the collar and an open pose for release). These poses are generated via teleoperation using LeapMotion, with physical feasibility considered during generation. In this way, the pose definition process inherently includes a manual filtering step to select the most suitable poses.\n\nThese task-specific hand poses are then transferred directly into the data collection process for that task category. That is, during data collection across different garments within the same task, the same predefined hand poses are reused. \n\nTo further validate the effectiveness of our automated data collection pipeline, we report the success rates for all tasks in the table below.\n\n| Task Name | Fling Dress | Fling Tops | Fling Trousers | Fold Dress | Fold Tops | Fold Trousers | Wear BowlHat |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |92.6%（100/108）|90.1%（100/111）|90.1%（100/111）|84.7%（100/118）|82.0%（100/122）|82.6%（100/121）|95.2%（100/105）|\n\n| Task Name | Hang Coats |Hang Dress | Hang Tops | Hang Trousers | Store Tops |  Wear BaseballCap | Wear Scarf |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |92.6%（100/108）|90.9%（100/110）|91.7%（100/109）|99%（100/101）|98.0%（100/102）|85.5%（100/117）|83.3%（100/120）|\n\n| Task Name | Fold Tops (Real_World) |Hang Tops (Real_World) | Wear Scarf (Real_World) | Wear_Hat (Real World) |\n|:---:|:---:|:---:|:---:|:---:|\n| Collection Success Rate |90.9%（50/55）|96.2%（50/51）|83.3%（50/60）|93.8%（50/53）|\n| Single Collection Time |0min50s|0min38s|1min15s|0min36s|\n\n---\n\n## **Question3:**\n\n> ......How to directly transfer the policy trained on shadow hands to the real robot? Would it require retraining the policy using Psi Bot in the simulator? ......the LeapMotion is designed for shadow hand -- how to generate grasping poses for Psi Bot?\n\n## **Answer:**\n\nTo prevent potential misunderstandings, we clarify this point in advance: the real-world experiments presented in the submitted paper are not the result of a policy-level sim-to-real transfer.\n\nActually, there are two ways using HALO for garment manipulation tasks:\n1) **transfer the data collection pipeline from simulation to real world, conduct automated data collection and policy training directly in the real world.**\n2) **train the policy in the simulation and transfer the policy to the real world.**\n\nIn our paper, we adopt **way 1** for demonstrate the effectiveness of HALO. In the section *Question2*, we report the efficiency and success rate of real-world data collection, highlighting the strong sim-to-real performance of GAM.\n\nSim-to-real transfer from the simulation environment remains an important aspect of our study. To this end, **we aligned the settings of both the simulation and real-world environments by using the same hardware setup—Shadow Hand and UR10e—and selected two tasks, Hang Trousers and Wear Hat, for policy-level sim-to-real transfer, which means way 2.** The evaluation criteria follow those used in the previous real-world experiments.\n\n| Task Name | Only Simulation Data | Simulation Data + 15 Real-World Data |\n|:---:|:---:|:---:|\nHang Trousers |  8 / 15 (53.3%) | 13 / 15 (86.7%) |\nWear Hat| 9 / 15 (60.0%) | 13 / 15 (86.7%) |\n\nSo if you want to train the policy in simulation and use a sim-to-real method to transfer policy, you need to align the settings of both the simulation and real-world environments, in the meanwhile, re-collect demonstrations for training. As for psibot setting, because psibot G0-R dexhand only has six freedoms, we can just define each freedom by ourselves to generate target hand pose, or we can also use teleoperation device in real world to get target pose."}, {"review_id": "sg70YkLtz5", "reviewer": "Reviewer_ntYt", "summary": "The paper introduces DexGarmentLab, a comprehensive framework for bimanual dexterous garment manipulation. Contributions include: (1) A new simulation environment built on Isaac Sim, featuring over 2,500 garment assets from ClothesNet across 15 task scenarios; (2) an automated data collection pipeline that is able to generate datasets of diverse trajectories from a single expert demonstration using a Garment Affordance Model (GAM) to identify manipulation points on new garments; (3) a hierarchical policy HALO which uses GAM to generate manipulation trajectories conditioned on visual and state information. The approach is validated in both simulation and in the real world.", "questions": "- The entire data collection process is bootstrapped from a single, manually defined expert demonstration. How does the system perform if this initial demonstration is varied? For instance, if you provide a slightly different (but still successful) folding sequence, how does that affect the diversity and quality of the 100 generated demonstrations and the performance of the resulting policy?\n- Could the authors comment on the extensibility of DexGarmentLab to facilitate future research? For example, what is the workflow for tuning the simulation parameters if a new garment is introduced? The paper mentions parameters were \"carefully tuned\"; is there a systematic procedure for this, or does each new asset require extensive, ad-hoc manual tuning? How readily can the policy and data collection frameworks be adapted to different hardware? How hard would it be to map the 60-DoF action space and the LeapMotion data collection to a new bimanual setup with different kinematics?", "limitations": "Yes, the authors have done a thorough discussion of the limitations in Appendix F.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- [Quality]: The DexGamentLab ecosystem is a complete package with a feature-rich simulation environment, a large-scale asset library, an automated data collection methodology, and a policy. 14 simulation tasks and 4 real robot tasks further strengthen its statistical significance.\n- [Clarity]: The paper is generally well-organized and easy to follow, considering the amount of work done.\n- [Significance]: The framework provides the first bimanual dexterous garment manipulation environment at this scale. Automated data collection reduces reliance on teleoperation. The development of a high-quality, open-source environment and a promising baseline policy is a valuable resource for further research in this area.\n- [Originality]: The integration of structural correspondence (GAM) with diffusion-based trajectory generation (SADP) for category-level generalization is novel.\n\nWeaknesses:\nThe paper's primary weakness stems from a significant mismatch between the scope of its claimed contributions and the focus of its empirical evaluation. The authors frame the work as having three main contributions: the DexGarmentLab environment, the automated data collection pipeline, and the HALO policy. However, the experimental section is almost exclusively dedicated to evaluating the performance of the HALO policy against baselines and ablations. The other two pillars of the contribution lack corresponding empirical validation, creating a structural imbalance in the paper.\n- [Lack of Experimental Validation for DexGarmentLab]: While the qualitative comparisons in Table 1 and the descriptive improvements to the physics model are informative, they are not supported by experiments. While a key claim is that these new physical parameters reduce the sim-to-real gap and provide more realistic interactions, there is no experiment designed to quantify this benefit. For instance, an analysis showing that training a policy is more sample-efficient or faster in terms of wall-clock time compared to other environments or using the AttachmentBlock method would be necessary to empirically substantiate the environment's value as a standalone scientific contribution.\n- [Lack of Quantitative Analysis for the Data Collection Pipeline]: Similarly, the automated data collection pipeline is also presented as a major contribution. However, the quality of the data it produces is never directly measured or analyzed. The paper provides no evidence to show that the generated demonstrations are indeed \"expert\" or how they compare to demonstrations collected via other methods, such as human teleoperation. The value is thus only implicitly suggested through the HALO policy trained on its data.\n\n**Note**: I acknowledge that this paper possesses a significant amount of effort. The above weaknesses only apply **if the intention is to present the contribution as a unified whole.** Alternatively, and perhaps more effectively, the paper's narrative could be reframed. The work would present as a more coherent and defensible contribution if HALO were positioned as the primary focus, with the environment and data collection pipeline framed as a novel and sophisticated infrastructure built to enable this core research. This would still be a strong contribution, and I will still maintain positive if the authors prefer reframing.\n\n- [Minor]: Citation for PBD is missing: Müller, Matthias, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. \"Position based dynamics.\" Journal of Visual Communication and Image Representation 18, no. 2 (2007): 109-118.", "was_revised": true, "final_justification": "During the rebuttal, all my questions and concerns about the other two core components of the manuscript were properly addressed by the authors. I believe this work is a nice addition to the dexterous robotics research community, with contributions spanning from infrastructure to algorithms. Therefore, I give a final recommended score of Accept.", "rebuttal": "# Rebuttal for ntYt\nThanks for your helpful and meaningful reviews! We will provide a detailed response below to the issues outlined in the Weakness and Question sections.\n\n---\n\n## **Weakness:**\n\n> [Lack of Experimental Validation for DexGarmentLab]: ......an analysis showing that training a policy is more sample-efficient or faster in terms of wall-clock time compared to other environments or using the AttachmentBlock method would be necessary to empirically substantiate the environment's value as a standalone scientific contribution.\n\n> [Lack of Quantitative Analysis for the Data Collection Pipeline]: ......the quality of the data it produces is never directly measured or analyzed. The paper provides no evidence to show that the generated demonstrations are indeed \"expert\" or how they compare to demonstrations collected via other methods, such as human teleoperation.......\n\n## **Answer:**\n\nThanks a lot for pointing out these issues! They are highly valuable and constructive suggestions for improving DexGarmentLab. Below, we provide additional clarification regarding the missing empirical validation in the **Environment** and **Data Collection Pipeline** components.\n\n1.As for **Lack of Experimental Validation for DexGarmentLab**:\n\nFirstly, it is important to clarify that garment simulation in our work is built upon NVIDIA PhysX's Position-Based Dynamics (PBD) framework, which currently offers the most realistic and stable performance for garment simulation. **Alternative simulators such as SofGym or PyBullet do not support PBD-based simulation and generally exhibit inferior performance in soft body dynamics. Moreover, they also lack support for fine-grained interactions between dexterous hands and garments.** Therefore, in terms of garment simulation fidelity and interaction capability, Isaac Sim remains the most suitable and advanced platform available to date.\n\nCurrently, the two main open-source garment simulation platforms based on Isaac Sim are GarmentLab and DexGarmentLab. GarmentLab adopts an **attach block-based approach** for garment manipulation, where grasping is achieved by explicitly attaching or detaching blocks to the garment. However, this method **has significant limitations.** Specifically, **attachment and detachment must be explicitly triggered in code, which makes it unsuitable for end-to-end or reinforcement learning-based policy training.** Instead, it is more appropriate for hierarchical planning or hardcoded control, such as those guided by vision-language models (VLMs).\n\nMoreover, **as discussed in the main paper (Lines 130–132), even minimal contact—such as a single finger block touching the garment—can result in attachment and lifting, which is clearly unrealistic.** In the early stages of our project, we experimented with training policies using the attach block method. However, during evaluation, the instability of the attach/detach mechanism often caused the garment to behave erratically or \"fly off,\" leading to frequent task failures. As a result, we ultimately abandoned this approach.\n\n**To further illustrate this point, we selected two representative tasks to evaluate the performance of the attach block-based method.** The results clearly show that this approach performs poorly across both tasks.\n\n| Typical Task Name | Fling Dress (Simulation) | Hang Coat (Simulation) |\n|:---:|:---:|:---:|\nHALO (AttachBlock)  |0.06±0.01|0.15±0.03|\nHALO (Ours)    |**0.82±0.06**|**0.90±0.01**|\n\n2.As for **Lack of Quantitative Analysis for the Data Collection Pipeline**:\n\nThis missing component indeed affects the overall structure of the paper. To address this, we have provided the relevant quantitative analysis below.\n\n**We use LeapMotion for teleoperation data collection in simulation and an exoskeleton device for teleoperation in the real world.** Three representative tasks are selected for data collection, training, and evaluation. The performance of teleoperated policies is then compared with that of policies trained on autonomously collected data. The results are presented below.\n\n\n| Task Name | Hang Tops (Simulation) | Wear Bowlhat (Simulation) | Fold Tops (Real_World)\n|:---:|:---:|:---:|:---:|\nHALO (Automated_Collection_Data) | 0.92±0.04 | 0.72±0.04 | 13 / 15 |\nHALO (TeleOperation_Data)| 0.88±0.03 | 0.70±0.06 | 13 / 15 |\n\nExperimental results indicate that **policies trained on teleoperated and automated data achieve comparable performance** during evaluation. However, from the perspective of human and time cost, teleoperation-based data collection is significantly more labor-intensive and time-consuming than automated data collection. It is also important to note that our data is not synthetically generated, but collected through real executions in either simulation or the real world, which means the demonstrations are physically realistic. The entire data collection pipeline is enabled by model inference and one-shot demonstrations.\n\n3.As for **summary**:\n\nThanks again for your valuable suggestions regarding the organization and presentation of the paper. To be honest, we did encounter considerable difficulty when structuring the overall narrative of the paper. Receiving such constructive feedback during the rebuttal stage has been truly helpful for us. **We will sincerely incorporate your suggestions in the revised version, including adding the missing content (such as citations for PBD) and further refining the structure and clarity of DexGarmentLab.** Thank you once again!\n\n---\n\n## **Question1：**\n\n > The entire data collection process is bootstrapped from a single, manually defined expert demonstration. How does the system perform if this initial demonstration is varied? For instance, if you provide a slightly different (but still successful) folding sequence, how does that affect the diversity and quality of the 100 generated demonstrations and the performance of the resulting policy?\n\n## **Answer:**\n\nTo address this question, we conducted additional experiments on the **Fold Tops** task by providing two new variations of initial demonstrations:\n\n 1）**First Variation**: The task sequence remains unchanged (i.e., folding the left sleeve to the right collar, the right sleeve to the left collar, then folding the bottom hem to the collar), but the demo points used to locate key regions are altered.\n\n 2）**Second Variation**: The task sequence itself is modified—first folding the left sleeve to the right hem, then the right sleeve to the left hem, followed by folding both hems toward the collar. The demo points for keypoint localization are also different from the original.\n\nFor each case, we collected 100 demonstrations and trained the HALO model accordingly. The evaluation results are as follows:\n\n | Demo Type | Original Demo | Demo Variant 1 | Demo Variant 2\n|:---:|:---:|:---:|:---:|\nHALO | 0.81±0.03 | 0.84±0.02 | 0.78±0.02 |\n\nFrom the evaluation results, models trained on demonstrations collected with different task sequences perform comparably during validation. This suggests that changing the task sequence does not significantly affect the quality or diversity of the demonstrations, nor does it have a notable impact on the final policy performance.\n\n---\n\n## **Question2-1：**\n\n> Could the authors comment on the extensibility of DexGarmentLab to facilitate future research? For example, what is the workflow for tuning the simulation parameters if a new garment is introduced? The paper mentions parameters were \"carefully tuned\"; is there a systematic procedure for this, or does each new asset require extensive, ad-hoc manual tuning? \n\n## **Answer:**\n\n**Our simulation parameters are carefully tuned for garment manipulation and are suitable for most tasks without the need of further adjustment.**\n\nFor researchers seeking customization, we will provide a markdown document in the GitHub repository detailing garment-related parameters, including official references and effect descriptions. Additionally, key parameters in **Env_Config/Garment/Particle(Deformable)_Garment.py** are marked with '**# important**' to help researchers focus on critical settings.\n\n---\n\n## **Question2-2：**\n\n> How readily can the policy and data collection frameworks be adapted to different hardware? How hard would it be to map the 60-DoF action space and the LeapMotion data collection to a new bimanual setup with different kinematics?\n\n## **Answer:**\n\nFrom an implementation perspective, extending the system to a new bimanual setup involves the following steps:\n\n**1）Motion Control in Simulation and Real World:**\n\nIn simulation, arm motion control can be implemented using libraries such as **Curobo**, while hand control can be adapted based on our provided codebase.\n\nIn the real world, most commercially available arms and hands come with well-encapsulated motion control interfaces, which can be directly used.\n\n**2）Customizing Hand Poses:**\n\nIf the hand has a low number of degrees of freedom (DoF), the joint configurations can be manually specified without relying on teleoperation.\n\nFor hands with higher DoF, teleoperation devices such as LeapMotion, VR systems, or manufacturer-provided interfaces are recommended. Several open-source repositories for hand teleoperation are also available and have reached a high level of maturity. For LeapMotion, users can refer to our codebase and modify it to accommodate different dexterous hand configurations.\n\n\n**3）Note:**\n\nWe do not recommend directly mapping the 60-DoF action space and the LeapMotion-based data collection to a new bimanual setup with different kinematics.\nThis is because not only the action space differs, but the visual observations also vary.\nTo collect data and train policies under a new bimanual setup, it is necessary to rebuild the simulation and real-world environment, and adapt both joint states and action spaces accordingly, in line with the kinematic structure of the new robotic system."}]

Author rebuttal:

Meta-review: ## Summary

This paper introduces DexGarmentLab, a comprehensive framework for dexterous garment manipulation. The work makes three primary contributions: a new simulation environment with a large dataset of 3D garment assets, an automated data collection pipeline that generates diverse demonstrations from a single expert example, and a novel hierarchical policy (HALO) for generalizable manipulation. The paper is a strong contribution to the field of robotic manipulation and is recommended for acceptance.

## Strength identified by Reviewers
All five reviewers unanimously highlighted the completeness and significance of the framework as a major strength. They praised the integration of a realistic simulation environment, an innovative automated data pipeline, and a novel, effective policy. This complete package is seen as a valuable resource that will accelerate future research in dexterous and deformable object manipulation. The AC believes that the data collection pipeline will be a strong contribution to the research community

## Weaknesses and Rebuttal
The initial reviews raised several constructive weaknesses, which the authors thoroughly addressed in their rebuttal.

- Limited Baselines (bHU9, b8Ya, MdvE): Multiple reviewers pointed out that the initial evaluation was limited, primarily comparing HALO against Diffusion Policy (DP) and 3D Diffusion Policy (DP3). 

  - Rebuttal: The authors responded by conducting an extensive new set of experiments, adding four new baselines: ACT (IL), π0 (VLA), RDT (VLA), and Eureka (RL+VLM). 

- Lack of Validation for the Environment and Data Pipeline (ntYt): One reviewer noted that there was no quantitative analysis of the data pipeline's quality compared to other methods (e.g., teleoperation) or the simulation environment's benefits.

  - Rebuttal: The authors provided new experiments directly comparing the performance of policies trained on their automatically collected data versus data from human teleoperation. The results showed comparable performance, validating the quality of their pipeline while highlighting its superior efficiency. 

- Generalization to Complex Garments (JjAQ): A reviewer questioned how well the method would generalize to garments with more complex patterns, textures, or atypical structures.

  - Rebuttal: The authors clarified that their policy relies on point cloud geometry, not color, making it robust to textures and patterns. They validated this with new real-world experiments on garments with varied prints and textures, achieving a high success rate. They also acknowledged the limitation that the policy may struggle with highly atypical garment structures, defining a clear boundary for the method's current capabilities.

- Clarity on Sim-to-Real Transfer (b8Ya, MdvE): There was some confusion about whether the real-world experiments involved direct sim-to-real policy transfer.

  - Rebuttal: The authors clarified that their primary real-world results came from training on real-world data collected via their automated pipeline. However, they also conducted new policy-level sim-to-real experiments, showing that while there is a performance drop, it can be effectively mitigated by fine-tuning with a small amount of real-world data.

## Recommendation Justification

This paper presents a complete and well-structured framework for dexterous garment manipulation, featuring a large-scale simulated environment (DexGarmentLab), an automated data pipeline that generates varied demonstrations from a single example, and a novel, high-performing policy (HALO). The work is technically sound, with rigorous experiments and ablation studies validating its key design choices on 14 simulated and 4 real-world tasks. The rebuttal led to a clear consensus, with all five reviewers confidently recommending acceptance and some increasing their scores. The resulting work is a significant and high-impact contribution that provides a valuable, open-source resource poised to accelerate future research in the challenging domain of deformable object manipulation.
Decision: Accept (spotlight)