ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks
Authors: Arth Shukla
Stone Tao
Hao Su
Venue: ICLR 2025 Poster (2025)
Keywords: benchmark
dataset
simulation
reinforcement learning
imitation learning
robotics
Primary area: datasets and benchmarks
TL;DR: We provide a GPU-accelerated implementation of the HAB which supports realistic low-level control, run extensive RL and IL baselines, and develop a rule-based trajectory filtering system which enables efficient, controlled data generation at scale.

Abstract: High-quality benchmarks are the foundation for embodied AI research, enabling significant advancements in long-horizon navigation, manipulation and rearrangement tasks. However, as frontier tasks in robotics get more advanced, they require faster simulation speed, more intricate test environments, and larger demonstration datasets. To this end, we present MS-HAB, a holistic benchmark for low-level manipulation and in-home object rearrangement. First, we provide a GPU-accelerated implementation of the Home Assistant Benchmark (HAB). We support realistic low-level control and achieve over 3x the speed of prior magical grasp implementations at a fraction of the GPU memory usage. Second, we train extensive reinforcement learning (RL) and imitation learning (IL) baselines for future work to compare against. Finally, we develop a rule-based trajectory filtering system to sample specific demonstrations from our RL policies which match predefined criteria for robot behavior and safety. Combining demonstration filtering with our fast environments enables efficient, controlled data generation at scale.

Published as a conference paper at ICLR 2025
MANISKILL-HAB: A BENCHMARK FOR LOW-LEVEL MANIPULATION IN HOME REARRANGEMENT TASKS
Arth Shukla, Stone Tao & Hao Su Hillhot Inc. and University of California, San Diego (a (arshukl stao, hacsu) @ucssu)uccsdeede
ABSTRACT
high-quality benchmarks are the foundation for embodied Al research, enabling sigmificant advancements in long-hone oonghooizonn navigation, manipulation and rearrange- ment tasks. However, as frontier tasks in robotics get more advanced, they require faster simulation speed, more intricate test environments, and larger demonstra- tion datasets. To this end, we present MS-HAB, a holistic benchmark for low- level manipulation and in-home object rearrangement. First, provide GPU- accelerated implementation of the Home Assistant Benchmark (HAB). We sup- port realistic low-level control and achieve over 3x the speed of prior magical grasp implementations at a fraction of the GPU memory usage. Second, we train extensive reinforcement learning (RL) and imitation learning (IL.) baselines for fu- ture work to compare against. Finally, we develop a rule-based trajectory filtering system to sample specific demonstrations from our RL policies which match pre- defined criteria for robot behavior and safety. Combining demonstration filtering with our fast environments enables efficient, controlled data generation scale.
Videos, models, data, code, and more at http: htap:::///arhhsshukk www.aattthshhlaa.iiiaaauu...a io/mso/al
INTRODUCTION
An important goal of embodied Al is to create robots that can solve manipulation tasks in home-scale environments. Recently, faster and more realistic simulation, home-scale rearrangement bench- marks, and large robot datasets have provided important platforms to accelerate research towards this goal. However, there remains a need for all of these features in one unified benchmark.
We present MS-HAB, holistic, open-sourced, home-scale manipulation benchmark with four key features: (1) fast simulation with realistic physics and manipulation, including low-level control, for efficient training, evaluation, and dataset generation, (2) home-scale manipulation tasks through the Home Assisitant Benchmark (HAB) (Szot et al., 2021), (3) extensive baselines for future work compare against, and (4) scalable, controlled data generation using automated, rule-based trajectory filtering system.
Fast Manipulation Simulation with Realistic Physies and Rendering: Using ManiSkill3 (Tao et al., 2024), we implement a GPU-accelerated version of the HAB (Szot et al., 2021), an apartment- scale rearrangement benchmark containing three long-howe ong-horizon tasks using the Fetch mobile manipu- lator (.LebraTeccholopee
2024). While the original HAB uses magical grasp (teleport closest object within 15cm to the gripper), we require realistic grasping.
The MS-HAB environments support low-level control for realistic grasping, manipulation, and in- teraction, while the original Habitat 2.0 implementation does not support such kind of low-level control. Furthermore, by scaling parallel environments, MS-HAB environments achieve over 4300 samples per second (SPS) while the robot actively collides with multiple dynamic objects and the environment renders 2 128x128 RGB-D images 3x faster than Habitat 2.0 a fraction of the GPU memory usage. This significant speedup allows us to scale training, evaluation, and data generation.
Reinforcement Learning (RL) Baselines: Online RL provides promising framework learn from online interaction without needing preexisting demonstration data. As in Gu et al. (2023a), train individual mobile manipulation skills and chain them solve long-horizon tasks. We

---

Published as a conference paper at ICLR 2025
hand-engineer dense rewards for the Fetch embodiment, designed for low-level control with mobile manipulation. Furthermore, we train manipulation policies overfit to one specific object's 's geometry, outperforming all-object policies when grasping many objects in conditions with tight tolerances depending on object geometry. Leveraging our fast environments, we run extensive RI. baselines, training 150 policies across 3 seeds (50 policies/seed) with 1.83 billion environment samples.
Automated Event Labeling and Trajectory Categorization: We use privileged information from the simulator distill trajectories into chronologically ordered lists of events (e.g. Pick events include 'Contact (object)', 'Grasped', 'Dropped', 'Success', and 'Excessive (robot) Collisions') ). Using these events lists, we define mutually exclusive, collectively exhaustive success and failure modes. For example, Pick success mode "reach success A cumulative robot collisions < 5000 N object not dropped" requires events list (Contact, Grasp, Success) and forbids events 'Dropped' and "Excessive Collisions'. We filter our dataset by selecting demonstrations labeled with success modes that guarantee particular behaviors (e.g. pick without dropping) and safety constraints (e.g- cumulative robot collisions < 5000 N). Furthermore, provide trajectory categorization statistics for all baselines in Appendix A.6 so future work can gear its methodology to solve frequent failure modes discovered by policies.
Dataset Generation and Imitation Learning (IL) Baselines: When generating our dataset, we trajectory categorization to filter demonstrations without needing manual labor, and we provide Imitation Learning (IL.) baselines using our dataset. Our results show that selecting demonstrations with particular behavior biases IL policies towards that behavior. Paired with fast simulator, users can generate massive datasets and control demonstration type in fast wall-clock time.
Summary of Contributions:
The contributions of MS-HAB are summarized follows: 1) GPU- accelerated HAB implementation which supports realistic low-level control and achieves over 4300 SPS while interacting and rendering. 2) extensive RL and IL baselines. 3) automated event labeling and trajectory categorization, providing success and failure mode statistics for all baseline policies, and 4) efficient, controlled vision-based robot dataset generation at scale.
2 RELATED WORK
Simulators and Scene-Level Embodied AI Platforms: Earlier scene-level simulators focus on navigation and simple interaction with realistic visuals (Savva et al., 2019). Other simulators add kinematic object state transitions (Kolve et al., 2017; Li et al., 2021), significant scene randomization (Deitke et al., 2022; Nasiriany et al., 2024), soft-body physics and audio (Gan et al., 2022), flexible and deformable materials, object composition rules, and so on (Li et al., 2022). However such complicated features often slow down simulation speed.
Habitat 2.0 forgoes additional features, supporting rigid-body dynamics, articulations, and magical grasping, to achieve best-in-class single-process scene-level simulation speed (Szot al., 2021). However, it is constrained by the limited parallelization of CPU simulation.
Other simulators focus on low-level, contact-rich control in simpler settings (James al., 2020; Zhu et al., 2020; Xiang et al., 2020). ManiSkill3 in particular achieves state-of-the-art GPU simulation speed (Tao al., 2024), however its suite of tasks are simpler than the Home Assistant Benchmark (HAB) (Szot et al., 2021), which we implement for MS-HAB.
Scalable Demonstration Datasets: Real-world robot datasets are promising for direct deployment to the real world (Brohan et al., 2023). However, these initiatives are limited in scaling and use cases due to small-scale toy setups (Ebert al., 2022), vision-only data (Dasari et al., 2019), or requiring massive coordinated (et al., 2024; Khazatsky et al., 2024) or distributed (Mandlekar et al., 201 8) human effort over many months or even years. Furthermore, real robot datasets cannot efficiently generate new data, and do not support online sampling.
Generative interactive world models allow some interactivity on similarly realistic data by generating new frames based on provided actions (Yang et al., 2024). However, these models suffer from arti- facts and long-term memory issues which rule out home-scale rearrangement, and low frame rates make training ligh-freguency low-level control policies intractable. Furthermore, neither real-robot datasets nor generative world models currently support querying privileged data from a simulator, which is necessary for MS-HAB's automated event labeling and trajectory categorization.
2

---

Published as a conference paper at ICLR 2025
Meanwhile, classical physical simulation supports data generation from variety of sources (ex- pert teleoperated, suboptimal human, etc), and machine- -eeeritte data is largely scalable; however, datasets like Fu et al. (2020); Mandlekar et al. (2021); Gu et al. (2023b) only support smaller-scale continuous control tasks.
RoboCasa combines different aspects of above approaches (Nasiriany al., 2024): physical sim- ulator, diverse Al-generated textures and models, 1250 numan-teleoperated demonstrations, and MimicGen scale data (Mandlekar al., 2023). However, RoboCasa achieves only 31.9 SPS with- oul rendering, does not support filtering trajectories by behavior, and its demonstrations alternate between manipulation and navigation. Meanwhile, we achieve 125x faster simulation while ren- dering 2 128x128 RGB-D images and interacting with multiple dynamic objects. We also support automated filtering under customizable constraints, and our demonstrations use whole-body control.
Chen et al. (2023) and Lee et al. (2021) use finetuning methods to bias the initial and terminal state distributions to increase handoff success while skill chaining. However, these meth- ods are applied to tasks with unchanging order (e.g. furniture assembly, block orient/grasp/insert) Meanwhile, Gu al. (2023a) formulate composable and reusable skills with mobility to create greater overlap in initial and terminal state distributions, achieving better results than stationary manipulation. However, Gu et al. (2023a) uses magical grasp with online RL, while we provide RL and IL baselines, and we include additional considerations for low-level grasping, such as new rewards and subtask success conditions, sampling grasp poses from Pick policies, and overfitting object manipulation policies to specific object geometries.
Skill Chaining:
3 PRELIMINARIES
3.1 TASKS, SUBTASKS, AND POLICIES
The Home Assistant Benchmark (HAB) (Szot et al., 2021) includes three long-horizon tasks which involve rearranging objects from the YCB dataset (Çalli al., 2015):
TidyHouse: Move 5 target objects to different open receptacles (e.g. table, counter, etc).
PrepareGroceries: Move 2 objects from the opened fridge to goal positions on the counter, then object from the counter to the fridge.
SetTable: Move bowl from the closed drawer to the dining table and apple from the closed fridge to the same dining table.
To solve these tasks, Szot et al. (2021) define parameterized skills: Pick, Place, Open Fridge/Drawer, Close Fridge/Drawer, and Navigate. For each skill, define corresponding subtasks. Successful low-level grasping is heavily dependent on an object's pose. So, depending on the subtask, the simulator provides ground-truth pose posse position apos for target articulation a, or 3D goal position 9pos. updated each timestep during ma- nipulation. Each subtask also fails if the robot cumulative force reaches beyond threshold. For more details, see Appendix A.I. We provide brief descriptions of the subtasks below:
[xroal [[pps]] for target object æ, ground-truth handle
Pick (a,
(Zoo pick object x (from articulation if provided).
Placeja (a, optionall (Ipose Ipos): ): place object in goal g (in articulation a, if provided)
Open(a] (appo))))po): open articulation a with handle at apos
Close(a](Apoz) close articulation a with handle at Apos
Nav(+pos): navigate
From reinforcement learning perspective, we formulate each long-horizon task as a standard
Markov Decision Process (MDP) which can be described tuple M = (S, A, R, T, P. with continuous state space S, action space A, scalar reward function R : S A - R, environment dynamics function T S A S. initial state distribution and discount factor y [0. 1]. Then, as Gu et al. (2023a), define subtask w as a smaller MDP (S. Rw, T, Pw, 7) derived from M. For each task M with subtask w, train low-level control policy nw - A with RL II..
3

---

Published as a conference paper at ICLR 2025
In this work, we study partially observable variant of each task, where the policy must 128x128 depth images to infer collisions and obstructions. We train different policies for each task/subtask combination (i.e.. TidyHouse Pick, PrepareGroceries Pick, etc). Additionally, Pick and Place RI, policies to overfit to specific objects, i.e., one policy for each task/subtask/objeet combination. Since this work focuses low-level control, subtask. Additional details on training policies and teleport navigation are provided in Sec. 5.1.
train
teleport for the Navigation
3.2 SKILL CHAINING
Similar to Szot et al. (2021), we split each task into a sequence of subtasks using a perfect task planner. The sequences defined below:
TidyHouse: For (wi, gi) E {<00 (o),.. ... (x4, complete:
(Xi.o))) + Pick(iii,ii (Ti.ppo))))) + (ov(si - Place (ii.see Di.pos
PrepareGrocerries: For (zi,, M1)) E {(20, go). (x1, m1 (22, 92)}; ). complete:
(av( + Pickpo[is1] (Ti.pose) + Nav (gi.pos)
Placepy((=2] ( (iippose Di.pos)
SetTable: For (I a;) E {E0: 90. Dr), (2o, Fr) complete:
Nav(Qi,pos) + Open,, (Qi.pos) Nav Nai. (os)
Pick Fi,pose) Nav(gi.os)
7
Place(Ii.pose: pi,pos) Nav(as.pos) Close.: (ai.pos)
3.3 TRAIN AND VALIDATION SPLITS
The ReplicaCAD dataset (Szot al., 2021) serves as the source for our apartment scenes. It com- prises 105 scenes divided into macro-variations, each containing 21 micro-variations. Macro- variations alter the layout of large furniture items such as refrigerators and kitchen counters, while micro- variations modify the placement of smaller furmishings like chairs and TV stands. The dataset split into three parts: macro- variationss for training, for validation, and for testing. However, as the test split is not publicly accessible, our study utilizes only the train and validation splits.
Furthermore, for each long-horizon task, HAB provides 10,000 training episode configurations and 1,000 validation configurations. These configurations specify initial poses for YCB objects and define target objects, articulations, and goals. Importantly, these configurations exclusively utilize ReplicaCAD scenes from their respective splits.
ENVIRONMENT DESIGN AND BENCHMARKS
By scaling parallel environments with GPU simulation, MS-HAB achieves 4300 SPS on a bench- mark involving representative interaction with dynamic objects - 3x Habitat 2.0's implementation. Our environments support realistic low-level control for successful grasping, manipulation, and in- teraction, while the Habitat 2.0 environments do not support such kind of low-level control. This section outlines environment design choices which leverage GPU acceleration and benchmarks MS- HAB against Habitat's implementation.
4.1 ENVIRONMENT DESIGN
Evaluation and Training Environments: First, we provide the base evaluation environment, Sequentio i Task, which supports executing different subtasks simultaneously on GPU. We per- form physics simulation and rendering for all environments in parallel, then slice data by subtask to compute success and fail conditions. does not support dense reward or spawn selection/rrjectton..
Second, provide training environments for each subtask, (SubtaskNe which extend the main evaluation environment. Each training environment provides dense rewards hand-engineered for the Fetch embodiment, supports spawning with randomization and rejection, and incorporates any additional features needed for training specific subtask skills.
Subtask aasarirrnin
We include target object pose, goal position, and TCP pose relative the base, an indicator of whether the target object is grasped. 128x 128 head and arm RGB-D images, and robot proprioception. For experiments, we use only depth images. As is standard for the ManiSkill
Observation Space:
4

---

Published as a conference paper at ICLR 2025
Interact Benchmark
Samples per Second (SPS)
5000
4096 envs
2048
4000
1024
512
3000
256
128
2000
64
16
32
48 envs
1000
2
5
10
15
20
25
GPU Memory Usage (GB)
MS-HAB (ours)
Habitat
Figure 1: Interact benchmark comparing MS-HAB (ours) with Habitat. Fach data point is annotated with the number of parallel environments used. SPS and GPU memory usage for each data point are averaged over 10 seeds; error bars representing 95% CIs are plotted, but are too small to see. Thanks to GPU acceleration, MS-HAB scales parallel environments to achieve over 3x the performance of Habitat while using a fraction of the GPU memory.
suite of tasks, the simulator computes ground-truth poses. We keep consistent observation space across all subtasks via masking to support different subtasks running in parallel.
Action Space: We fully actuate the Fetch embodiment's arm, torso, and head pan/tilt joints. We support joint-based controllers and end-effector based controllers. For our experiments, we use a PD joint delta position controller for the arm, torso, and head joints. The agent provides linear and angular velocity to control the base. The action space is normalized to [-1,1].
Additional Details: Our environments load the ReplicaCAD dataset provided by Habitat 2.0. How- ever, since Habitat 2.0 uses magical grasp, the original ReplicaCAD dataset's s collision meshes do not include handles for the kitchen drawers and fridge. So, alter these collision meshes to include handles based on the provided visual meshes. We additionally provide navigable position meshes for the Fetch embodiment with Trimesh, ManiSkill3 does not currently support navmeshes.
4.2 BENCHMARKING
We adapt Habitat 2.0's Interact benchmark, which originally had the Fetch robot execute precom- puted trajectory to collide with two dynamic objects (Szot et al., 2021). While we retain the same precomputed trajectory, assets, and scene configuration, able magical grasp, allowing it to interact with five objects instead. Our setup includes two mounted 128x 128 RGB-D cameras, with simulation frequency of 100Hz and control frequency of 20Hz (the standard for low-level control in ManiSkill3). We collect observation data from vectorized en- vironments at each step () call. Our benchmarking conducted on a machine equipped with 16-core/32-thread Intel 19-1KKS processor and an Nvidia RTX 4090 GPU with 24 GB VRAM.
modify the robot's initial pose and dis-
is important to note that running the exact same episode different simulators exceedingly dif- ficult since different simulation backends will result in interactions and collisions behaving slightly differently. Still, the full rollout similar in both simulators, and the measured performance increase of MS-HAB in an interactive setting is significant.
Habitat's Additional Optimizations: While the Habitat simulator already has best-in-class single process simulation speed, it provides optional additional optimizations: concurrent rendering and auto sleep. However, their experiments suggest that concurrent rendering can negatively impact train performance (Szot et al., 2021),
enable auto-sleep and disable concurrent rendering.
Per Fig. 1, while Habitat achieves stronger performance per parallel envi- ronment, its peak performance is limited to 1397.65 + 11.02 SPS 22.60 GB VRAM due CPU simulation's 's parallelization limitations and a less efficient renderer. Mcanwhile, by scaling up to 4096 environments, MS-HAB is able to achieve 4299.18 + 26.36 SPS 15.35 GB VRAM usage
Benchmark Analysis:

---

Published as a conference paper at ICLR 2025
Pick
Place
Open
Close
Figure 2: Renders of low-level, whole-body control policies solving Pick, Place, Open, and Close subtasks. We render 512x512 image and 128x128 sensor images. Note the base's moving position relative to surroundings. Goal spheres invisible to sensors. Full videos in supplementary.
3.08x faster with 32% less VRAM. Furthermore, our environments support realistic low-level control for successful grasping, manipulation, and interaction, while the Habitat 2.0 environments do support such kind of low-level control.
5
METHODOLOGY
5.1 TRAINING REINFORCEMEN LEARNING POLICIES
We choose Reinforcement Learning (RL) to learn our subtask policies as RL does not require prior demonstration data, and it can take advantage of our highly parallelized environments solve tasks in fast wall-clock time. We use a similar subtask formulation as M3, which trains mobile manipula- tion skills to solve each subtask from region spawn points.
Pick: Without magical grasp, our Pick policies must learn grasp poses which are valid, stable, and reachable within the kinematic constraints of the mobile Fetch robot. Furthermore, the policy must leam action sequences which can reach these grasp poses and retrieve the target object within the specified horizon while keeping the robot under the cumulative collision force limit.
As result, learning successful grasping for multiple objects with different geometries in addition whole body control with collision constraints is difficult. So, we opt to train individual Pick policies for each object, thereby overfitting to the geometry of that object. Our experiments show these per-object Pick policies achieve improved subtask success rates compared to all-object policies when handling many objects with varied geometries. In other words, train unique per-object Pick policy for every mask/subtaskoobjet combination.
Place:
We train per-object Place policies well. Our experiments show that, in settings where object geometry is more important (c.g. placing fridge with tighter tolerances), per-object Place policies reach higher success than all-object policies.

---

Published as a conference paper at ICLR 2025
Additionally, without magical grasp, there is not ground-truth means of spawning the robot while grasping an object. So, we train our Pick policies before Place, then we sample grasp poses from our Pick policies to initialize the robot in the Place subtask.
Following (Gu et al., 2023a), we train different Open and Close policies for the kitchen drawer and the fridge. Furthermore, we find that opening the kitchen drawer is particularly difficult due to its small handle. So, we perturb the initial state distribution of our Open kitchen drawer subtask during training to accelerate learning: 10% of the time, we initialize the kitchen drawer opened 20% of the way. During evaluation, we do not alter the initial states.
Open and Close:
A Algorithms and Hyperparameters: We stack consecutive frames for image observations han- dle partial observability.
We train Pick and Place using SAC (Haarnoja et al., 2018; Xing, 2022) with Im replay buffer size. Visual observations are encoded by D4PG's 4-layer CNN (Barth-Marge et al., 2018) and concatenated with state observations. Actor and critic networks are 3-layer MLPs and the critic has LayerNorm to avoid value divergence (Ball et al., 2023). We train Pick with 50M timesteps and Place with 25M timesteps.
We train Open and Close using PPO (Schulman et al., 2017; Huang et al., 2022). Visual observations are encoded by a NatureCNN (Mnih al., 2015) and concatenated with state observations. The actor and critic networks are 2-layer MLPs. We train Open Fridge with 15M timesteps, Open Drawer with 50M timesteps, Close Fridge with 25M timesteps, and Close Drawer with 15M timesteps.
We train 3 seeds for each task/subtask /object combination, evaluating on 189 episodes every 100,000 train samples. We select the checkpoint with highest evaluation success once rate as our final policy.
Metrics: We run 1000 episodes for every evaluation run (task/subtask evaluation, ablations, etc).
We evaluate subtask policies (Pick, Place, Open, Close) by success once rate (%), which is the percentage of trajectories that achieve success at least once in an episode with 200 timesteps. We evaluate long-horizon task success (TidyHouse, PrepareGrocerries SetTable) by Progressive Com- pletion Rate (%). Here, the success of each subtask requires the success of every previous subtask. Hence, the completion rate of the final subtask is the completion rate of the entire long-horizon task.
Furthermore, since we are primarily interested in low-level control and manipulation, we replace navigation with a simple teleport. The robot is teleported to the target location with the same arm, base pose, and spawn location randomizations as in subtask training, described in Appendix A. In long-horizon tasks, we move to the next subtask as soon as the current subtask achieves success.
5.2 AUTOMATED TRAJECTORY CATEGORIZATION AND DATASET GENERATION
Thanks to fast simulation environments, we can quickly generate 10s to 100s of thousands of demon- strations. However, our experiments show that our Imitation Learning (IL) policies are sensitive to demonstration behavior. To filter out "suboptimal" demonstrations, we use privileged information from our simulator group demonstrations into mutually exclusive, collectively exhaustive success and failure modes without significant manual labor. Furthermore, we use this trajectory labeling system identify types and causes of failure in our bascline policies in Sec. 6.2 and Appendix A.
We provide a high-level overview of trajectory labeling on the Pick subtask. For detailed definitions of events and labels, see Appendix A.6. First, we define "events" which occur at any timestep t: 1) Contact: nonzero robot/target pairwise force, 2) Grasped: object not grasped at step (-1 and grasped at step 3) Dropped: object grasped at step f- and not grasped at step I, and 4) Excessive Collisions: robot cumulative force exceeds 5000 N. For Pick trajectory Tpick = (su, ag. ..... Sn, a,), we create chronologically ordered event list Epick = (E1....,,e)) .k).
Example of Pick Subtask:
Next, we define success and failure modes. For example, one success mode is "straightforward success" with Epick = (Contact, Grasp, Success), requiring success without dropping the object or colliding too much. One failure mode is "dropped failure," defined as (Excessive Collisions Epick) A (Dropped E Epick) A (i < j for maximal i, such that e; = Grasped, e, = Dropped). "Dropped failure" trajectories fail because the robot irrecoverably drops the target object.
In generating our Pick subtask dataset, we apply filters to include only "straightforward success" trajectories. These trajectories are characterized by the absence of dropping and minimal collisions.
7

---

Published as a conference paper at ICLR 2025
As the dataset generation code is publicly available, users have the flexibility to create their datasets with custom constraints tailored to their specific requirements.
dataset using Behavior Cloning (BC) (Bain & Sammut, 1995; Ross et al., 2011; I; Daftry ct al., 2016) (implementation based on Huang al. (2022)). Visual observations encoded the 5-layer CNN from Bojarski et al. (2016), then concatenated with state observations. The actor network is a 3-layer MLP.
Imitation Learning Baselines: We train IL baselines
Dataset Size: With fast environments and automated trajectory filtering, users generate many 10s or 100s of thousands demonstrations as needed controlled manner. For our baselines, we generate 1000 demonstrations per task/subtaskoohject combination using our per-object RI. policies on the train split. The filters used are listed in Appendix 1. with definitions in Appendix A.6.2.
6 RESULTS
6. BASELINES
Fig. shows the RL and IL policies' progressive completion rate. We provide an optimistic upper bound on progressive completion rate by (incorrectly) assuming that the completion of each subtask is independent of every other subtask, thus directly multiplying subtask success once rates. Table shows success once rate for individual subtasks. We find 4 major avenues for improvement.
First, our optimistic upper bound shows low expected success rate on the long long-horizon tasks. Even with per-object RL policies, our low-level mobile manipulation subtasks are difficult train on dense reward, and improving subtask success rate is the most direct way to improve overall task completion rate. Second, TidyHouse and SerTable RL baselines have some gap between upper bound and real completion rate, indicating potential handoff issues or disturbance to prior target ob- jects in success states. Meanwhile, the PrepareGroceries RL baseline has large drop in completion rate during the second Pickı, subtask, indicating that the first Picky, causes too much disturbance to objects the fridge. So, improving policy performance cluttered spaces is important. Third, our IL policies perform notably worse in Pick and Place, indicating need for methods and architectures which can handle multimodalities in the data. Finally, while most RI, policies generalize well to the validation split, the Close Fridge policy completely fails validation scenes because the fridge door opens into wall, preventing the arm from reaching the handle. This is not an issue with magical grasping (Gu al., 2023a), indicating that low-level control may need more scene diversity.
TidyHouse
PrepareGroceries
SetTable
RL Policies
compeetion rate 1M
80
28
82
40
DO60
ploc.
ceso.0
100
100
100
IL Policies
Comp mute (N4
60
60
20
Figure 3: Long-horizon task progressive completion rates (%) on train and validation splits averaged
over 1000 episodes. Futhermore, we provide an 'upper bound' on performance based on the success rates of each subtask policy. Best viewed zoomed.
8

---

Published as a conference paper at ICLR 2025
Table I: Subtask success once rates for RL and baselines. The RL-Per vs All column shows the difference in per-object RL policy performance and its all-object counterpart. We do not train all-object policies for Open or Close subtasks.
TASK
SUBTASK
SPLIT
RL-PER
RL-ALL
IL
RL-PER vs ALL
Train
81.75
71.63
61.11
+10.12
Pick
Val
77.48
68.15
59.03
+9.33
TidyHouse
Train
65.77
63.69
61.81
+2.08
Place
Val
65.97
66.07
63.79
-0.10
Train
66.57
51.88
44.64
+14.69
Pick
Prepare Groceries
Val
72.32
62.10
52.78
+10.22
Train
60.22
53.37
50.00
+6.85
Place
Val
65.67
58.63
56.75
+7.04
Train
80.85
75.69
60.71
+5.16
Pick
Val
88.49
79.86
72.62
+4.63
Train
73.31
72.82
71.23
+0.49
Place
Val
67.06
68.25
62.20
-1.19
Train
83.43
74.01
Open
Val
88.10 10
53.67
SetTable
Train
84.92
79.86
OpeN
Val
84.52
78.57
Train
86.81
86.90
Closer
Val
0.00
0.00
Train
88.79
88.39
Closepr
Val
89.29
87.60
6.2 ABLATIONS
6.2.1 RL POLICIES: ALL-OBJECT VS PER-OBIECT POLICIES
The goal of training per-object RL policies for Pick and Place is to improve subtask success rate since policies with higher success rates allow us to generate successful demonstrations under more initialization conditions. To verify this, run two ablations.
Does training per-object Pick and Place policies improve subtask success rate compared to all- object policies? Per Table per-object policies perform notably better in TidyHouse and Prepare Groceries Pick, which involve objects, with more modest improvement SetTable Pick, which has only 2 objects. Per-object policies perform significantly better in PrepareGroceries Place, which involves placing with tight tolerances on a cluttered fridge shelf, while performance differences are negligible TidyHouse and SetTable Place, which only involve open receptacles. So, per-object Pick and Place policies learn improved manipulation when grasping a greater variety of objects, or when manipulating objects in areas with tighter constraints.
Are per-object policies necessary to learn grasping for certain objects in the Pick subtask? In Table 2, we run our automated trajectory labeling system Pick YCB object #003, the Cracker Box (Çalli et al., 2015). The Fetch robot's 's parallel gripper can only grasp the Cracker Box along its shortest dimension, so the of valid grasp poses are highly dependent on the object's pose relative to the robot. The all-object policy is 1.88-2.42x more likely to fail to excessive collisions and 1.87- 12.37x more likely to fail to grasp the object, indicating that overfitting to a specific geometry important for our RL policies to learn grasping on difficult geometries. For more detailed trajectory labeling definitions and statistics, please see Appendix A.6.
6.2.2 IL POLICIES: LABELING AND FILTERING DATASET TRAJECTORIES
IL Policies: Can we control the behavior of our policies by filtering for specific demonstra- tions? Our PrepareGrrcerris Place RI. policies have two similarly frequent success modes: place

---

Published as a conference paper at ICLR 2025
Table 2: Trajectory labeling on Pick Cracker Box with and per-object RL policies. We group the trajectories into four categories: success once (S-Once), excessive collision failure (F-Col), cannot grasp failure (F-Grasp), and other failure modes (F-Other). We provide the percentage of trajectories which fall into each category, and each row sums to 100% (barring any rounding errors).
TASK
SPLIT
TYPE
S-ONCE
F-COL
F-GRASP
F-OTHER
RI-AII
29.46
34.52
28.17
7.85
Train
RL-Per
71.63
17.26
2.48
8.63
TodyHouse
RE-AII
33.73
33.13
24.50
8.64
Val
RL-Per
73.41
16.67
1.98
7.94
RL-AII
11.51
60.62
16.17
11.70
Train
Prepare Groceries
RL-Per
51.98
25.10
8.63
14.29
RL-AII
14.19
57.24
26.88
1.69
Val
RL-Per
56.15
30.46
9.72
3.67
in goal (release the object within 15cm of Ipos) and drop to goal (release beyond 15cm). Although MS-HAB does not simulate state transitions like breaking, placing objects without dropping is desirable, safe robot behavior avoid excessive damage.
We generate datasets with 500 demonstrations per object: only, and 3) 50/50 split ("place", "drop", and "split"). We fit IL policies to each dataset and run trajectory labeling to determine policy behavior, shown in Table 3. The place and drop policies show bias towards executing place and drop trajectories respectively, but still perform the opposite behavior somewhat frequently. The split policy is somewhat biased towards dropping. likely because the I-dim gripper action drop easier to learn under MSE loss than 7-dim arm action place.
1) place goal only, 2) drop goal
Table 3: Success once rate (S-Once, %) and ratio of "place in goal" to "drop to goal" trajectories (Place Drop). Note that some success trajectories are labeled place in goal drop goal, as there are other possible success modes described in Appendix A.6.
FILTERS
SPLIT
S-ONCE
PLACE DROP
Train
45.73
3.17:
Place in goal
Val
54.46
2.55:
Train
49.21
1:2.22
Drop to goal
Val
51.19
1: 2.86
Train
50.30
1.71
50/50 Split
Val
55.56
1:41
Thus, data filtering can generally influence policy behavior, but additional methods are needed to fully control behavior (e.g. online finetuning, reward relabeling, more advanced IL. methods, etc).
CONCLUSION AND LIMITATIONS
We present MS-HAB, a holistic home-scale rearrangement benchmark including a GPU-accelerated implementation of the HAB which supports realistic low-level control, extensive RL and IL. base- lines, systematic evaluation using our trajectory labeling system, and demonstration filtering for efficient, controlled data generation at scale. However, there is significant room for improvement on our baselines, and we do not claim transfer to real robots; both these we leave to future work. Whole-body low-level control under constraints in cluttered environments, long mog-horizon skill chain- ing, and scene-level rearrangement are challenging for current robot learning methods; we hope our benchmark and dataset aid the community in advancing these research areas.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
Thank you to members of Hillbot, Inc. and the Hao Su Lab for providing valuable feedback on the project.
REPRODUCIBILITY STATEMENT
We open source all code for environments, training, evaluation, and data generation, and we release our dataset for public use, all available on our website: mshab.
shirhh- whukla.github. io/
REFERENCES
Michael Bain and Claude Sammut. A framework for behavioural cloning. Koichi Furukawa, Donald Michie, and Stephen H. Muggleton (eds.), Machine Intelligence 15, Intelligent Agents [St. Catherine's 's College, Oxford, UK, July 1995]. pp. 103-129. Oxford University Press, 1995.
Philip J. Ball, Laura M. Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 1577-1599 PMLR, 2023.
Gabriel Barth Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional deterministic policy gradients. In 6th International Conference Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Mariusz: Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1664.07336, 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang-Huel Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu (cds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10- 14, 2023, 2023. doi: 10.15607/RSS. 2023.XIX.0X
Berk Çalli, Aaron Walsman, Arjun Singh. Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. Benchmarking in manipulation research: The YCB object and model set and benchmark- ing protocols. CoRR, 0051100233444 2015.
Yuanpei Chen, Chen Wang. Li Fei-Fei, and C. Karen Liu, Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. CoRR, abs/2309.00987, 2023. doi: 10.48550/ARXIV. 2309.00987.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024.
11

---

Published as a conference paper at ICLR 2025
Shreyansh Daftry, J. Andrew Bagnell, and Martial Hebert. Learning transferable policies for monoc- ular reactive MAV control. In Dana Kulic, Yoshihiko Nakamura, Oussama Khatib, and Gentiane Venture (eds.), International Symposiun Experimental Robotics, ISER 2016, Tokyo, Japan, October 3-6, 2016, volume 1 of Springer Proceedings in Advanced Robotics, pp. 3-11. Springer,
2016. doi: 10.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Leslie Pack Kaelbling, Danica Kragie, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learning. CoRI. 2019, Osaka, Japan, October 30 November I, 2019, Proceedings, vol- ume 100 of Proceedings of Machine Learning Research, pp. 885-897. PMLR, 2019.
Sudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, and Sergey Levine. The ingre- dients for robotic diffusion transformers. CoRR, abs/2410.10088, 2024. doi: 10.48550/ARXIV. 2410.10088.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied AI using procedural generation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Frederik Ebert, Yanlai Yang. Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Kris Hauser, Dylan A. Shell, and Shoudong Huang (eds.), Robotics: Science and Systems XVIII, New York City, NY, USA, June 27 July 1, 2022, 2022. doi: 10.15607/RSS.20222
Abby O'Neill et al. Open x-embodiment: Robotic learning datasets and RT-X models : Open X-embodiment collaboration. In IEEE International Conference Robotics and Automation, ICRA 2024, Yokohuma, Japan, May 13-17, 2024, pp. 6892-6903. IEEE, 2024. doi: 10.1 109/ 0...7714.2224.1000 1477.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.0721 ..2219 2020.
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel L. K. Yamins, James J. DiCarlo, Josh H. McDermott, Antonio Torralba, and Joshua B. Tenenbaum. The threedworld transport challenge: A visually guided task-and-motion plan- ning benchmark towards physically realistic embodied AI. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pp. 8847-8854. IEEE, 2022. doi: 10.1 1091100//AA466
1091CRA46639 SAA.CCRAA666392229.998888 2022.982399.
Jiayuan Gu, Devendra Singh Chaplot, Hao Su, and Jitendra Malik. Multi-skill mobile manipulation for object rearrangement. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a.
Jayyan Gu. Fanbo Xiang, Xuanlin Li. Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 2022.
12

---

Published as a conference paper at ICLR 2025
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics Autom. Lett., 5(2):3019-302,, 2020. doi: 10.1109/1RA.2000227777000
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Iija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Ro- han Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, and et al. DROID: A large-scale in-the-wild robot manipulation dataset. CoRR, doi: 10448550/ARXI.22403.1 12945.
abs/2403. 114400..12945 12945, 2024,
Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12- THOR: an interactive 3d environment for visual AL. CoRR, abs/ 2017.
Youngwoon Lee, Joseph J. Lim, Anima Anandkumar, and Yuke Zhu. Adversarial skill chaining for long-horizon robot manipulation via terminal state regularization. In Alcksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference Robot Learning, 8-11 November 1021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 406-416. PMLR, 2021.
Chengshu Li, Fei Xia, Roberto Martin-Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simula- tion for robot learning of everyday household tasks. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-1 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 455 465.. PMLR, 2021.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine. Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R. Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR- IK: A bench- mark for embodied AI with 1, 000 everyday activities and realistic simulation. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.), Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 80-93. PMLR, 2022.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. ROBOTURK: A crowdsourcing platform for robotic skill learning through imitation. In 2nd Annual Conference on Robot Learning. CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 899- PMLR, 2018.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese. Yuke Zhu, and Roberto Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, /1 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 1678-1690. PMLR, 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj S. Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations, In Jie Tan, Marc Toussaint, and Kourosh Darvish (eds.). Conference Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pp. 1820-1864. PMLR, 2023.
Detlef Mewes and Fritz Mauser. Safeguarding crushing points by limitation of forces. Inter- national journal of occupational safety and ergonomics : JOSE, 9:177-91, 02 2003. doi: 10. 00110000000354.2003.3.11
13

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioamnis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane I Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 5187744)995 -33333,, 2015. doi: 10. 10.1038/NATURRE44335
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for gener- alist robots. CoRR, abs/2406.00523,, 2024. doi: 10.45500ARX1.224660023..
Allen Z. Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. CoRR, 00w/2409000588 2024. doi: 10.48550/AR2IV.249900055
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, AISTATS 2011, ,, Fort Lauderdale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings. pp. 627-635. JMLR.org, 2011.
Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 2019. doi: 10. wt..10//CCV.2211.000900
IEEE,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.0633 2017.
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John M. Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel X. Chang, 7solt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 251 351-266,770 2021.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin. Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Ar- nay Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskil13: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.0000 00425, 2024.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, Hc Wang. Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11094 11104. Computer Vision Foundation / IEEE, 2020. doi: 10. (199CVPP42260022000.011
Jinwei Xing. Pytorch implementations of reinforcement learning algorithms for visual continuous KarnKaraia Kar MaiirxXing//RRR /R////LLVVL/ Biaiii/gg--VVi-lll---- wwws-Contttaaaaaeeeeeoo Continuous -wwoonnContoooo.....
control. https: /t/ptttttttu////////////// con/ 2022.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
ZebraTechhologges. Autonomous mobile robots, 2024.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martin, robosuite: modular simu- lation framework and benchmark for robot learning. CoRR, abs/2009.. 12293, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SUBTASK DEFINITIONS AND INITIALIZATION
A.1.1 GENERAL INITIALIZATION
We refer the robot end-effector as ee, and rest position as poss- The end-effector resting position is (0.5 m, 0m, 1. 1.25 m) relative to the base'. Let aarm be the arm joint positions, Tarm be the arm resting joint positions, and farm be the arm joint velocities. Similarly, for the torso define Soors Too, and htoo.. Let Ubose be the base linear velocity in ms 1s_,, and Ubose,as Ubase,g its X and components respectively. Let Whase be the base angular velocity in rads-1.
Farm: We initialize the robot to P F'or with Qurm (0...0). 0), 9tor Then, farm is perturbed by clipped Gaussian noise clip(N((0,0.1) 0.2, 0.2). the base position is perturbed by clip(W((O, 0.1), -02,, 0.2), and the base rotation is perturbed by clip(AV 0.25), 0.5..
0, Vbsse = 0, Whase
A.2 SUBTASK DEFINITIONS
Let dğ = I|opos bpos l1 units in For example, d'e is the distance in between the end-effector and its rest position. Next, let jæ = max maxisisloj absolute difference in the arm joint positions and corresponding resting positions. Let Clo.r] be the cumulative robot collisions in N until step t. Finally, we define two commonly-used success conditions for the Fetch robot:
Tk.il- i For example, Jerm the maximum
grasped(2) = f i is grasped (computed by simulator)}
lss = =1
max farm, S 0.2) A Ubuse, S 0.05 A Ubase.y y S 0.05 A Whose S 0.05
Quml
Pick la, optional] (ppose): Pick object x (from articulation a, provided).
Initialization: Spawn robot facing within 2 m of x, with noise, and without collisions.
Success:
1 (grasped(x) A dec 0.05 A Jarm 0.6 li static Cio.a] 5000
Failure: Clo.)] > 5000N
Placela, optional (Xpoze 9pos): Place object goal (in articulation a, if provided).
Initialization: Spawn with grasp pose sampled from Pick( policy, robot facing g. within 2 of g. with noise, and without collisions.
Success:
(r Adg S 0.15Add S 0.05Ajarm S 0.2A S 0.01A1 is static ACaii S 7500
Failure: Cjo.] > 7500
Open [a] (apos): Open articulation a with handle apos-
Initialization: Spawn facing If a is fridge, spawn within J0.933, -0.6] x [1.833, 0.6] region in front of a, otherwise within [0.3, -0.6) x [1.5, 0.6). With noise, without collisions.
Success: Let Ggy Ogmazy agymnn be the current, max, and min joint positions for the target articulation (drawer or fridge). Then, let Oofree {0.75 if is fridge else 0.9}. We define
Lopen(a) {a, 2 anfrae (Agman aqmin)) + agmin) }
Hence, we have success condition
Iopen(a) A d= < 0.05 ^ Jarm 0.2 Jtor 0.01 A 1is static A Coas S 10000 TTe z-axis 'up' in ManiSkil13.
15

---

Published as a conference paper at ICLR 2025
Failure:: Coo.)] 10000
Closeld [a]] (dpos): ): Close articulation with handle at Opos-
Initialization:
Spawn facing If a is a fridge, spawn within [0.933. -0.6] x [1.833, 0.6) region in front of a, otherwise within [0.3, -0.6) x (1.5, 0.6). With noise, without collisions.
Success: Let Cg. Gymaz agmin be the current, max, and min joint positions for the target articulation (drawer or fridge). We define
cl)se(a) = 1 {ag S 0.01 (Agmaz - agmin) + agminl
Hence, we have success condition
close(a) dee 0.05 Jarm 0.2 ftor 0.01 A statii Owas 10000
Failure:: Coo:] 10000
A.3 RL SUBTASK EVALUATION CURVES
TidyHouse
PrepareGroceries
SetTable
100
100
100
Sucosa Orice Rata (N)
00
00-
Pick
13
200
100
100
Succes Oncc hate 190
002
02-
000
Place
82
02
40-
5.0
Per-Object
All-Object
Figure 4: Per-object all-object RL success once rate (%) evaluation curves for Pick and Place policies across tasks. We run 3 seeds for each per-object policy and seeds for the all-object policy. TidyHouse and PrepareGroceries involve objects, while SetTable involves objects. Since we group runs for different per-object policies into one curve, we use minimum and maximum for the shaded region. Best viewed zoomed.
During training, we evaluate our policies every 10000 steps on 189 episodes. The per vs all-object training curves in Fig. 4 demonstrate similar trend as seen in Sec. 6.2.1: per-object policies show the most significant improvements when grasping many objects with different geometries (Tidy- House Pick and PrepareGroceries Pick) or when manipulating objects in tight constraints where object geometry is important (PrepareGrocerries Place).
Per Tables 10-13, the performance limitations for Open and Close seen in Fig. 5 are caused primarily by the 10 000 N cumulative robot force limit set. which is not used in the original implementation of the HAB (Szot et al., 2021).
See Appendix A.4.4 for more analysis on performance under lower collision thresholds.
Oriiinally, the HAB does specify collision limits for Open Close, but add them enforce safety.
16

---

Published as a conference paper at ICLR 2025
SetTable Open
SetTable Close
Success Once Rabe (%)
100
100
60
60
40
40
20
20
0.0
0.5
1.0
1.5
2.0
2.5
Environment Samples
1e7
Environment Samples
1e7
Fridge
Drawer
Figure 5: Open and Close training success once rate (%) curves for Drawer and Fridge. Since success once rate jumps very quickly once the policy learns to solve the task, minimum and maximum for the shaded region.
A.4 ADDITIONAL EXPERIMENTS
A.4.1 DATASET SIZE
To highlight the importance of generat- ing scalable datasets, we train IL poli- cies for SywyHooueereppaeerorrrrrssss///s
Table 4: Success once rate (SoR) with 95% CIs depending on demos per object (Demos).
Pick/Place subtasks at 1, 10, 100, 500, and 1000 demonstrations per object. In Table 4, we run 1000 evaluation episodes per policy, and group results by demonstrations per object. We then report average success once rate and 95% CIs for each demonstrations per object value.
DEMOS
SoR
0.00 0.00
10
0.02 = 0.03
100
0.27 0.19
We find that 1000 demonstrations per object leads to the most performant policies. Further- more there are large jumps in success rate as demonstrations per object increases from 10 to 100 to 500.
500
0.53 0.13
1000
0.62 = 0.09
A.4.2 PERFORMANCE WITH TASK SIMPLIFICATIONS
In Sec. 6.1, we find that improving subtask success rate is the most effective way to increase long- horizon task success rate. In Fig. 6, we simplify the long-horizon tasks by (1) removing all collision requirements, and (2) marking Place subtasks successful the object remains anywhere on the target receptacle surface. We find that progressive completion rate increases for both our RL and IL policies, but we achieve no higher than 20% overall task success on any task or split. Hence, the largest challenge in training subtask policies is low-level whole-body control. Meanwhile, collision requirements and subtask success conditions pose some difficulty, but not as much.
A.4.3 SAC vs PPO FOR RL TRAINING
To justify our choices of RL algorithm for each policy, we compare SAC and PPO performance across tasks and subtasks. For Pick and Place, we compare all-object SAC and PPO subtask success once rate (%) the train split. As described in Sec. 5.1, train SAC with 25 million samples. Since PPO trains faster wall-time, we provide it 50 millions samples for fair comparison.
Furthermore, for Open and Close, compare per-object success with the same total samples as listed in Sec. 5.1, and we train SAC with 20 million samples per run.
rate (%). We provide PPO
For Pick and Place, despite training double the total samples, PPO policies achieve notably lower performance compared to the SAC policies. We hypothesize this is because our large replay buffer store greater diversity of examples across objects, spawn locations, and obstructions, allowing
17

---

Published as a conference paper at ICLR 2025
TidyHouse
PrepareGroceries
SetTable
RL Policíes
comp cemp-etionnaaa mate
80
8
60
60
pick
Alace
PK4.
0p900
atang 0.0
IL Policies
Comp at mutto to
40
1
20
2802
Hick
place
open.
plate
Su812se
Suttack
Scbass
Train Scenes
Validation Scenes
Figure 6: Progressive completion rate (%) simplified long-horizon tasks with RL and II. policies. We remove all collision requirements, and allow placing on the full target receptacle surface. Best viewed zoomed.
dodyo
PreparoGracerics
SctTablc
SetTable Open
SetTable Open Fridge
SetTable
Fridge
Figure 7: SAC vs PPO subtask success once rate (%) curves on the train split. Lines are averaged across 3 seeds; since success rate can jump rapidly, shaded regions represent min/max values. For Pick and Place, we compare all- all-object SAC and PPO policies, and for Open and Close, we compare per-object policies. Note that for PrepareGroceries and SetTable Place, lines are drawn but near- zero. Best viewed zoomed.
18

---

Published as a conference paper at ICLR 2025
Tidykousee Pek (Tnil))
TidyHnuse (990
Tidyicuue Place (hain))
TMyloouse Pacce (hll
26
303 3390
1590
3000
900
3200
3500
3000
600
560
1508
7540
600
500
1240
7500
SocTablle Pick (1r
Seribble Pice (400)
SetTuble Place (Tra01
SerThMee Place NN9)
100
330
030
100
46-
400
203
530 300
1530...
600 200
060 Fwr.a TwraTrraaaac
Thrushold
(91
RL-Per
IL
RL-AII
Figure 8: Success once rates (%) for RL-Per, RL-AII, and IL policies in Pick and Place subtasks under varying cumulative collision thresholds. Best viewed zoomed.
SAC policies to better learn manipulation in diverse settings (Haarnoja et al., 2018). Hence, we use SAC for RL Pick and Place baselines.
Meanwhile, in Open Fridge and Close Drawer PPO policies perform better than SAC policies, in Close Fridge PPO policies perform marginally worse than SAC polices, and in Open Drawer PPO policies perform better only with many more samples. Since performance between PPO and SAC is generally comparable in Open and Close, we choose PPO for our baselines since it has faster wall- time training. For consistency, we use the same RI. algorithm across all Open and Close variants.
A.4.4 PERFORMANCE UNDER Low COLLISION THRESHOLDS
To evaluate the safety of our policies in real-world setting, we compare performance for RL-Per, RL-AII, and IL policies Pick and Place subtasks under low cumulative collision thresholds. Per industry safety standards, we use 1400 N for as the measure for safe execution (Mewes & Mauser, 2003).
As seen in Fig. 8, we observe a 5-20% drop in performance depending on subtask when using a cu- mulative collision threshold of 1400 N. Additionally, the per-object RI. policies notably outperform all-object RL policies under lower collision thresholds.
A.4.5 PER VS ALI-OBIECT POLICY LONG-HORIZON PERFORMANCE
In addition to superior subtask success rates, as seen in Fig. 9, per-object RL policies outperform their all-object counterparts on full long horizon tasks both train and validation splits.
A.4.6 DIFFUSION POLICY BASELINES
To explore more complicated methods, we train diffusion policy (DP) baselines. We use a setup similar to the original DP paper, with UNet backbone and DDPM scheduler (Chi et al., 2023; 2024). For our visual encoders, we use a simpler 4-layer CNN rather than a ResNet. For consistency, use the same architecture and hyperparmeters for all subtasks.
As seen in Table 5, our DP baselines surprisingly perform generally worse than our BC baselines. Performance is closer in Place subtasks, but worse in Pick, potentially due the increased potential
19

---

Published as a conference paper at ICLR 2025
TidyHouse
PrepareGroceries
SetTable
200
100
100
Train Split
Completton hare 1900
20
20
2000
prace
poice
p
4040.0
Val Split
Complction mate 1N4
60-
80
8
60
8
1902
p00ce
p00.0
place.0
p63.3
open
place 0.
pia.1
placo
RL-Per
RL-AII
Figure 9: Success once rates (%) for RL-PPer and RL.-Al policies on long-horizon tasks in train and validation split. Best viewed zoomed.
Table 5: Subtask success once rates for BC and DP baselines.
TASK
SUBTASK
SPLIT
BC
DP
Train
61.11
28.37
Pick
Val
59.03
27.18
TidyHouse
Train
61.81
58.63
Place
Val
63.79
59.92
Train
44.64
19.35
Pick
Prepare Groceries
Val
52.78
19.74
Train
50.00
39.09
Place
Val
56.75
50.40
Train
60.71
23.71
Pick
Val
72.62
24.40
Train
71.23
64.19
Place
Val
62.20
55.36
Train
74.01
62.10
Open
Val
53.67
63.79
SetTable
Train
79.86
16.17
Open
Val
78.57
15.18
Train
86.90
64.09
Closer
Val
0.00
0.10
Train
88.39
89.29
Closedr
Val
87.60
85.81
20

---

Published as a conference paper at ICLR 2025
for collisions when picking objects. Interestingly, DP is the only baseline which achieves non-zero success on Open Fridge on the validation split, despite the fridge being against a wall (unseen in the train split).
Our results suggest that, while smaller backbones or limited tuning can solve simpler tasks like Push-T those in the ManiSkill3 standard task suite, the ManiSkill-HAB tasks might require larger/different backbones (e.g. diffusion transformer), tuning hyperparemeters per subtask, or in- cluding newer methods like online finetuning (e.g. DPPO) (Dasari et al., 2024; Ren et al., 2024).
A.5
RAY TRACING AND VISUAL FIDELITY
Figure 10: Left: ManiSkill-HAB with ray tracing on. Right: Behavior- Ik with ray tracing on. Both images live-rendered. The right image is taken from the Behavior- 1k Google Colab demo notebook (Li et al., 2022).
For visual realism, we provide live-rendered ray- ray-tracing with tuned lighting, which can be selected with only one line in the code. We compare rendering performance and quality with Behavior- Ik, platform known for its visual realism (Li et al., 2022).
To compare performance, use a single Nvidia RTX 4090, render 128x 128 RGB-D image, and simulate dynamics with a simulation frequency of 120Hz and control frequency of 30Hz. Each evaluation run consists of 300 steps of random actions clipped to [-0.3, 0.3]. We report mean and 95% CIs over 10 evaluation runs.
altered version of Behavior- 1k's rendering benchmark. We
While live-rend with ray tracing, ManiSkill-HAB achieves 69.90 + 0.25 samples per second (SPS) while using 6.26 + 0.00 GB of GPU memory, while Behavior- 1k is limited to 19.92 + 0.04 SPS while using 7.62 + 0.04 GB of GPU memory.
Hence, ManiSkillHHAB is 3.51x faster than Behavior- lk while using 17.85% less GPU memory, while also retaining similar ray-tracing render quality seen in Fig. 10.
A.6 TRAJECTORY CATEGORIZATION AND DATASET FILTERING
In this section, we provide definitions for our event labeling and trajectory categorization system. We additionally provide statistics on policy success and failure modes using trajectory categorization system. Some example videos for Pick and Place failure modes are provided in the supplementary and project website.
A. DATASET FILTERING AND GENERATION
We generate 1000 demonstrations per object/articulation for each subtask using per-object RL poli- cies on the train split. We use our trajectory labeling system to filter demonstrations (full definitions in Appendix A.6.2). For Pick, we require "straightforward success" demonstrations, where the agent successfully picks the object without dropping while remaining within the cumulative col- lision threshold. For Place, we require "placed in goal success" demonstrations, where the agent releases the object within 15cm of the goal, the object stays in the goal without rolling or falling out, and the agent remains within the cumulative collision threshold. For Open and Close, require "open success" and "closed success" demonstrations, where the agent opens/closes the articulation without excessive collisions, and the articulation remains within the open/close state.
21

---

Published as a conference paper at ICLR 2025
A.6.2 DEFINITIONS
For each success and failure mode definition, provide plain text description in addition to the boolean definitions. We heavily rely on notation used in Appendix A.I, addition to those defined below.
Using the criteria defined below, for each trajectory Tsubtask = (su, ao...... Sny an), we create chronologically ordered event list Bubttask (ck)....ck)) Using Bsubtask, categorize Tsubtask into a success/failure mode.
Let eeent {index of Gevent in Esubtask if Cevent E Psubtask else 1}. Also, let Fa,b.t be the pairwise force between a and in at timestep I.
Pick (a, (x Pick object x: (from articulation a, if provided).
Events: We define time-conditioned events at timestep t:
Coontact
F -e, A Fee, 2
Egrasped - gr) A gr))
dropped (x)) A -1 grasped(a)).
Ssuccess
- -scccess, A lscccess,t
Civ:e-11 S 5000 Clo.) >> 5000
Fexcesiive.oollsions
For & {I,... n (in increasing order), we evaluate each Cevent in the order shown above. If Cevent = 1, we add it to Epick
Success Modes: if Gsuccess E Epicks then categorize using the following success modes:
Straightforward success: Agent successfully grasps and returns to rest without drop- ping or excessive collisions.
Epick (Ccoutact, Cgrasped, Csuccess)
ii Winding success: Agent (eventually) successfully grasps x (but drops xr along the way) and returns to rest without excessive collisions.
Epick (Econtacts Egrasped: .... Csuccess) Epick| 3 A Cexcesiivecollliion & Epick
Success then drop: Agent successfully picks x and returns rest without excessive collisions, but irrecoverably drops x after.
Cdropped Eppick A dropped > "subtask A fexcessive.oolliions & Expick
iv Success then excessive collisions: Agent picks r and returns to rest, but exceeds col- lision threshold afterwards.
Cexcessive collisions E Epick
Failure Modes: if Esuccess & Epicks then categorize using the following failure modes:
V Excessive collision failure: Agent exceeds collision threshold.
Eexcessive collisions E Epick
vi Mobility failure: Agent cannot reach æ.
Epick ()
vii Can't grasp failure: Agent reaches æ, but cannot grasp it. Epiik = (Ccontact)
viii Drop failure:
Agent grasps z, but drops it before returning to rest.
edropped Epicke A buubback bsubtaskk A cexcessive.collssoos Epick
Too slow failure: Agent (eventually) grasps æ, but the episode truncates before it can reach success.
Egrasped E Epick irabbask.gresspd subbrakk A eexcessive.collisions & Epick
Place (a, optional] (X Ipos): ): Place object goal g (in articulation a, if provided).
22

---

Published as a conference paper at ICLR 2025
Events: We define time-conditioned events at timestep t:
Cgrasped -1 grappod(())) A gr)s
Cobj.at.goal d2 d22,-1 0.15 A d²,, 2, S 0.15
Creleased at goal d% - 0.15 A grasped (z),t-1 -1 (r))
Creleased oottsdeeaa d% > 0.15 A grasped(=),
grasped(u),
Eobj left. goal d2 2,1-1 0.15 A d' 0.15
Esnccess -usccess A guocess,,
Cexcessive.collisoon = Clo:t-11 S 7500 Clo:1) > 7500
For t {l,...m] n (in increasing order), we evaluate each Cevent. in the order shown above. If Covent = 1, we add it to Eplace.
Success Modes: if Gsuccess E Eplace: then categorize using the following success modes:
Place in goal success: Agent releases and successfully places æ to within 15em of Ipos: then returns to rest.
Eplacel I 4 A (Creleased at goal E Eplace V d 2.0 I 0.15) A &place,obj left goal S
(place.obj goal A Cexcessive collisions $ Eplace
ii Dropped to goal success: Agent releases x beyond 15em of Iposs x drops into the region within 15cm of 9poz. and the agent returns to rest.
pplccel I A (Creleased .outsie.ga
Éplace V a,0 > 0.15) A iplace, S
jbjatc,.bj..t.guall A Guxcessive.collssoon Eplace
Dubious success: is manipulated to within 15em of gpos, and the robot returns to rest, but æ leaves before truncation.
"placc.obj.at..uu. "place,obj.kee loat fexcessivee_colisinn & Epllace
Winding success: leaves the goal least once, but the agent (eventually) success- fully places/drops to within 15cm of Ipos, where it remains as the agent returns to rest.
|Eplace| > 4 oplace,obj at goal > |place,obj left goal A Cexcessive collisions & Eplace V Success then excessive collisions: The agent successfully places/drops to within 15em of Spos and returns rest, but exceeds collision threshold after.
Texcessive.colllsionn E Eplace
Failure Modes: Csuccess & Eplaces then categorize using the following failure modes:
Excessive collision failure: Agent exceeds collision threshold.
fexcussive.collssions E Plplace
vii Didn't 't grasp failure: Agent fails to grasp at initialization.
Eplace = () A Exxcessive collisions & Epiace
viii Didn't reach goal failure: Agent grasps v, but cannot manipulate within 15em of Spos.
Eplace| 0 gobj & Eplace A Euscessive.collssion & Eplace
ix Place in goal failure: Agent places x to within 15cm of 9poss but x leaves this region (i.e., rolls or falls out) before the agent returns to rest.
First, we define
Epllla 2A do S 0.15) V (fplace.releseedaat..aaall polace,relensten rewlossed.outsido.gooo.. oousside_goall A relace,. goal > iplace.
Hence, we have failure mode definition
fobj.at.goal E Eplace A placed latest sequence A &place,obj obj geat &place,ob).att abt. A Gexeussive.collssoons & Eplace
lequueeeeee
X Dropped to goal failure: Agent drops beyond 15em away from Jpos: and x drops into the region within 15em of Sposs but leaves this region (i.e., rolls or falls out) before the agent returns to rest. First, we define
ldropped.ss isteesdssstttttttt Iatest_eequence = (lEplacel 2nd 0.15) (iplace, releasedd.utsside.goo
pplace, relessed.at. goal A iplace (place, www.easeedooussiee.ooo.. goal Hence, we have failure mode definition
(raape))
Eohj goal E Eplace Addropped coxcessive.collssions & Feplace
sequemee A placo,,hj goal > plaace,obj goal A
23

---

Published as a conference paper at ICLR 2025
Won't it let go failure: The agent is able manipulate within 15em of Sposs but does not release 2.
eobj.at.goal E Eplace A &place.grasped
ppacceggraspe
"place, saleccceelooseelootttiie.... A
collisions Pplace
xii Too slow failure: The agent is able to manipulate x to within 15cm of the goal, releases but is unable to return to rest before truncation.
The condition is no other failure mode is applicable. This also implies
pplace,obj.. gual obj.ft,
Cexcessive collisions
colllsoive.oollisions & Eplace
Open a (apos): Open articulation with handle apos-
Events: First, define indicator
sllightlyy.oeeneddaa) 1 {agtt 2 0.1 (Aqmaz Oqmin) ) + agminn
Now, we define time-conditioned events at timestep t:
Fee Fee,a,t-1/- = A |ec,a,t) 2 20
Ccontact
Copened 1 open(o)) open(a)).
Calightly .opened
openedll
Celosed 1 open(a))e-11 - open(aa)
Csuccesss 1 success, A 1 success,
Cexcessive collisions = Cior--1 S 10000 Coaa; > 10000
For i .n) (in increasing order), we evaluate each eevent in the order shown above. If Gevent = 1, we add it to Eopen-
Success Modes: if Esucress E Eapen then categorize using the following success modes:
Open success: Agent successfully opens a and returns rest without excessive colli- sions.
eexlessive.colleiios & Eopen kopen.opened > Ropen,closer
ii Dubious success: Agent successfully opens a and returns to rest without excessive collisions, but accidentally closes a after
eexlessive.oolliiions & Eopen A oppend < <lopen,coosed
Success then excessive collisions: Agent successfully opens a and returns to rest, but exceeds collision threshold after.
eexlessive.collsions & Eopen
Failure Modes: if Csuccess & Eopen. then categorize using the following failure modes:
iv Excessive collision failure: Agent exceeds collision threshold.
Cexcessive collisions E Eopen
Can't 't reach articulation failure: Agent cannot reach a.
Ccontact & Eapen A Cexcessive collisions & Eupen Closed after open failure: Agent opens but closes it before returning rest.
Cclased E Fopen A closen > oopenn A Alpen,elosed
lapen, lopened A
dexcussive.colllsionn. & Eupen
vii Slightly opened failure: Agent at least slightly opens a, but cannot fully open it. Previous failure modes not applicable, and sipen, hipeyyyoeollyyy hopen
Popen.slightly..ogg sopened > close, A &esceesive collisions E Fapen
viïi Too slow failure: Agent is able open but cannot return to rest in time.
Previous failure modes are not applicable, and Copenod E Eapen
ix Can't open failure: Agent reaches a, but cannot open it.
The condition is no other failure mode applicable. This also implies Ccontact E Enpen A Copened Eopen
Close(a) (Apos): ): Close articulation with handle at apas-
24

---

Published as a conference paper at ICLR 2025
Events: First, define indicator
Aglightly_cloood(()) {ag, < 00,00 - 0.05 (Aymar Agmin }
Now, define time-conditioned events timestep
Ccontact Fee,a,t-11 = 0 Fee,a,t 2
Eclosed
1 clowod(a)) A cl)s
"elightly_closed (liightlyyclooood()))-- A calishdlyyclossd(aa)
Copen
1 closed(a)).-1 A I closed(a).
Cauccess -ucccces, A success,,
Gexcessive.collisioss = C0o:t-11 10000 A Ciart) > 10000
For t {l,...n] n (in increasing order), we evaluate each Cevent in the order shown above. Cevent = 1, we add it to Eclose
Success Modes: if esuccess E Eeloser then categorize using the following success modes:
Close success: Agent successfully closes and returns rest without excessive colli- sions.
eexcessive.collising collissons & Eclose felose,closed cooose telose,openee opened
ii Dubious success: Agent successfully closes a and returns to rest without excessive collisions, but accidentally opens after.
eexcessive.collliinn & Eclose cllose,closed < iolose opened
ii Success then excessive collisions: Agent successfully closes a and returns to rest, but exceeds collision threshold after.
eexlessive.colllsion & Eelose
Failure Modes: Cauccees & Eclose. then categorize using the following failure modes:
Excessive collision failure: Agent exceeds collision threshold.
fexcrssive.collisinns E Exlose
Can't 't reach articulation failure: Agent cannot reach
Ccontact & Eelose A Cexcessive collisions & Eelose
vi Opened after closed failure: Agent closes a, but opens it before returning to rest.
eclosed Feelane Velase,opened > iclose,closed A felose,opened
felose,slightly .closed
Cexcessive, fexcussivee.olllsioons wol.isioooolliiiisooooooo & Felose
vii Slightly closed failure: Agent at least slightly closes a, but cannot fully close it. Previous failure modes are not applicable, and delosn,slightly closed
> éelnse,closer A
slightyl closed felnse,apened A Cexcrssive,collisions Eetose
viii Too slow failure: Agent is able close a, but cannot return to rest in time.
Previous failure modes are not applicable, and éclosed E Eelose
Can't close failure: Agent reaches but cannot close it.
The condition is no other failure mode applicable. This also implies écontact
Eelose A Celosed Eokcse
A.6.3 TRAJECTORY CATEGORIZATION STATISTICS
In Tables 6-13, categorize trajectories with our automated event labeling method. We run 1000 episodes for every task/subtask/target combination using all relevant policies (RL-Per, RL-AII, IL) for each object/articulation. We provide success once rate (SoR), success at end rate (SaeR), failure rate (FR), and proportions for each success and failure mode as percentages (labeled with roman numerals corresponding those in Appendix A.6.2).
25

---

Published as a conference paper at ICLR 2025
Table 6: Train split Pick policy trajectory labeling on 1000 episodes per target object. All numbers are pereentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SacR
(
(i5)
(i1)
(iv)
FR
(v))
(vii)
(viil)
(ix)
RL-Pa
82.34
72.52
70.63
1.58
0.00
9.82
17.66
13:79
3.37
0.40
0.10
0.00
002
RL-AII
78.97
60.81
58.43
2.38
0.00
18.15
21.03
13.69
5.65
0.89
0.50
0.30
72.62
70.63
67.76
2.88
0.20
1.79
27.38
21.03
0.69
1.79
1.29
2.58
Re-
71.63
62.80
60.91
1.58
0.10
8.73
28.37
17.26
7.64
2,48
0.89
0.10
003
RL-AII
29.46
23:41
23.02
0.40
0.00
6.05
70.54
34.52
6.25
28.17
1.19
0.40
17.96
17.16
16.87
0.30
0.00
0.79
82.04
49.01
0.20
31.25
0,89
0.69
Re-
78.47
65.15
65.48
2.68
0.00
10.32
21.53
13.29
6.25
1.19
0.60
0.20
74.60
55,16
2.48
0.00
16.96
16.27
1.88
0.40
0.60
1L
60.81
57.44
54.27
3.17
0.00
3.37
39.19
27.18
0.50
6.05
1.59
3.87
RL-PSY
81.05
78:27
75.10
3.17
0.00
2.78
13.95
15.67
2.38
0,40
0.10
0.20
005
RI-AII
76.69
62.30
59.13
3.17
0.20
14.19
23.31
17.06
5.16
0.30
0.40
0.40
74.01
70.73
66.37
4.37
0.10
3.17
25.99
21.13
0.40
0.79
0.89
2.78
Per
83.23
80.36
78.37
1.79
0.20
2.68
16.77
10.52
5.95
0.20
0.30
0000
007
RL-AII
77.98
70.44
69.44
0.99
0.00
7.54
22.02
15.97
5.56
0.10
0.10
0.30
TH
76.59
75.20
72.42
2.78
0.10
1,29
23.41
18.95
1.59
1.69
0.20
0.99
Per
75.99
72.22
63.19
9.06
0.50
3.27
24.01
17.36
3,08
2.18
0.79
0.60
73.91
55.95
6.85
0.60
26.09
18:25
1.69
0.69
0.10
IL
68.18
62.50
54.07
8.43
0.50
2.18
34,82
26.39
0.40
3.77
2.18
2.08
RL-Per
81.25
76,00
71.63
5.06
0.40
4.17
18.75
12,40
5,46
0.50
0.30
0.10
009
RI-AII
71.43
60.42
50.10
10.32
0.40
10.62
28.57
20.54
4.86
2.28
0.60
0.30
67.16
64,29
55.85
8.43
0.79
2,08
32.84
25.00
0.79
3.08
2.38
1.59
RL-Per
81.65
52.78
48.61
4.17
0.10
28.77
18.35
12.40
4,86
0.79
0.20
0.10
010
RL-AIl
75.50
61,90
55.16
6.75
0.00
13,59
24.50
18.95
4,27
0.99
0.20
0.10
54.17
48.02
6,15
0.50
3.67
41.67
26.98
0.79
8.43
1.19
4.27
Rer
82.74
78.47
68.35
10.12
0.20
4.07
17.26
12.90
3.67
0..30
0.20
0.20
73.91
56.15
8.93
0.010
8.83
0.69
0.10
IL
62.20
$8.63
50.40
8,23
0.69
2.88
37,80
25.20
0,20
7.74
2.18
2.48
Per
81.75
73.41
67.26
6.15
0.20
8.13
18.25
12.40
4.17
1.19
0.30
0.20
all
RL-AIl
71.63
59.13
54.17
4.96
0.30
12.20
28.37
19.74
4,96
3.17
0.30
0.20
IL
61.11
59.42
54.56
4,86
0.20
1,49
38.89
26.39
0.69
7.64
0.89
3.27
RL.
69.05
55.16
50.89
4.27
0.40
13.49
30.95
14.58
16.17
0.10
0.00
0.10
002
RL-AII
62.70
49.40
38.10
11.31
1.49
11.81
37.30
24.60
11.31
0.89
0.50
0.00
IL
63.10
59.82
55.16
4.66
0.20
3.08
36.90
31.55
2.38
1.09
0.99
0.89
RL-Per
51.98
43.15
40.67
2.48
2.08
6.75
48.02
25.10
12.30
8.63
1.79
0.20
00.3
RL-
11.51
8.13
7.64
0.50
0.60
2.78
88.49
60.62
11.21
16.17
0.20
0.30
16.27
14.38
13.79
0.60
1.29
0.60
83.73
67.06
2.28
12.10
1,98
0.30
RL-Por
64.48
52.8%
50.20
2.68
0.20
11.41
35.52
19:44
13.99
0.69
0.10
1.29
004
RL-AII
59.82
46.03
37.10
8.93
1.39
12.40
48.18
26.98 98
11.41
0.99
0.40
0.40
51.09
47.32
44.84
2.48
0.50
3,27
48.91
39.98
2.38
2,88
1.98
1.69
RL.
61.21
48.51
44.74
3.77
0.30
12.40
38.79
25.79
11.51
0.10
0.20
1.19
005
RL-AII
63.29
$1.29
39.58
11.71
0.69
11.31
36.71
25.89
10.42
0.10
0.30
0.00
11
61.01
56.25
54.07
2.18
0.20
4,56
38.99
34.33
2,28
0.50
0.89
0.99
Rer
66.57
52.58
47.02
5,50
0.10
13.89
33,43
18.55
13.29
1.19
0.50
0.10
04/7
RL-
64.38
45.63
34.42
11.21
0.89
17.86
35.62
23.71
10.81
0.99
0.10
0.00
PG
61.61
57.04
2.78
0.01
1.79
38.39
3.08
1.59
0.89
0.69
RL-Por
62.90
45.75
37.00
8.73
0.89
16.27
37.10
21.33
14.38
0.79
0.50
0.10
008
RL-AII
45.93
34.03
21
12.40
2.58
9,33
54.07
36.81
12.40
3.17
1.49
0.20
40.38
35.71
32.24
3.47
1.19
3.47
59.62
49.80
2.78
3.97
2.58
0.50
Rer
63.59
46.73
41.67
5.06
1.09
15.77
36.41
18.75
16.47
0.30
0.50
0.40
009
RL-AIl
49.01
38.49
26.19
12.30
1.69
8.83
50.99
32.14
13.00
3.67
1.69
0.50
II
43.25
39.78
36.01
3.77
1.19
2,28
56.75
45.73
3.77
4.56
2.18
0.50
63.18
31.49
441.64
6.83
0.30
13.39
34,822
13.29
0.20
0.30
0.10
010
RL-AIl
54.76
42.06
27.58
14.48
1.49
11.21
45.24
30.26
12.10 10
2.18
0.30
0.40
50.99
48.61
41.77
6,85
0.10
2,28
49.01
41.27
2.68
2.38
1.69
0.99
Per
74.60
56.45
39.58
16.87
0.30
17.56
25.40
15.87
8.73
0.30
0.20
0.30
024
RL-AII
51.09
35.02
22.92
12.10
0.79
15.28
48.91
37.00
10.12
1.09
0.60
0.10
20.73
18.25
14.68
3.57
0.60
1,88
79.27
66.37
1.88
7.54
1.49
1.98
RI-
66.57
52.78
46.63
6.15
0.40
13.39
33.43
19.54
11.81
1.09
0.60
0.40
RI-AII
37.70
28.17
9.52
1.79
12.40
33.93
2.48
0.30
IL
44.64
42.36
39.38
2.98
0..00
1.8%
55.36
47.22
1.49
4.27
1.49
0.89
RL-Per
65.87
54.07
47.82
6.25
0.30
11.51
34,13
11.11
21.73
0.00
0.40
0.89
RI-AIII
59.03
33.04
2.68
0.10
23.21
40.97
16.57
24.01
0.10
0.00
0.30
IL
53.67
39.48
35.62
3.87
0.89
13.29
46.33
43.45
1.39
0.69
0.50
0.30
94.35
75.30
10.52
0.20
8.33
3.65
0.50
0.10
0.00
ST
024
RL-AII
93.95
79.37
64.68
14.68
0.20
14.38
6.05
5.06
0.30
0.00
0.50
0.20
58.63
51
6.75
0.20
6.75
34.42
24:21
1.29
5.26
2.68
0.99
Rer
80.85
69.64
60.62
9.00
0.20
11.01
19.15
8.04
10.12
0.000
0.30
0.69
all
RL-AII
75.69
56.55
47.52
9.03
0.10
19.05
24.31
11.01
12.40
0.30
0.50
0.10
60.71
50.40
44.74
5.65
1.09
9.23
29.29
31.25
1.88
3,87
1.29
0.99
26

---

Published as a conference paper at ICLR 2025
Table 7: Val split Pick policy trajectory labeling on 1000 episodes per target object. All numbers are percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SacR
(iii)
(iv)
(vi))
(viii)
(ix)
KL-Fer
80.06
70.63
68.95
0.30
9.13
19.94
15.77
3.67
0.30
0.00
0.20
002
RI.-AII
78.57
61.71
59.82
1.88
0.00
16.87
21.43
15.18
5.36
0.69
0.10
0.10
73.61
72.22
69.64 64
2.58
0.20
1.19
26.39
21.03
0.20
1.69
2.38
Rer
73.41
64.38
61.61
2.78
0.10
8.93
26.59
16.67
6.94
1.98
01.99
0.00
RL-AII
33,73
26.29
7.34
33.13
24.50
1.09
0.20
IL
16.67
16.17
15.58
0.60
0.10
0.40
83.33
47.52
0.10
33.93
0.69
1.09
RL-Per
77.28
69.44
05.96
0.00
7.34
22.72
13.39
6.85
1.29
0.79
0.40
004
RI--AI1
75.20
57.44
54.07
3.37
0.00
17.76
24.80
15.99
6.35
1.88
0.20
0.40
IL
59.62
57.34
54.76
0.20
2.08
40.38
27.78
0.40
7.24
1.19
3.77
81.15
78.08
75.79
2.28
0.10
2.98
18.85
15.87
2.78
0.20
0.00
0.00
005
RL.AII
75..5
61.90
58.43
3.47
0.10
13.49
24.50
18.55
5.06
0,20
0.40
0.30
IL
71.03
69:44
65.77
3.67
0.20
1.39
28.99
21.92
0.99
2.28
0.79
2.98
RL-Por
82.74
77.98
76.69
1.29
0.00
4.76
17.26
11.61
5:46
0,20
0,000
0.00
007
RL-AI1
77.78
69.44
68.75
0.69
0.00
8.33
22.22
15.18
6.55
0.40
0.00
0.10
TH
IL
72.72
71.23
67.66
3.57
0.10
1.39
27.28
22.22
1.29
1,79
0.10
1.88
RI-Per
75 75
70.34
9.72
4.37
24. 60
18.45
2.58
2.48
0.79
0.30
008
RLI-AII
72.02
63.19
55.95
7.24
0.40
8.43
27.98
19.54
5.26
2.38
0.79
0.00
61.61
60.32
54.76
0.79
38.39
29.37
2.58
1.69
RL. Per
78:57
74.70
69:15
5.56
0.30
3.57
21.43
14.99
5.26
(1.89
01.20
0.10
RL-AII
71.63
60.66
.2.778
10.71
20.34
0.59
0.10
IL
64.48
62. 20
54.96
7.24
1.09
1.19
35.52
28.57
1.09
2.78
1.88
1.19
RL-Per
82.24
53.37
50.00
3.37
0.00
28.87
17.76
13.19
0.50
0.00
0.00
010
RI.-AI1
74.90
63.99
94
7.04
0.10
10.81
25.10
19.25
4.46
0.79
0.30
0.30
IL
55.75
52.68
46.83
5,85
0.10
2.98
44.25
27.88
0.30
9.42
1.69
4.96
Per
79:56
74.31
63.79
10:52
0.50
4.76
20.44
15.38
4,46
01.20
0.200
0.00
024
RL-AII
69.74
62,90
53.37
9.52
0.00
6.85
30.26
22.32
7.14
0,50
0,20
0.10
IL
58.04
54.76
44.94
9.82
0.40
2.88
41.96
29.96
0.30
7.04
1.39
3.27
RL-Per
77.48
69:44
65.77
3:67
0.30
7.74
22:52
16.37
4.36
0.89
0,00
0.20
RI..AnI
15
53.97
10.81
31.85
21.73
0.10
0.40
IL
59.03
57.04
53.87
3.17
0.20
1.79
40.97
28.47
0.30
8.44
1.39
2.38
RL-Per
84.62
67.56
63.69
3.87
0.89
16.17
15.38
12.10
2.88
0.10
0.10
0.20
RL-AnI
76.98
54.27
41.47
12.80
2.68
20.04
23.02
20.34
0.99
1.29
0.30
0.10
IL
74.11
68.45
64.58
3.87
1.19
4.46
25.89
20.93
0.50
1.29
2.18
0.99
RL-
56.15
42.56
2.88
2.38
11:21
43.85
30.46
2.08
9.72
1.39
0.20
00.3
RI-AIII
14.19
11.41
10.02
1.39
0.60
2.18
85.81
57.24
0.69
26.88
0.99
0.00
16.77
0.79
81.75
61.90
16.67
2.58
0.40
RL-Por
69.74
60.32
57.24
3.27
0.30
8.93
30.26
25:00
2.08
1.29
0.40
1:49
RL-AII
71.23
19.25
23.61
2.78
0.89
0.20
IL
58.43
53.57
51.19
2.38
0.40
4.46
41.57
34.33
0.30
4.17
1.49
1.29
Re-
7649
64.
60.12
4,76
0.20
11.61
23.31
21:43
0.69
0.60
0.30
0.30
003
RI.-A11
75.66
53.97
40.87
13.10
1.19
20.44
24.40
21.03
1.09
1.79
0.30
0.20
II
75.00
68.35
65.67
2.68
0.30
6.35
25.00
20.54
0.79
1.19
1.29
1.19
Rer
76.98
56.25
50.69
5.56
0.30
20:44
2X.022
19.35
2.58
(..0)
01.40
0.10
007
RL-AII
76.98
56.15
41.96
14.19
0.40
20.44
23.02
19.64
1.59
1.79
0.00
0.00
PG
IL
72.02
69.84
65.48
4.37
0.10
2.08
27.98
21.83
1.69
2.78
0.79
0.89
RL-Por
73.21
48.91
37.60
11.31
0.69
23.61
26.79
22.52
1.59
1.98
0.30
0.40
RI.-An1
58.93
24.70
18.15
3.67
12.40
41.07
29.07
2.38
2.98
0.00
II
44.35
37.40
32.04
5.36
1.88
5.06
55.65
44.15
1.59
5.65
3.47
0.79
70.73
38.00
1.88
30.85
26.19
01.20
0.20
009
RI-AIII
62,90
46.92
29.37
17.56
2.88
13.10
37.10
26.09
2.38
5.36
2.98
0.30
44.54
38.79
1.79
3.97
42.56
3.67
0.40
RL-Por
74.30
55.26
48.61
6.63
0.60
18.65
25.30
23.91
0.30
0.79
0.30
0.00
010
RL-AI1
70.04
49.40
34.42
14.98
2.88
17.76
29.96
23.41
2.38
3.57
0.60
0.00
IL
59.23
57.04
52.18
4,86
0.50
1.69
40.77
33.93
0.69
2.98
2.18
0.99
Rer
78.17
55.65
39.09
16.57
0.10
22.42
21.83
15.08
5.75
0.50
0.10
0.40
024
RI.-An
61.21
37.00
25.79
11.21
1.19
23.02
38.79
35.81
0.69
1.49
0.79
0.00
IL
22.82
20.14
16.27
3.87
0.30
2.38
77.18
65.28
0.79
7.64
0.89
2.58
RL-Pez
72.32
$3.47
45.93
7.54
0.50
18.35
27.68
23.02
1.39
2.38
0.79
0.10
RL-AII
62.10
43.75
31.25
12.50
1.79
16.57
37.90
28.47
0.89
7.54
0.89
0.10
IL
52.78
47.52
43.75
3.77
0.50
4.76
47.22
36.90
0.69
6.55
1.79
1.29
RL-Per
80.36
58.43
48.02
10.42
0.20
21.73
19.64
17.96
0.30
0.00
0.20
1.19
013
RI-AII
66.67
26.29
19.74
6.55
0.10
40.28
33.33
31.94
0.89
0.30
0.20
0.00
IL
78.08
55.56
51.09
4.46
1.49
21.03
21.92
18.06
1.49
0.30
0.69
1.39
RL-Par
93.65
83.33
71.83
11.51
0.20
10.12
6.35
5.85
0.10
0.20
0.20
0.00
RI-AII
89.98
77.38
62.00
15.38
0.40
12.20
10.02
0.30
0.20
0.20
0.00
II
62.20
54.56 86.86
48.61
6.25
0.20
7.14
37.50
26.59
2.28
5.95
1.39
1.59
Rer
89.49
70:04
18.45
11.51
10.42
0.40
0.69
all
RL-AII
79.86
52 58
40.58
12.00
0.20
27.08
20.14
19.44
0.30
0.10
0.20
0.10
72.62
58.63
52.98
13.79
27.38
21.03
2.68
1.09
0.69
27

---

Published as a conference paper at ICLR 2025
Table 8: Train split Place policy trajectory labeling on 1000 episodes per target object. All numbers are percentages. Best viewed zoomed.
TASK
TYPE
SoR
SacR
(i)
(ii
(iii)
(v)
(vi)
(vii)
(vii)
(ix)
(x)
(xii)
RL-Par
69:44
65.38
44.35
19.05
1.39
1.98
2,68
30.56
20.14
2,85
2.28
3.67
1.39
0.00
002
RL-AIl
67.76
58.04
26.69
29.37
0.79
1.98
8,93
32.24
19.15
0.00
1.19
4.27
6.85
0.79
0.00
II.
70.44
61.11
51.59
7.84
0.66
1.69
8.63
29.56
18.25
0.00
3.47
4.66
2.38
0.40
0.40
RL-Per
54.07
40.48
27.
1.79
2.38
11:21
45.93
0.50
4.17
0.99
7.64
0.69
003
RL-AII
52.58
42.86
22.92
18.25
0.99
1.69
8.73
17.42
31,05
0.00
0.79
8.43
5.95
0.89
0.30
II.
47.32
35.71
20.54
4.86
4.37
10.32
7.24
52.68
32.74
0.20
4.66
5.06
5.06
4.27
0.69
RI-
54.76
46.23
6.25
34.92
0.50
5.06
8.04
45.24
5.16
1.29
6.85
0.50
004
RL-AII
55.06
47.22
21.13
24.40
0.50
1.69
7.34
44.94
24.21
0.00
1,59
5.16
7,94
5.95
0.10
II
$1.59
45.44
15.97
27.08
0.89
2.38
5,26
48.41
22.32
0.10
5.16
4.66
6.65
9.03
0.50
RI-
65.97
56.05
43.06
8.13
0.30
6,86
9.62
34.03
24.01
0.10
1,49
4.37
1.39
2.48
0.20
005
RL-AII
65.77
59.23
21.53
35.42
0.69
2.28
5,85
34.23
22.62
0.10
2.18
2.48
6.55
0.20
0.10
IL
62.90
55.95
41.27
12.00
0.69
2.68
6.25
37.10
21.92
0.60
7.04
3.97
3.08
0.40
0.10
RL-Per
73.61
66.37
29.96
33,93
0.10
2.48
7.14
26.39
17.96
0.00
0.69
1.39
6,05
0.00
0.30
007
RL-AII
70:54
63.19
23:12
39 09.09
0.20
0.99
7.14
29,46
18.95
0.00
2.38
3,47
4.17
0.30
0.20
TH
Re-
IL
70.93
75.40
69.74
63.10
30.36
33.93
27.48
35.71
0.10
0.30
3.67
1.69
5.56
7.54
24.60
29.07
19.05
17.76
0.00
0.40
0.79
4.86
0.69
1.59
3.77
1.79
0.10
2.08
0.20
0.60
RI-AII
68.06
61.51
20.93
38.
0.00
1.29
6.55
31.94
23.51
0.00
3.27
2.088
1.88
1.19
0.00
IL
68.65
60.12
38.49
19.84
0.20
1.79
8,33
31.35
21.43
0.50
2.98
1.49
1.88
2.98
0.10
Re-
68.35
53.67
13.69
38.79
0.10
1.19
14.58
31.65
24.11
0.40
2.18
1.49
2.98
0.40
0,20
RI-AII
65.67
19.54
38
0.00
1.59
6.05
34.33
25.99
3.27
1.09
3.47
0.50
0.00
IL
59.62
$1.09
35.81
13.29
0.00
1.98
8.53
40.38
27.48
0.69
4.56
2.9%
1.19
2.68
0.79
Re-
71.43
37.54
24.30
30,16
0,10
2.73
13.49
23.37
22.62
0.00
2.08
0.39
1.29
1.69
0.00
RI-AII
73.31
64.98
24.01
0.10
1.49
8.23
26.69
17,46
1.98
2.28
4.07
0.89
0.00
IL
69:35
60.71
39. 38
18.85
0.00
2.48
8.63
30.65
21.03
0.00
2.98
3.27
1.39
1.09
0.89
Re-
70.73
60.62
15.21
7.6:
0.00
7.74
10.12
29.27
21.33
0.00
0.10
5.36
0.79
0.79
0,40
RLAAII
64.38
26.79
23.81
0.10
6.05
35.62
27.38
2.48
3.67
0.20
II.
64.78
57.55 54
43.06
9.62
0.10
86
7.14
35.22
20.83
0.10
5.96
6.45
2.18
0.00
0.30
RL-Per
65.77
55.65
26.79
25,30
0.30
3.37
9.82
34.23
22.82
0.00
1.79
2.68
3:17
3.37
0,20
RL-AIl
63.69
23.12
31.75
0.50
6.55
36.31
2.38
2.58
4.76
0.20
11.
61.81
54.55 56
35.32
15.97
0.30
3.27
6.94
38.19
23.02
0.40
5.06
3.27
3.17
2.68
0.60
Per
62.30
52.48
14.88
33.93
1.29
3.67
8.73
37.50
26.69
0.10
6.05
0.60
1.79
1.98
0.30
002
RI-AII
56.35
32.64
21.23
9.13
2.18
2,28
21.53
43.66
27.88
0.69
8.23
2.78
2.58
1.09
0.40
56.65
29.17
2.78
3.47
11.21
3.77
4.17
Rer
52.28
49:21
22.92
24.31
0.99
1.98
2.08
47.72
30.85
0.10
3.56
4.17
2.08
4.86
0.10
0033
RL-AII
47.82
28.57
20.93
4.76
3.57
2,88
15.67
52,18
34,52
0.10
6.94
8.13
1.49
0.50
0.50
41:57
17.76
15.28
2.78
5.26
13.79
8,83
4.56
55.95
46.92
22.52
20.34
0.10
3.87
8.93
44.00
32.34
0.00
3.37
2.68
2.68
2.58
0.20
004
RL-AII
52.08
32.84
23.12
6.94
0.20
2.78
19.05
47.92
29.96
0.00
6.15
5.36
2.08
3.87
0.50
49.1
35
12,70
0.40
1.49
4.37
30.06
8.83
6.75
3.47
0.40
RL-Ver
62.60
50.89
39.38
7.34
0.50
3.87
11.21
37,10
25.20
0.10
4.16
3.16
2.38
0.10
0.00
005
RL-AII
56.
37.20
26.39
8.53
0.20
2.28
18.75
43.85
29.96
0.30
7.04
1.19
3.47
1.29
0.60
11.
56.94
$1.98
41.17
8.53
0.89
2.28
4.07
43.06
27.08
6.45
4.76
3.37
1.09
RL-Ver
63.79
55.95
25.17
22.52
0.20
5,26
7.64
36.21
27.08
0.00
5.65
1.49
1,88
0.00
0.10
00/
RL-AII
56.35
35.91
23.12
11.61
0.00
1.19
20.44
43.65
28.57
0.30
8.33
1.29
1.49
3.57
0.10
PG
RL-
11.
62.30
58.95
$1.49
$2.08
29.17
16.96
21.23
32.14
0.20
0.60
2.98
1.09
10:02
3.87
44.05
37.70
28.17
23.12
0.10
0.30
8.13
7.24
2.48
3.27
3.17
3.47
0.10
1.09
0.89
0.20
008
RL-AII
55.46
38.19
23.02
13.49
0.00
1.69
17.26
44.54
30.95
1.09
7.94
1.19
1.39
1.59
0.40
II.
54.27
50.69
27.18
19.64
0.10
3.87
3.47
45.73
23.21
0.99
9.23
4.07
4,46
1.79
1.98
RL-
63.39
54.56
24.31
28.27
0.10
1.49
8.73
36.61
25.89
0.10
6.35
1,38
2.38
0.00
0.00
009
RL-AII
52.58
33.13
18.75
12.20
0.00
2.18
19.74
47.12
35.42
0.50
5.95
2.38
0.99
1.59
0.30
II
56.65
53.77
35.62
15.77
0.10
2.38
2.78
43.35
24.90
0.79
8,83
4.07
1.98
0.89
1.88
Rer
63.10
41.77
18.15
19.35
0.30
4.27
21.03
36.90
31.75
0.10
3.57
0.60
0.60
0,20
0.10
010
RL-AII
$7.74
38.39
22.22
13.29
0.10
2,88
19.25
42.26
27.98
0.60
6.94
1,98
2.58
1.98
0.20
II
55.56
44.94
27.78
15.08
0,10
2.08
10.52
44.44
30.85
0.40
6.55
3.17
0.99
0.10
2.38
62.44
46.41
13.89
22.82
0.00
9.72
15.87
37.30
27.78
2.88
0.89
6.05
0.10
024
RL-AII
51.9%
29.59 56
12.00
9.72
0.00
7.84
22. 22.42
48.02
38,79
0.00
4,37
1.88
2.58
0.20
0.20
II
50.30
46.73
21.33
19.15
0.30
6.25
3,27
49.70
31,55
0.10
12,20
2.18
3.37
0.00
0.20
Rer
60.22
48.61
22.31
22,72
0.30
3.57
11.31
39.78
27.38
5.75
2.45
2.68
0.99
0.30
all
RL-AII
53.37
35.13
18.85
10.12
0.89
4.17
19.35
46.63
33.13
0.60
7.34
2.18
1.69
1.59
0.10
IL
50.00
45.63
25.99
17.26
1.09
2.38
3,27
50,00
28.67
0.50
11.01
4,96
3.37
0.10
1.09
RL-
68.95
61.71
18.45
39.38
1.59
3.87
5.65
31.05
22.022
01000
0.60
1.69
1.98
4,76
0.00
RL-AII
39.29
24.11
0.99
3.97
6.45
25,20
0.60
2.28
2.38
II
65.58
62.50
11.31
46.33
1.29
4.86
1.79
34.42
21.03
0.89
5.95
1.49
4.66
0.00
0.40
RC-
78367
76.39
33.31
293.46
0.30
13.39
1.79
21.33
16.07
(1.00
0.40
1,85
2.28
01.41
0.30
ST
RL-AII
71.33
30.65
12.00
0.20
20.24
8.23
28.67
3.27
1.88
0.10
II
73.51
72.72
23.41
39.68
0.50
9.62
0.20
26.49
10.52
0.00
6.75
4.17
4.96
0.00
0.10
Re-
73.31
68.35
25.20
34.23
(1700
8.93
4:27
26.89
17.56
0.20
0.79
2.58
2.88
2.38
0.30
RL-AII
72.82
35.02
17.36
12,70
6.85
27.18
20.14
1.29
2,18
2,88
0.30
IL
71.23
69.15
19.05
43.55
0.89
6.55
1.19
28.77
14.19
0.50
6.05
3.17
4.56
0.00
0.30
28

---

Published as a conference paper at ICLR 2025
Table 9: Val split Place policy trajectory labeling on 1000 episodes per target object. All numbers are percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SaeR
(i8)
(ii)
(iv)
(4)
FR
(vi)
(vii)
(viii)
(i)
(X)
(xi)
(xii)
RL-
61.70
58.25
37.40
18.65
1.79
2.18
2.68
37.30
25.10
0.10
5.27
3.17
3.17
2.28
0.20
002
RL-AII
66.37
58.04
26.88
29.17
0.60
1,98
7.74
33.63
22.02
0.00
1.39
3.27
4.96
1.79
0.20
11
69
60.71
50.00
9.05
1.59
1,69
7.14
30.56
20.73
0.10
3.17
3.87
1.49
0.79
0.40
RL-Fer
36.65
10.37
27.95
1,98
3.27
10.91
12:50
43.33
28.17
0.00
0.40
4.37
1.39
8.43
0.60
003
RL-AII
53.57
44.15
26.09
15.28
1.39
2.78
8.04
46.43
30.95
0.00
1.09
10.42
3.67
0.30
0.00
1L
46.23
36.21
19.84
4,86
11.51
53.77
34,82
0.00
2.98
4.76
1.39
Rl.
52.88
24.34
6.135
12:14
0.79
6.35
7.54
47.12
24.50
0.00
4,36
2:28
9:13
5.55
0.09
004
RL-AII
52.88
45.75
21.92
22.22
0.50
1.59
6.65
47.12
26.79
0.00
1.88
5,46
7.24
5.36
0.40
53.77
48.1
26.79
3.08
23:41
5.85
6.05
0.40
Re.
65.97
58.33
48.02
6.15
0.40
4.66
6.75
34.005
22.62
0.00
1.69
5.26
0.89
2.88
0.69
005
RL-AII
67 066
59.25
25.55 89
91.45
0.50
1.88
7.34
92.994
21.23
0.00
1.39
3.27
6.65
0..40
0.00
11.
59.82
47.92
9.92
1,98
34.92
21.43
0.40
3.77
3.87
0.60
RL-Par
72.22
64.98
30.35
32.24
0.00
2.38
7:24
27.78
19:44
0.00
0.89
1.38
5.16
0300
0.40
007
RL-AII
69.74
64.19
25.00
38.10
0.20
1.09
5.36
30.26
20.93
0.10
1.69
2.28
4.66
0.50
0.10
TH
RL-Par
11.
75 30
63
64. 98
69.44
27.37
35.12
39.48
27.58
0.00
0.30
2.28
1,59
6.35
5.85
28.37
24.70
18.65
19.35
0.00
0.00
0.99
2.78
0.69
2.78
3.17
1.79
2.28
0.30
0.30
008
RL-AIl
71.13
62.50
22.52
38.59
0.00
1.39
8.33
28.87
21.63
0.00
1.69
1.09
2.98
1.39
0.10
I1
73.41
65.87
44.54
19.74
0.00
1.59
7.54
26.59
18.75
0,20
1.88
0.79
0.99
3.57
0.40
RL-Per
68.55
54.27
17.96
34.33
1.98
31.45
24.70
0.10
009
RI--AII
63.44
57.24
20.54
34.82
0.00
1,88
6.25
36.51
28.27
0.00
3.57
1.29
2.68
0.69
0.00
11
58.23
50.20
35. 12
12.00
0.10
3,08
7,94
-1.77
29.07
0.50
5.06
1.29
2.18
2.78
0.89
08.
511.56
27.18
3.97
31.25
23.20
010
RI--
68. 15
59.33
21.13
37.20
0.10
0.99
8.73
31,85
20.73
0.00
2.68
2.28
5.36
0.69
0.10
IL
66.37
61.41
40.28
18.15
0.00
2.98
4.96
33.63
24.70
0.00
3.57
2.28
1.09
1.29
0.69
RT.
74.
63.09
6.63
10.11
25.30
20.83
0.00
2.78
11.9
0.10
024
RL-AII
66
58.75
27.88
22.32
0.00
8.53
8.93
32.34
24.40
0.10
1.19
2.08
4.57
0.10
0.10
IL
68.45
59.82
46.73
9.33
0.00
3.77
8.63
31.55
18.45
0.00
4.96
5.85
L.98
0.00
0.30
Rl.
65.99
56.45
12.04
20.24
0.20
4.17
9.33
34,03
24.31
0.00
1.09
2.08
3.67
2.68
0.20
all
RL-AII
66
53.23
24:90
31.25
0.20
2,08
7.64
33,93
21.23
0.00
1,88
3.37
5.36
1.88
0.20
11.
63.79
56.94
37. 30
15.38
0.89
4.27
5.95
36.21
21.83
0.20
3.97
3.37
3.77
2.68
0.40
Re.-Fer
69.15
55
17.36
33.33
1.39
4.96
11.61
30.85
23.52
0.00
2.58
0.79
2.38
2.38
0.20
082
RI-AII
18.83
29.96
21.92
6.45
1.98
1.59
26,
26.88
41.17
29.27
1.39
3.67
3.08
2.48
1.19
0.10
58.93
50.79
22.42
27.08
1.29
17.16
Rer
58.33
53.37
24.50
23.30
1.09
3.37
3:27
41.67
27.18
0.00
3,08
4,46
3:47
2.95
0.50
0003
RI-AII
50.20
27.58
21.03
3.87
1.79
2.68
20.83
49,80
36.81
0.00
3.97
7.64
0.79
0.50
0.10
44.13
39.55
1.29
20.63
3.67
RI
60:71
71-71
27.58
17.16
5.57
11:21
39.25
28108
0.39
5:57
3:17
0.30
004
RL-AIl
59.52
30.56
20.75
5.85
0.00
3.97
28.97
-0.448
27.98
0.00
2.28
4.37
1.09
4.46
0.30
II
53.77
49.21
33.23
14.98
0.99
0.99
3.57
46.23
23.12
0.10
6.65
8.73
4.66
2.08
0.89
RT.
47:72
18:99
5.36
5:37
18:25
32.64
21112
0.10
1:29
13:40
0.10
005
RL-AIl
62. 10
37.00
26.09
8.83
0.30
2.08
24.80
37,90
28.67
0.30
2.38
2.48
2.98
0.99
0.10
II
62 20
58.56
44.25
8.23
1.59
3.08
5.06
37.80
19.64
0.30
11.11
3.77
1.88
0.00
1.09
RL-Par
75.79
39.92
27.5%
26.29
5.75
19.25
0.40
007
RL-AIl
62.70
35.02
22.72
10.32
0.10
1.98
27.58
37.30
28.97
0.40
2.78
0.79
1.88
2.28
0.20
PG
RL-Pa
11.
71.53
65.77
58.35
59.62
29.76
27.88
36.90
0.60
3.97
1.98
5.46
28.47
34.23
18.45
16.07
0.20
9.62
2.28
3.17
2.98
0.00
2.18
008
RL-AIl
54.46
31.35
16.47
13.10
0.00
1.79
23.12
45.54
34.42
0.79
5,06
1.79
1.19
2.18
0.10
11.
$9.42
55.56
28.87
23.51
0.30
3.17
3.57
40.58
17.46
0.69
9.82
3.37
4.37
2.78
2,08
RL.
66.37
57.44
25.39
28.77
0.00
2.78
9.42
33.13
23.62
0.50
5.56
1.88
2:38
0.10
009
RI-AII
54.86
33.04
20:24
11:31
0.00
1,49
21.83
45.14
32:24
0.79
6,25
1.98
1.49
1.88
0.50
IL
59.13
55.85
34.42
18.06
0,20
3.37
3.08
-10887
15.28
1.09
15.67
3.87
1.79
1.85
1.29
RL-
68.00
39.38
17.36
19.64
0.10
1,88
28:57
31.94
27.98
0.00
1.39
0.79
1.39
0.40
0.00
RI-AIl
37.
21.92
12.80
2.18
27.38
35.52
27.38
0.79
2.78
1.29
0.10
IL
62. 30
49.31
29.76
17.66
0.50
1,88
12.50
37.70
21.63
0.50
8.13
3.37
1.69
0.10
2.28
17
49.31
12:30
23.31
0.101
13.79
17.76
32.34
23:21
0.000
1.19
1.39
6:34
03101
0.21
RL-AII
56.75
9.72
6.15
28.37
37.00
0.00
II
$4.07
48.02
24.90
17.96
0.10
5.16
5.95
45.93
23.71
0.00
14.09
3:77
3.97
0.00
0.40
65.07
52.38
24:31
22.72
0.79
5.36
12.30
34.33
25.10
0.000
2,78
2.58
3.57
0359
0.1R1
RL-AIl
33.04
11.31
2.78
24.70
31.65
IL
$6.75
52.08
30.16
19.15
0.60
2.78
4.07
43.25
21.33
0.40
12.00
3.97
2.98
0.69
1.88
RL-Per
64.38
59.05
19.35
36.11
0.79
3.57
4.56
35.62
25.00
0.00
0.30
1.59
2.28
6.45
0.00
013
KI-AII
67.96
60.91
40.08
16:27
1.09
4.56
5.95
32.04
25.60
0.10
1.19
2.78
1.69
0.60
0.10
IL
61.71
59.62
10:22
45.24
0.89
4.17
1.19
38.29
25.50
0.30
6.65
1.29
4.37
0.00
0.20
RL.
69:44
66.47
24:21
11.31
2.68
30.56
25.10
0.30
ST
024
RI-AII
67.26
60.12
29.96
10.81
0.10
19.35
7.04
32.74
25.10
0.00
2.28
3.08
1.9%
0.00
0.30
IL
65.67
64.68
21.03
37.10
0.40
6.55
0.60
34,33
18.65
0.10
8.83
2.68
3.97
0.00
0.10
RL.
67.06
30.06
7.24
32.94
24,50
1.88
all
RL-AIl
68.
61.31
33.73
16.07
0.10
11.51
6.55
31.75
21.90
0.00
1.19
3,08
2.58
0.00
0.00
IL
62.20
59.92
16.57
38.10
1.49
5.26
0.79
37,80
22.32
0.40
7.84
3.08
3.57
0.10
0.50
29

---

Published as a conference paper at ICLR 2025
Table 10: Train split Open policy trajectory labeling on 1000 episodes per target articulation. All numbers are percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SacR
(
(ii)
(ii)
FR
(iv)
(V)
(vi)
(vii)
(viii)
(ix)
RL-Per
84.92
84.92
84,92
0.00
0.00
15.08
11:41
1.69
0.00
1.49
0.20
0.30
drawer
ST
79.86
79.86
79.86
0.00
20.14
13.59
0.89
1.09
0.10
0.79
fridge
Rer
83,43
82.24
82.24
0.00
1.19
16.57
14.58
0.20
0.00
1.49
0.20
0.10
IL
74.01
68.85
68.85
0.00
5,16
25.99
22.12
0.99
0.10
1.69
0,40
0.69
Table 11: Val split Open policy trajectory labeling on 1000 episodes per target articulation. All
numbers percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SaeR
(i)
(ii)
(iii)
FR
(iv)
(v)
(vii)
(viii)
(ix)
RL-Per
84,32
84.52
84.32
0.00
0.00
13,48
11.41
1.88
0.00
1.88
0.00
0.30
drawer
78.57
78.37
78.37
0.00
0.20
21.43
13.69
1.29
4.17
0.89
ST
fridge
Re-Per
88.10
37.30
37.30
0.00
30.79
11.90
6.94
0.00
0.00
4.07
0.89
0.00
11
53.67
1.49
1.49
0.00
52.18
46.33
43.85
0.00
0.00
1.29
0.89
0,30
Table 12: Train split Close policy trajectory labeling on 1000 episodes per target articulation. All
numbers percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SaeR
(i)
(ii)
(iii)
FR
(iv)
(v)
(vi)
(vii)
(viii)
(ix)
RL-Per
88.79
88.79
88.79
0.00
0.00
11.21
3.17
2.68
0.30
0.89
4.17
0.00
drawer
88.39
88.39
0.00
0.00
11.61
3.27
3.08
0.00
ST
fridge
Rer
86.81
25.00
25.00
0.00
61.81
13.19
13.19
0.00
0.00
0.00
0.00
0.00
11.
86.90
29.96
29.96
0.00
56.94
13.10
12.90
0.00
0.00
0.20
0.00
0.00
Table 13: Val split Close policy trajectory labeling on 1000 episodes per target articulation. All
numbers percentages. Best viewed zoomed.
TASK
OBJ
TYPE
SoR
SaeR
()
(8)
(i1)
FR
(iv)
(v)
(vi)
(vii)
(viii)
(ix)
RL-Per
89.29
89.29
89.29
0.00
0.00
10.71
4.56
2.18
0.20
0.20
3.37
0.20
drawer
ST
Re-
0.00
0.00
0.00
0.00
0.00
100.00
81.15
18.85
0.00
0.00
0.00
0.00
fridge
I1
0.00
0.00
0.00
0.00
0.00
100.00
95.93
4.07
0.00
0.00
0.00
0.00
30


Reviews:
[{"review_id": "OOGQLoRwsf", "reviewer": "Reviewer_XeS8", "summary": "The paper introduces **ManiSkill-HAB**, a comprehensive benchmark for low-level manipulation and in-home object rearrangement tasks. It aims to enhance the Home Assistant Benchmark (HAB) with a GPU-accelerated implementation for improved simulation speed and realism. The authors included extensive reinforcement learning (RL) and imitation learning (IL) baselines and developed an automated rule-based filtering system for generating demonstrations that are compliant with predefined safety and behavior rules. \n\nTogether, this benchmark provides:\n1. fast, realistic, diverse simulation tasks and environments for home scale manipulation challenges\n2. support for low-level control that enables realistic grasping and interaction\n3. extensive RL and IL baselines\n4. vision-based robot dataset at scale\n\nI recommend acceptance for this paper since it provides clear and important contributions to facilitate the advancement of in-home manipulation and embodied AI research.", "questions": "1. Simulation limitations:\nWhat are the limitations of simulation technologies used in this work? Would it be possible to simulate deformable objects, fluids, or more intricate rigid objects such as tools? Are there plans to expand the manipulation tasks beyond pick + place and open + close?\n\n2. Real-World Application:\nWhat are the anticipated challenges in transferring the learned behaviors from the simulated MS-HAB environment to real-world robots, particularly in unstructured environments like typical homes?\n\nFor future works, it would be interesting to study how methods that solve this benchmark at X% success rate would transfer to real world robotics.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality\n- While there have been benchmarks for robotic manipulation and household-related task manipulation, this work focuses on tasks and objects prevalent in home assistant settings.\n- Environments in this work are GPU-accelerated and significantly outperform prior works in terms of speed and computational costs.\n- While filtering trajectories with privileged information from the simulator is not unseen, the ability to scale up the simulation and rendering at a faster speed makes sampling and filtering more trajectories feasible.\n\nQuality:\n- From the supplementary videos, the simulation environments and rollouts appear to be high quality.\n- The comparisons to prior work show a clear advantage of the proposed method in simulation speed.\n- The RL & IL baseline methods are extensively studied using this benchmark, providing future research good baselines.\n\nClarity:\n- The writing, figures, and supplementary materials are well-presented and easy to follow.\n- The evaluation protocols for the baseline methods are structured and presented clearly.\n- The authors also included failure modes for each task in the supplementary material.\n\nSignificance:\n- The benchmark attempts to address a critical need in the robotics community for more efficient and realistic simulation tools that can keep pace with the increasing expectation of robots performing complex tasks in daily environments.\n- The potential impact on future research, particularly in home rearrangement tasks, is significant, providing a robust platform for developing and testing new algorithms and approaches.", "weaknesses": "1. Currently, this work consists of three long-horizon tasks: TidyHouse, PrepareGroceries, and SetTable. For future iterations, it would be beneficial to expand the tasks and manipulated objects beyond HAB and YCB datasets. Potential tasks could include cleaning dishes, laundry tasks, and tool usage.\n\n2. The RL and IL baselines include SAC, PPO, and BC. It would be greatly beneficial to the research community to have more recent baseline methods such as TD-MPC2, ACT, Diffusion Policies, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5le4aa3A3D", "reviewer": "Reviewer_W29p", "summary": "The paper updates HAB benchmark to MS-HAB for GPU-based simulation and larger-scale evaluation. Especially, it adopts Maniskill3 (Sapien) as the backend instead of the original Habitat (Unity) backend. It is worth mentioning that the evaluation is now end-to-end in the physics engine and there is no magical grasping like in Habitat. Furthermore, it conducts more experiments on reinforcement learning / imitation learning methods for long-horizon manipulation tasks, and show the challenge of the benchmark.", "questions": "1. Comparison with other benchmarks.\n2. SPA pipeline for IL dataset.\n3. More baseline comparison.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The updates are necessary for speed and physical fidelity.\n3. The tool is widely useful, especially for long-horizon manipulation tasks like home-scale rearrangment challenge.\n4. The baselines are more comprehensive in this paper comparing to previous papers, which provides a large amount of insight into the benchmark.", "weaknesses": "1. The novelty of the benchmark needs discussion. Especially when RoboCasa, Behavior-1k platforms are already published for a while (these are non-current work). In terms of the scale / scene diversity / task difficulty, why MS-HAB is comparable or is different with these existing pipelines.\n2. The imitation dataset is generated with RL. Howerver, one can also generate (for pick and place) with sense-plan-act pipeline. Why not generate dataset with SPA pipeline?\n3. It will be great if one provide baseline results for Senes-Plan-Act (TAMP), and with oracle information. This will provide valuable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "94C4gy8bMg", "reviewer": "Reviewer_KUnv", "summary": "This paper presents MS-HAB, a benchmark for low-level manipulation and in-home object rearrangement aimed at supporting embodied AI research. The benchmark features a GPU-accelerated Home Assistant Benchmark (HAB), which the authors claim achieves over three times the speed of previous implementations, and provides extensive reinforcement learning (RL) and imitation learning (IL) baselines for future comparisons. Additionally, a rule-based trajectory filtering system has been developed to select demonstrations that meet specific behavior and safety criteria. Ultimately, this enhances simulation efficiency, and the authors hope their work will support scalable data generation for robotics research.", "questions": "1. Why does the paper's baseline training algorithm not compare with the methods proposed in M3? M3 achieves a much higher task completion rate for the three tasks than the performance presented in this paper.  \n2. Does the per-object policy refer to training a specific pick or place policy for each different object (e.g., bowls, cans, cups, etc.)? While this might improve completion rates for certain tasks, could it significantly limit the generalization capability for complete long-horizon tasks?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The benchmark provides a holistic framework for home-scale rearrangement, featuring a GPU-accelerated implementation of the Home Assistant Benchmark (HAB) that supports realistic low-level control, enabling more effective manipulation tasks.\n2. It includes extensive RL and IL baselines, allowing researchers to compare their methods against established standards and fostering advancements in these areas.\n3. The systematic evaluation approach, utilizing a trajectory labeling system, enhances the benchmark's reliability and provides detailed insights into performance.\n4. The implementation of demonstration filtering allows for efficient and controlled data generation at scale, which is crucial for developing robust robotic systems.", "weaknesses": "1. There is significant room for improvement in the existing baselines, as the current performance may not meet the highest standards set by more advanced methods in the field.\n2. The authors do not claim that the benchmark supports transfer to real robots, indicating that there are still challenges to be addressed in applying these methods in practical scenarios.\n3. In fact, most of the ideas and techniques used in this paper have appeared in prior work, particularly the M3 framework proposed by Gu et al. [1]. Both the skill sequence partitioning for three long-horizon tasks and the training algorithms for various skills are fundamentally consistent with M3. However, the authors do not provide a detailed comparison with it in the paper.\n4. The technical contribution is quite limited. The simulation environment, baseline algorithms, and even the subtask definitions used in the paper have already been proposed in previous work. This makes the draft somewhat more like a technical report.\n\n[1] Gu, Jiayuan, et al. \"Multi-skill Mobile Manipulation for Object Rearrangement.\" The Eleventh International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hb5iiSUdjk", "reviewer": "Reviewer_HBjR", "summary": "This paper introduces a robotic manipulation benchmark for home-scale rearrangement tasks, called ManiSkill-HAB. The authors implement the Home Assistant Benchmark (HAB) within the GPU-accelerated simulator ManiSkill3. The resulting environments achieve high simulation throughput, outperforming previous implementations by a factor of three. Additionally, the robot and object interactions are simulated with accurate rigid body physics to facilitate the learning of low-level manipulation skills. Finally, reinforcement learning (RL) and imitation learning (IL) models are trained to serve as baselines for future research. A rule-based trajectory filtering system is employed to selectively subsample demonstrations generated by RL policies.", "questions": "- In Figure 7 (Appendix A.4.3), you compare the performance of training per-object policies to that of a generalist policy capable of grasping all objects. Are the per-object policies trained concurrently, and does their combined experience total 1e7 environment samples? If each per-object policy is trained on 1e7 samples individually, it seems that the RL-All variant should also be allocated num_objects * 1e7 environment steps to ensure a fair comparison.\n- You mention that by using depth images, you address a partially observable variant of the MDP (line 161). However, the policies are parameterized as MLPs following a CNN encoder. Are there any mechanisms, such as history-awareness or the concatenation of consecutive frames, to handle the partial observability? Alternatively, is there an argument that partial observability has minimal impact on the tasks being considered?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Highly Relevant Task**\n- The problem of household rearrangement is highly relevant in robotics, and the research community greatly benefits from accessible, high-quality benchmarks. In this regard, the paper provides valuable foundational elements for future research to build upon.\n- Furthermore, introducing accurate low-level control instead of relying on \"magical\" grasping is an important addition. For instance, using realistic initializations for the Place task by sampling grasp poses from the learned Pick policies is a compelling improvement.\n\n**Writing**\n- Overall, the writing is clear, and the logical flow of the paper effectively conveys the goals and proposed contributions. However, I noticed a few minor issues where I think the authors could improve the writing:\n    - In line 213, when describing the observation space, you use the phrase \"if the object/target is grasped.\" To enhance clarity, I suggest rephrasing it to something like \"an indicator of whether the object/target is grasped\".\n    - In line 366, \"pariwise\" should be \"pairwise\".\n    - The Open X-Embodiment reference is quite lengthy, taking up almost an entire page. To improve readability and structure, I recommend using \"et al.\" after the first author’s name, rather than listing all the authors.\n\n**Reproducibility**\n- The code and data used in the paper are publicly available, and the experiments are described in detail.\n\n***\n\nOverall, the paper tackles an important problem, and making the environment code available to the research community will benefit other researchers by providing a foundation for future work.", "weaknesses": "**Novelty**\n- The primary contribution of this work is the implementation of the HAB in the ManiSkill3 simulator. While this undoubtedly makes it more efficient for researchers to work on this problem, the novelty is somewhat limited. That said, the combination of low-level manipulation and long-horizon tasks, common in household settings, is intriguing. In particular, exploring how these tasks can be integrated to mitigate hand-off issues between independent modules could add significant value. However, by studying the subtasks in isolation and replacing navigation with robot teleportation (e.g., lines 348-350), the tasks are simplified to pure manipulation problems. It’s also worth noting that ManiSkill3 already includes a drawer-opening task with the Fetch robot out-of-the-box (https://maniskill.readthedocs.io/en/latest/tasks/mobile_manipulation/index.html).\n\n**Baseline Methods and Evaluation**\n- Of the four tasks studied (Pick, Place, Open, Close), SAC and PPO are each applied to two tasks, respectively. This makes it difficult to assess the relative difficulty of the tasks or the comparative strengths of the RL methods used. While Appendix A.4.3 explains that PPO was chosen for the Open and Close tasks to enable faster wall-time training, I find this reasoning unclear, especially since SAC demonstrated superior performance in both per-object and all-object grasping tasks. Given that the Open and Close tasks are not trivially solved, with success rates still below 90%, a structured evaluation of both SAC and PPO across all tasks would likely provide more meaningful insights.\n- The rationale for using imitation learning with behavior cloning (BC) as a second baseline is unclear to me. First, since the RL teacher policies can be queried for expert actions, I would expect that using DAgger, which continually aggregates the dataset during training, would lead to better performance than relying on BC with a static dataset. The rule-based filtering of trajectories before adding them to the replay buffer in DAgger could be applied similarly here. Secondly, it’s unclear what is gained from this imitation learning step. Since the RL policies already operate from visual observations, there doesn’t appear to be any knowledge distillation that would justify the need for IL policies (for example to transfer the knowledge to a deployable observation space). If the goal is to shape behavior towards specific aspects of the RL policy’s learned behaviors to boost performance, we would expect an improvement in task success rates, which, according to Table 1, does not seem to be the case.\n- The reported success rate is defined as \"the percentage of trajectories that achieve success at least once in an episode\" (lines 343-344). However, without a clear mechanism to infer from the used visual observations whether a subtask has been successfully completed and then halt execution, this measure seems overly optimistic. The success rate should either be measured at the end of an episode after a fixed time, or the policy should be equipped with the ability to terminate an episode when it determines that the task has been successfully completed. These adjustments would provide a more accurate reflection of the performance expected on a real system.\n- While I understand that the primary focus of this work is on the simulation benchmark, incorporating real-robot transfer would greatly enhance the ability to assess how realistic the simulated rigid-body physics are in enabling low-level manipulation behaviors. This is particularly important given the claim of \"realistic low-level control\" made in the Conclusion. One concern I have is the classification of cumulative robot collisions exceeding 5000N as \"excessive collisions.\" In collaborative robotics, the acceptable force range is typically an order of magnitude smaller (https://pubmed.ncbi.nlm.nih.gov/12820907/). Additionally, in lines 910-912 of the Appendix, it’s mentioned that violations of the 10000N force limit are the primary performance bottleneck in the Open and Close tasks. This raises questions about the deployability and realism of the learned manipulation behaviors.\n\n***\n\nOverall, while the implementation of the HAB benchmark in a GPU-accelerated simulator is valuable for the research community, the contribution is somewhat incremental. A more structured evaluation of the proposed environments, concerns about the realism of low-level control behaviors due to high observed collision forces, and the lack of calibration or transfer to a real-robot system— which would significantly strengthen the claim of realistic low-level control—leave room for further improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)