FetchBench: A Simulation Benchmark for Robot Fetching
Authors: Beining Han
Meenal Parakh
Derek Geng
Jack A Defay
Gan Luyang
Jia Deng
Venue: CoRL 2024 (2024)
Keywords: Grasping; Benchmark; Imitation Learning

Abstract: Fetching, which includes approaching, grasping, and retrieving, is a critical challenge for robot manipulation tasks. Existing methods primarily focus on table-top scenarios, which do not adequately capture the complexities of environments where both grasping and planning are essential. To address this gap, we propose a new benchmark FetchBench, featuring diverse procedural scenes that integrate both grasping and motion planning challenges. Additionally, FetchBench includes a data generation pipeline that collects successful fetch trajectories for use in imitation learning methods. We implement multiple baselines from the traditional sense-plan-act pipeline to end-to-end behavior models. Our empirical analysis reveals that these methods achieve a maximum success rate of only 20%, indicating substantial room for improvement. Additionally, we identify key bottlenecks within the sense-plan-act pipeline and make recommendations based on the systematic analysis.

FetchBench: A Simulation Benchmark for Robot
Fetching
Beining Han" Jack A Defay
Meenal Parakh
Derek Geng Jia Deng
Gan Luyang
Princeton University
Fetching, which includes approaching, grasping, and retrieving, is a critical challenge for robot manipulation tasks. Existing methods primarily focus on table-top scenarios, which do not adequately capture the complexities of en- vironments where both grasping and planning are essential. To address this gap, we propose a new benchmark FetchBench, featuring diverse procedural scenes that integrate both grasping and motion planning challenges. Additionally, Fetch- Bench includes data generation pipeline that collects successful fetch trajecto- ries for use in imitation learning methods. We implement multiple baselines from the traditional sense-plan-ac pipeline to end-to-end behavior models. Our em- pirical analysis reveals that these methods achieve a maximum success rate only 20%, indicating substantial room for improvement. Additionally, we iden- tify key bottlenecks within the sense-plan-act pipeline and make recommenda- tions based on the systematic analysis. The code for the benchmark is available https: thub. www.ccoo//...... htwpe//etteee///eeeeeeeeeee--eee
Abstract:
Keywords: Grasping: Benchmark; Imitation Learning
Introduction
We study robotic fetching: the full process of approaching, grasping, and retrieving an object from the environment. Fetching unknown objects in unseen environments is crucial to various manipu- lation tasks and applications. Yet, current approaches [1, 2, 3, 4, 5] primarily focus on computing grasp poses for unknown objects limited environment settings, such table-top and bins. This contrasts with everyday scenarios where often need to pick objects from shelves, cabinets, draw- and other locations. In these situations, the task extends beyond just finding suitable grasp poses. It also involves moving objects from cluttered environments to desired location, e.g., the free space near the robot for in-hand manipulation [6] or above a container to drop the object [7].
Given the gap between the complexities of real-world scenarios and those typically addressed in current research, remains unelear whether existing fetching methods can handle unknown objects in novel and complex daily environments and what are the failure edge cases of such methods.
To address these questions, we introduce FetchBench, a benchmark designed for the large-scale evaluation of methods that integrate grasp prediction with motion planning. FetchBench specifically targets the task grasping and retrieving objects in complex scenarios commonly encountered in daily life, such as retrieving items from shelves, drawers, and cupboards. This contrasts with existing manipulation benchmarks (listed in Table 1), which predominantly focus general skill learning and have scenarios limited to picking objects from tabletop environments.
We use procedural scenes to generate fetching queries, which are unlimited and comprehensive (Figure 1). contrast, all other simulation benchmarks use hand-crafted environments and thus are
CCoresponding author: (47032@@prineeoonu
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Figure 1: Examples of the fetching task in our benchmark. The red object is the target object that needs to be retrieved from the cluster to the free space in each task. Our scenes and tasks are generated with procedural rules, which mimic daily environments like shelves, cabinets, drawers, baskets, etc. The images are rendered with Isaac-Sim [8].
limited in the variation of the spatial constraint of the cluttered scene. Compared to Cabinet [9], whose procedural assets are not publicly available, build our scenes with public Infinigen [10] assets and have increased diversity of in-basket objects.
Based on the benchmark, we implement and evaluate several baselines for the task. In particular, the common pipeline used in previous works [1, 11, 5] is to use grasp pose prediction models to propose candidate grasp poses and then use motion planning [12] algorithms to search for collision-free free motion reach the target location. We find in Section that this pipeline is longer reliable when fetching from novel, complex, and partially-observed environments, achieving only 13% success rate. We analyze the failure cases by placing oracle components within the pipeline and point out several challenges that remain on existing motion planning algorithms and grasp pose prediction models (Section 6.2).
In addition to the common pipeline, we opt to learn large-scale behavior model for the task. We generate a dataset using the procedural scenes and expert grasping demos with CuRobo [13] and ground truth grasp annotations. We find that the best performance comes with combining the com- mon pipeline with the behavior model (Section 6. 6. using the conventional pipeline in the approach- ing phase while using the end-to-end behavior model for moving the grasped object to free-space.
In summary, we make the following contributions:
FetchBench: A benchmark designed evaluate robotic methods for fetching unknown objects unseen environments.
An empirical study conducted on the benchmark, which identifies key challenges and failure cases of existing methods.
dataset generator developed for imitation learning methods in robot fetching tasks.
Related Works
Manipulation Benchmark in Simulators. Benchmarking robotic systems and algorithms in sim- ulation has the advantage of high efficiency, low cost, and reproducibility. Compared to exist- ing benchmarks [14, 17, 31, 22, 18, 24, 15, 16, 27. 29, 21] for robot manipulation (Table 1), our benchmark is unique in the following aspects: First, we specialize in evaluating robot fetching of unknown objects from novel environments, while other benchmarks either focus on general- purpose robot learning algorithms, e.g. imitation learning [22, 18. 17, 27], reinforcement learning [18, 14, 24, 16, 17, 29], long-horizon embodied AI challenge by simplifying the grasping pro-

---

Table 1: Comparison with other robot manipulation benchmarks in simulation. Some entries adapted from [14]. For rows of [14, 15, 16], the number of scenes denotes the number of rooms. IL: Imitation learning. RL: Reinforcement learning. LfD: Learning from demonstration. EA/: Embodied AI SPA: Sense-Plan-Ace [17, 15J. BRL: Batch RL/Offline RL. EAI-G: Embodied Al with abstraction of grasping process. M2RL: Meta and Multi-task RL.
Platform
Focus
Scenes
Objects
Demos
Baselines
Robosuite [18, 19]
Mujoco [20]
LID.RL
1
10
IL,BRL,RL II.
RoboCasa [21]
Robosuite [18]
LID
120
2509
RLBench [22]
CoppeliaSim [23]
LfD.RL
1
28
IL,RL
Metaworld [24]
Mujoco [20]
M2RL.
1
80
M2RL
Behavior- [M0 [16]
iGibson2 [25]
EAI
100
1.2k
RI
Maniskill2 [17]
Sapien [26]
LfD.RL
1
2k+
SPA.IL,RL
FurnitureSim [27]
Isaac-Gym [28]
LID
35
IR,BRI.
eumanoidBench [29]
Mujoco [20]
RL
15
20+
RI
Behavior- 1k [14]
OmniGibson [14]
EAI
306
5.2k+
RL
Habitar2-HAB [15]
Habitat [30]
EAI-G
6
100+
SPA.RL
ManipulaTHOO [31]
A12-Thor [32]
EAI-G
30
150
RL
FetchBench (Ours)
Isaac-Gym [28]
Fetching
Proc
5.5k+
SPA.IL
cess [15, 31]. Second, our benchmark provides abundant procedural environments for training and evaluation, with total of 5.5k objects for grasping. Other benchmarks are limited by the number of artist-designed environments and the number of objects to be grasped.
Robotic Fetching. Robotic fetching (grasping) a popular and challenging problem [33]. Re- searchers have built robotics systems that can grasp diverse and unknown objects from table-tops [4] and bins [5, 4]. In these works, the common pipeline to predict grasp poses [35, 3, 2, 34] and use motion planning to approach the grasp pose and retrieve the object of the cluttered scene [13, 12, 9]. The alternative is learn end-to-end policy with imitation learning [36, 37]. However, the evaluations in these works, though real-world, are severely limited to ~10 scenes and <100 objects [1, 3, 33] or only focus on collision detection and motion planning while ignoring the grasp- ing [9]. In contrast, FetchBench evaluates the full pipeline study both the modular approaches built around the common pipeline and end-to-end approaches, ablating over different components with oracles. Previous works have not conducted such a study.
Task Setup
We procedurally generate scenarios (Section 4) in IssacGym [28] and spawn 7-DOF Franka robot arm equipped with a Franka gripper front of the scene. We use two cameras on either side of the robot for partial point-cloud of the environment. The camera poses are randomized while maintaining good view of the whole scene. Additionally, we assume that segmentation masks of the robot and the target object (to fetch) provided, and the point cloud is converted into the robot- base frame. These assumptions valid in practice with eye-to-hand calibration, with off-the the-shee- segmentation [38], and tracking models [39].
9..... +
Joiint gr ippe- states
[Aq Aq Aqp]
[g]
delta joint novements
Agent
Initial State
Seprented Paintcloud
Success
Figure 2: Example of the fetching task, including the initial state, the success grasp state, and the task inputs segmented point cloud and joint states.
3

---

Given the segmented point cloud of the scene and the joint states, the goal of the robot is to fetch the target object from the scene to the free space near the robot (Figure 2). The robot's trajectory considered successful the task completed, with no other object significantly displaced. Fur- ther, similar to [13, 12], we use the average computation time and C-space trajectory length of all successful grasps as secondary metrics. Please refer to Appendix A. for more details task setup.
Benchmark & Dataset
To benchmark the robot's ability to fetch objects in complex environments, create FetchBench to mimic various commonly occurring but challenging scenarios from the real world (Figure 1). Here, we describe the scene generator and the dataset generation pipeline for fetching.
4.1 Procedural Task Generation
Scenes. We create 13 types of procedural scenes made with objects such as shelves, cabinets, drawers, dressers, baskets, etc. Each scene type combines several different procedural assets from Infinigen [40, 10]. Our scene generation procedure has a total of 310+ independent parameters that be randomly sampled to create diverse scene instances. Next, automatically annotate support surfaces (e.g., table-tops, shelf boards) in procedural scenes spawn the objects. In particular, there are four categories of support surfaces: on-table, on-shelf, in-drawer, and in-basket. These labels allow us to evaluate methods based on scene category. For more details on procedural scene generation, please refer to Appendix A.2.
Objects. While majority of the objects come from the ACRONYM dataset [41], we additionally include procedurally generated objects from Infinigen [10] such plates, food bags, containers, forks, etc. We collect total of 5544 objects and label the grasp poses in [41, 4] with Isaac-Gyn simulator [28]. We use a train-test split of roughly 7:1 for the objects, where the test split is solely used for evaluation and benchmarking purposes.
Figure 3: Examples of procedural scenes designed with Infinigen [10] assets (first row). A similar scene can be replicated in the real world with IKEA furniture (second row). The scenes are rendered Infinigen.
Tasks. Each task instance is generated by randomly sampling a scene and placing several objects on the support surfaces with random stable poses, similar to [41]. The robot position, camera poses and the physical constants of friction, coefficient of restitution, and object densities are also random- ized within a suitable range.
We note that some task instances might be infeasible to solve, so we filter out instances where (1) an object's initial pose IK solution does not exist for all annotated grasp poses, and (3) the target object is near absent in the input point cloud. In all, generate total of 6k tasks for testing in benchmark: 1526 on-table
static (i.e. the object is falling off after initialization), (2) collision-free

---

cases, 2724 on-shelf cases, 891 in-basket cases, and 859 in-drawer cases. Appendix A.3 shows more examples.
4.2
Fetch Dataset Generation
To train and evaluate imitation learning methods, generate own dataset of fetch trajectories. For random scene composed of objects in the train split, we iterate over all valid grasp poses of the target object and use motion planning for the approach and retrieval phases. In particular, we use CuRobo with ground truth objects and robot meshes for motion planning. the retrieving phase, when the object attached to the end-effector, we use the sample-surface approximation [13] of the target object. We find that CuRobo [13] is more efficient and finds shorter trajectories as compared OMPL. [42] planner.
While unlimited data can be generated with the above pipeline, here we generate a dataset of 27.5k trajectories with over 3.6M frames on 5.7k fetching task instances.
Baselines
We choose baselines involving both the sense-plan-act pipeline (Figure 4) and the imitation learning approach trained on our Fetch Dataset. For the former, we further consider variants with different motion generation modules: CuRobo [13], RRTConnect [43], and MPPI [44] in Cabinet [9]. For the latter, we consider fully end-to-end implementation and a hybrid approach that combines the pipeline with the behavior-cloned policy in the retrieval phase.
The sense-plan-ae pipeline starts with point cloud inputs, and motion plans the trajectory from the initial state to pre-grasp and similarly from post-grasp to free space. Note that while planning a motion for post-grash to free space, we treat the object as an additional fixed link to the end-effector.
gggasp prodiction
pre-grasp
3. to gresp pose cartesian Inear mation gripper
4. post-grasp
5. free-space
Predict object grasp
object grasp posa with cartesian
poses from partial point- deud, eg. CGN (10)
to pre-grasp pose, P.g. CuRabo
post-grasp pose to
Figure 4: The sense-plan-act pipeline commonly used in grasping frameworks [1].
ContactGraspNet-MMooii Planning]. We follow the pipeline in Figure 4. Here, we use Contact- GraspNet for 6D grasp pose prediction [1]. We test two motion plamning algorithms, i.c., CuRobo [13] and RRT-Connect [43], and name them as CGN-CuRobo, CGN-RRTCCnnect respectively.
Cant Murali et al. [9] proposes a neural collision checker for partially ob- served objects and scenes. We re-implement the same pipeline above, but MPPI [44] iterations with Cabinet's neural collision checking and waypoints proposals as the motion planning module.
End-to-End Behavior Cloning, We train an end-to-end policy to grasp the object from the cluster with expert demos. In our architecture, we first encoder the segmented point cloud into 1-D embed- dings and concatenate them with joint states and end-effector embeddings. These embeddings are then passed backbone to output the delta movements. We two variants for the backbone: an MLP [19] that takes the current step embeddings, and transformer [45] that takes in a history of embeddings. We name these E2EImit-MLP and 22Elmit--Taanffnnn respectively. Please refer to Appendix B.2 for details.
Learning. This method is combination of CGN-CuRobo and imitation learning. It similar CGN-CuRobo except for retrieval steps (4) and (5) in Figure 4, end-to-end behavior model to command the robot. Similarly, we implement where instead Ceroooo-Imittteee Imit-MLP and CON-CuRobo-Iminn MaauRoooomit-TTaaffrreeee
ContacGrrsppeeeCCRooooniiiii
two variants: CGN- CGN-CuRobo-
5

---

Table 2: Evaluation of baselines benchmark. We show the average rate, the average computation time and trajectory c-space length on the success cases.
Total Suce
Computation Time (s)
C-Space Length (Rad)
CGN-CuRobo
0.094
16.4
6.17
CGN-RRTConnect
0.131
139.0
12.4
CGN-Cabinet
0.121
42.3
5.82
E2Elmit-MLP
0.082
58.6
38.4
PHElmit-Transfone
0.099
57.5
22.3
CCN-CulRobo-mit-MM.P
0.200
17.0
20.7
CCN-CuRabb-lli--Tranofoo
0.203
16.1
6.32
Experiments
We perform all experiments in Isaac-Gym [28] simulator on 6k test task instances. We all the methods' hyper- parametars on held-out validation tasks generated from the train split.
We design experiments to address the following key questions:
How do the baselines compare on FetchBeneh? the fetching task close be solved?
2. In the pipeline-based approaches, which component(s) present bottlenecks in object fetching?
6.1 Baseline Comparisons
Table 2 shows the performance of all baselines the benchmark. We first note that the maximum success that any baseline could achieve only about 20%, indicating that the task is far from being solved.
Next, we observe that the end-to-end behavior cloned policies have < <10% success rate. Specifically, we find that the grasping phase presents significant challenges for an end-to-end trained model, the model must learn both collision-fri planning and generalize grasp various object shapes. Thus, dataset and benchmark offer challenging testbed for researching imitation learning methods in object fetching, where the methods be evaluated based on their ability to generalize unknown objects and scenes.
Lastly, we found that the CGN-CuRobo-Ine Imit variants perform the best, with the transformer variant achieving 20% success rate. Moreover, this variant is nearly as fast and efficient as other motion pootion-planning based variants. The performance im- provement seems to result from strategically decomposing the task into three main components and selecting algorithms op-
Eatlegoryy
rondtinel
Butts
timally suited for each. In the approaching phase, it's more effective to train a grasp pose prediction model that general- izes well novel objects and scenes rather than an end-to- end policy. During the retrieval phase, the behavior model sidesteps motion planning challenges in partial point-cloud en- vironments by leveraging prior knowledge implicitly learned from training demonstrations.
Figure 5: The success rate of dif- ferent methods by category.
We also compare the performance across the 4 scene categories described in Section 4.1. As shown in Figure 5, fetching from shelf, boards, and baskets is significantly more challenging than from table tops and drawers. This experiment emphasizes the importance of evaluation benchmarks that encompass the diverse see- marios typical in daily life.

---

6.2 Baseline Ablation Study
In this section, conduct an ablation study for methods that follow the pipeline in Figure 4, specif- ically testing it with oracle components: ground truth annotated grasp poses and an accurate scene mesh for motion planning context. Here, we answer the following questions:
Table 3: Success rate of different ablations of the common grasping pipeline. Additionally, we break down into approach phase motion-planning success rate (Approach Plan), approach phase execution success rate (Approach Exec), retrieval phase plan (Retrieval Plan). The approach phase execution is considered successful if the end-effector reaches the commanded pose within small erorr threshold.
Approach Plan
Approach Exec
Retrieval Plan
Final Suce
GA-Mesh-
0.865
0.861
0.776
0,671
GA-Mesh-RRTCommect
0.947
0.903
0.681
0.529
GA-Ptd-CuRobo
0.792
0.705
0.321
0.190
GA-Pid-RRTCommect 0.949
0.678
0.502
0.306
CGN-Mesh-CuRone
0.611 1
0.602
0.546
0.336
CGN-Mesh-RRTCommed
0.607
0.478
0.275
Q1: How successful is the fetching pipeline when it uses accurate grasp annotations and perfect scene meshes? In this ablation, we replace CGN with annotated grasp poses that offer collision- free IK solution and employ the ground-truth scene mesh for motion planning. We term this variant as Grasp Anmotation-Mestt Motion Planning ablation (GA-Mesh-[MP)) In Table 3, we find that even with ideal grasp and scene information, the success rates achieved are only 67% with CuRobo [13] and 53% with RRTConnect [43]. Analyzing the failure cases, we find with GA-Mesh-CuRee motion planning fails in 14% of cases reaching the pre-grasp pose and 9% reaching the end-state. In another 10% of cases, the object slips from the gripper during fetching. This slipping is often due to the discrepancy between the grasp annotation process in free space and the actual grasp execution in cluttered scenes. While in free space, the object pose adjusts according to the gripper's 's pose; in cluttered scenes, objects may collide with nearby items or support surfaces, causing failures for several grasp poses.
Q2: How much does using partial point-cloud data for motion planning contribute to bottle- necks? We use partial point clouds for motion planning context alongside annotated grasp poses in this ablation, termed GA-Ptd-[MP]. In Table 3, we find that GA-Prd-CuRobo achieves 19% success rate, whereas GA-Ptd- RRTConecttt reaches 31 Planning with a partially observed point cloud often tends to underestimate the collisions during execution, leading to more failures: GA- Pid-RRTTCnneet finds a collision-free free path from the initial state to the pre-grasp pose in 95% of cases but only achieves a 68% success rate execution. In the retrieval phase, although plans successfully 50% of the time, the final success rate is just 31%. This drop from planning success to final success notably larger compared to the GA-Mesh ablation, trend that is consistent with the CuRobo variant.
Q3: How much does low-quality grasp pose prediction contribute bottlenecks? We use CGN to predict the grasp poses and use scene mesh for motion planning. We denote this ablation as CGN-Mesh-[MP) ablation. We find that
achieves 34% and CGN-Mesh- drop of approximately 50% compared to GA-Mesh ablations, suggesting space for improvement of the grasp pose prediction model.
RRTConnect achieves 28%, which corresponds
Can we solve the grasping tasks by naively repeating the pipeline multiple times? To determine repeating the fetching procedure can resolve failures, we retry up to five times initially fails to complete the task. Figure 6, we compare between with only one iteration and repeat x5 under different ablations. We observe that repetition improves success rates by margin of 13% for different ablations. However, the task still remains challenging. In addition, despite the improvements, the extra cost is substantial: the average computation time almost doubled, and
Q4:
7

---

Sata
RATGooneet CoN--hc-Culttta CGN Pho--Po--- onnCt
5.c0me Pare
L
Dee
Pageral
Figure 6: The ablation of repeating the common grasping pipeline by maximal of 5 times. We show the comparison the success rate, computation time and trajectory length. Here, CGN-Ptd- [MP] denotes standard setting with only point-cloud and joint states as inputs. the average trajectory length increased by 75%. In all, doing naive retrials is inefficient strategy, and we believe developing more flexible re-grasping skills necessary.
7 Real Robot Experiment
We show that the challenge of fetching from complex environment exists in real-world environment. Figure shows the example scenes of real-world fetching tasks. We evaluate on 52 on-table tasks, 83 on-shelf tasks, and 32 in-basket tasks with total of 26 different objects. For the real-robot experiment, we use CGN-RRTConneet with Movelt[[2].
In total, success rate of 21%, in- cluding 38% on on-table lasks, 13.3% on on-shelf tasks and 12.5% on in- basket tasks. The result suggests the challenge of fetching unknown objects from complex and cluttered scenes (e.g., shelves, bas- kets) in the real-world. Moreover, out of all the failure cases, 25.8 % comes out of low-quality grasp poses (the object slips away), 27.3 % comes out of motion planning failure, 23.5 % comes from unexpected collision with the scene, and the remaining comes from no grasp proposals. We refer Appendix for the experiment details, evaluations and videos.
achieve
Figure 7: Examples of real-world fetching experiment environments.
8 Discussion and Limitation
In our study, we introduce FetchBench, simulation benchmark for evaluating various robotic meth- ods for the fetching task. We evaluate several methods FetchBench, ranging from traditional sense-plan-act pipelines to end-to-end models. These methods struggle in complex scenarios with a maximum success rate of just 20%, indicating substantial room for improvement. Through the ablation studies and the use of oracle components, we have pointed out critical bottlenecks related partial observability, grasp prediction inaccuracies, and ineffective retrial strategies. We hope that the introduction of this benchmark will serve as valuable tool for effectively measuring progress in the fetching task, which is crucial component of many robotic manipulation systems.
Finally, our study has several limitations. First, FetchBench is simulation based benchmark that may not accurately represent real-world performance due to sim-to-real discrepancies, especially in physics simulations during contact. Second, our evaluations assume perfectly segmented point clouds, unlike real conditions where segmentation noise exists and non-Lambertian objects present challenges in point cloud extraction. Third, while framework is general, our evaluations are only centered around Franka arm and gripper. Future work should extend the benchmark to different robotic systems, with increased coverage of object categories.

---

Acknowledgments
This work was partially supported by the National Science Foundation.
References
[1] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact- -ottactgrasppnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 18438-13444 2EEE,00111
[2] H.S. Fang, C. Wang, M. Gou, and Lu. Graspnet-Ibillion A large-scale benchmark for general object grasping. In Proceedings the IEEE/CVF conference computer vision and pullern recognition, pages 11444-11450 2020.
[3] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4g: Amodal single-view single-shot (3) grasp detection in cluttered scenes. In Conference robot learning, pages 53-65, PMLR, 2020.
[4] M. Gilles, Y. Chen, T. R. Winter, F. 7. Zeng, and A. Wong. Metagraspnet: A large-scale bench- mark dataset for scene-aware ambidextrous bin picking via physics-based metaverse synthesis. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), pages 220-227. IEFE, 2022.
[5] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv: 170.70033333 19311, 2017.
[6] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In- hand reorientation of novel and complex object shapes. Science Robotics, 8(84))eade9244, 2023. fof:100.126/ssiiroootiss... ad44444.. URL https://wwt http:://wwsssceeceeee.. science. org/ hetg//di//ee mosi/abs/10. 1126/scirobotics.oo adc9244.
[7] M. Bajracharya, J. Borders, R. Cheng, D. Helmick, Kaul, D. Kruse, J. Leichty, J. Ma, C. Matl, Michel, et al. Demonstrating mobile manipulation in the wild: metrics-driven approach. arXiv preprint arXiv:2401.01474. 2024.
[8] Isaac sim robotics simulation and synthetic data nvidia developer. https://www.einsaan wwt.s//wwww.nn.c.......m com/. Accessed: 2024-03-07.
[9] A. Murali, A. Mousavian, C. Eppner, A. Fishman, and D. Fox. CabiNet: Scaling neural collision detection for object rearrangement with procedural scene generation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2023. URL https: / //arxiv...iii////a org/ abs /20// 09302..
[10] A. Raistrick, M. Lingjie, K. Karhan, David, Z. Yiming, Beining, W. Hongyu, Meenal, S. Alexandropoulos, L. Lahav, M. Zeyu, and D. Jia. Infinite indoors: Photorealistic indoor scenes using procedural generation. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[11] A. Goyal, Mousavian, C. Paxton, Y.W. Chao, B. Okom, Deng, and D. Fox. Ifor: Iter- ative flow minimization for robotic object rearrangement. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 14787-14797, 2022.
[12] M. Görner, R. Haschke. H. Ritter, and Zhang. Moveit! task constructor for task-level motion planning, In 2019 International Conference Robotics and Automation (ICRA), pages 190- 196, 2019, doi: 10. 10991/......199873338888

---

[13] B. Sundaralingam,, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8112-8119. IEEE, 2023.
[14] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[15] A. Szot, A. Clegg, E. Undersander, Wijmans, Y. Zhao, Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266 2021.
[16] S. Srivastava, C. Li, M. Lingelbach, R. Martin-Martin, F. Xia, K. E. Vainio, Z. Lian, C. Gok- men, S. Buch, K. Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. Conference on robot learning, pages 477-490.
PMLR, 2022.
[17] J. Gu, F. Xiang, X. Li, 7. Ling. X. Liu, T. Mu, Y. Tang. S. Tao, X. Wei, Y. Yao, et. Maniskil12: A unified benchmark for generalizable manipulation skills, arXiv preprint arXiv:2302.0466 2023.
[18] Y. Zhu, Wong, A. Mandlekar, R. Martin-Martin Joshi, S. Nasiriany, and Y. Zhu. ro- bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprini arXiv:2009. 12293, 2020.
[19] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2 03298, 2021.
[20] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[21] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.
[22] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 3026,, 2020.
[23] E. Rohmer, S. Singh, and M. Freese. Coppeliasim (formerly v-rep): versatile and scalable robot simulation framework. In Proc. of The International Conference Intelligent Robots and Systems (IROS), 2013.
[24] T. Yu, D. Quillen, 7. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[25] C. F. R. Martín-Martin M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks, arXiv preprint arXiv:2108.0322 2021.
[26] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, Liu, Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. Guibas, and H. Su. SAPIEN: A simulated past-based interactive environ- ment. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

[27] M. Heo, Y. Lee, D. Lee, and J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. Robotics: Science and Systems, 2023.
[28] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[29] C. Sferrazza, D.-. Huang, X. Lin, Y. Lee, and P. Abbeel. Humanoidbench: Simulated hu- manoid benchmark for whole-body locomotion and manipulation, 2024.
[30] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, Liu, V. Koltun, J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339-9347, 2019.
[31] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mot- taghi. Manipulathor: A framework for visual object manipulation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4497-4506, 2021.
[32] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 2017.
arXiv:
[33] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, C. Eppner, J. Leitner, J. Bohg, A. Morales, T. Asfour, D. Kragic, et al. Deep learning approaches to grasp synthesis: A review. IEEE Transactions on Robotics, 2023.
[34] A. D. Vuong, M. N. Vu, H. Le, B. Huang, B. Huynh, T. Vo, A. Kugi, and A. Nguyen. Grasp-anything: Large-scale grasp dataset from foundation models. arXiv preprint 0rXiv:2309.098/8 2023.
[35] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2901-2910, 2019.
[36] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
[37] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[38] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-.. Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
[39] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640- 658. Springer, 2022.
[40] A. Raistrick, L. Lipson, Z. Ma, L. Mei, M. Wang, Y. Zuo, K. Kayan, H. Wen, B. Han, Y. Wang, al. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12630-12641, 2023.
[41] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: A large-scale grasp dataset based simulation. In Under Review al ICRA 2021, 2020.
[42] I. A. Şucan, M. Moll, and L. Kavraki. The Open Motion Planning Library. IEEE Robotics & Automation Magazine, 19(4):72-2,, December 2012.
org.
11

---

[43] J. J. Kuffner and S. M. Valale. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37063), volume 2, pages 995-1001. IEEE, 2000.
[44] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1714-1721. IEEE, 2017.
[45] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:23305. /6309, 2023.
[46] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space, Advances neural information processing systems, 30, 2017.
[47] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922-928. IEEE, 2015.
[48] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
[49] Z. Zhao, V. Kumar, Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2023.
[50] D. Prattichizzo and C. Trinkle. Grasping. Springer handbook of robotics, pages 955-988, 2016.
[51] D. Morrison, P. Corke, and J. Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, 5(3): 4368 4375,335,222 2020.
[52] S. La Valle. Rapidly-exploring random trees: A new tool for path planning. Research Report 9811, 1998.
[53] M. Danielezuk, Mousavian, C. Eppner, D. Fox. Object rearrangement using learned im- plicit collision functions. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 6010-6017. IEEE, 2021.
[54] C. Chi, Feng. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04133, 2023.
[55] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310 10.00664, 2023.
[56] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myy ers, J. Kim, M. Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning. pages 1723-1736 PMLR, 2023.
[57] A. Khazatsky, K. Pertsch, S. Nair. A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv: 2024.
12

---

A Benchmark Details
A.1 Task Setup
Robots and Cameras. The robot is placed at the front of the scene Figure 9, with axis facing forward. The base position is determined by the positions of the support surfaces and the scene type category. For example, for DoubleDoorCabineel type, we place the robot [0.2, 0.5Jm beneath the lower deck, to mimic cases when the robot needs to fetch items from the cupboard in the kitchen. The camera positions are also sampled hased on the supports. To be specific, we use simple heuristic set position and look-at position [28], to ensure that the scene objects (shelves, tables) and all surfaces are visible in the field of view. This prevents cases where the robot collide into objects that completely out of the view. However, should emphasize that occlusion, c.g., shelf boards, baskets, is still common challenge and primary cause for failure and collision.
Initialization. The robot is initialized with default joint state. All objects are placed at the pose specified by the configuration file, which is stored after the filtering in Section 4.1.
Evaluation. After the algorithm terminates, we wait for extra 10s for the scene to be stabilized. Then, the grasping is considered successful if the following criterion are satisfied:
The center of mass of the target object has z-value > 0.3m in the robot-base frame.
2. The center of mass of the target object has x-value < 0.0m in the robot-base frame. (x+ points the front of the robot)
3. The center of mass of all other objects in the scene has a movement < 0.1mm from its initializa- tion.
Here, these thresholds are carefully tuned to prevent outlier scenarios.
A.2 Procedural Scenes
We create 13 types of procedural scenes. Figure 8 shows some examples of the scenes. For the EketShelf, randomize the size of the cells and its placement the wall. For the LargeShelf, randomize the height, width, depth of the shelf, the height of each cell, the basket geometry and size, and the placement of the baskets on each layer of the shelf. For the TriangleShelf, we randomize the board, leg thickness and placement, the board contour geometry, the gap between boards and the
IKEA Products
Procedural Scenes
Figure 8: Examples of procedural scenes and their IKEA counterparts. The procedural scenes rendered with Infinigen [10]. The first row is the EketShelf type, the second row is the LargeShelf type, and the last row is the Thien type.
13

---

Table 4: Statistics of different types of procedural scenes of the testing tasks.
Scenes
Tasks
# On-Table
# On-Shelf'
In-Drawer
# in-Basket
CellShelfiesk
10
600
265
235
0
100
Desk
6
360
360
0
0
0
DeskWall Wall
7
420
255
165
0
0
DoubleDoorCabinet
6
360
0
353
0
Drawer
6
0
360
DrawerSholf
10
600
0
48
499
53
EketShelf
5
300
0
300
0
0
LargeShelf
10
600
0
481
0
119
LargeShelfDesk
9
540
208
266
0
66
LayerShelf
10
600
0
362
0
238
RoundTable
9
540
237
0
0
303
SingkdDoorCabbnetDessk
420
201
214
0
TriangleSheltDesk
5
300
0
300
0
Total
100
6000
1526
2724
891
859
size and height of the table-top and the position and rotation of 2 shelves. In Table 4, we show all types of the scenes and statistics in the 6k testing tasks.
A.3 Task Examples
In Figure 9, we show the examples of our evaluation task from the view of one of the camera, with the Franka robot in place.
B Baseline Implementations
B.1 Sense-Plan-Aet
For all sense-plan-act methods, i.e, CGN-CuRobo, CGN-RRTTConnect CGN-Cabinet, the CGN takes the point-cloud and the target-object segmentation mask as input, and outputs the candidate grasp poses. The pre-grasp pose is 4cm retracted along the approach direction [33] and the post- grasp pose is 2cm lifted from the grasp pose. These offset values are carefully tuned on held- out validation tasks. For CuRobo [13] and RRTConnect [43] algorithms, the scene point-cloud is converted to mesh with marching cube algorithm with voxel size of 5mm. For Cabinet [9], input points to the collision checker and waypoint predictor are first moved to the origin based on the position of the target object. For CGN-CuRobo, partially observed object in the retrieval phase (when it is attached to the end-effector). For CGN- RRTConnect, we use PyBullet and Trimesh for mesh-based collision checking.
sample-surface approximation for the
B.2 Imitation Learning
For our imitation learning models, we use PointNet++ [46] and Voxel-CNN [47] as the encoder on segmented point cloud the target object, the scene and the robot. Furthermore, we encode the proprioceptive states with MI.P encoder. We concatenate all embeddings as the final embedding feed into the policy network. For the MLP variant, we use the embedding of current step and 3-layer MIP to output the delta movement. The output is parameterized distribution. For the Transformer variant, into an Optimus policy head [45] and output a Tanh Gaussian mixture distribution.
Tanh Gaussian the consecutive last steps' embeddings to feed
Additionally, as expert trajectories are collected two stages, i.e., approaching and retrieval, we augment the model input with one additional one-hot vector to indicate the stage during training- Furthermore, we require the model output one additional signal of whether the stage should come to end (termination output). Namely, for all the training data, the very last step of each trajectory stage will have value otherwise -1
14

---

Figure 9: Examples of grasping tasks from one of the camera view when the robot is place. The scenes are rendered with Isaac-Sim [8].
Though the model is trained for different stages, for E2Elmit variants, we use the model an end- to-end mode with re-grasping behavior. To be specific, if the model is in approaching stage and it outputs the termination signal, then it transits to the retrieval stage and close the gripper. However, if the gripper is fully closed during the retrieval stage (we know the grasp is failure), then open the gripper and transit back to the approaching stage. For the CGN-CuRobo-Imin methods, we only use in the retrieval stage.
All models are trained on our demonstration dataset with Adam [48] optimizer. We select the best aeper-parametert by validating held-out validation tasks. In Appendix C, we show more results on our imitation learning models.
15

---

C Additional Benchmark Experiment
We provide additional experimental results on FetchBench.
C.1 Other Manipulators
Our benchmark uses Franka Research robot. However, we should note that the benchmark envi- ronments be easily extended other robots.
Different robot hardware will affect the overall performance of the fetching task. In Table 5, we evaluate two different hardware setups: Kinova Gen3 with Franka Hand and Kinova Gen with Robotiq Hand. For simplicity, we use the ablation experiments of CGN-CuRobo.
We found that altering the morphology alone only slightly impacts performance due to vari- ations in motion planning problems. However, changing the gripper significantly affects perfor- mance. This is because grasp annotations were evaluated using the Franka Hand [41], while the Robotiq gripper has a different morphology and different physical process when closing the gripper.
Table 5: Results of success rate in ablation experiments different robot hardwares.
Franka Arm Franka Hand
Kinova Gen3 Franka Hand
Kinova Gen3 Robotiq Hand
GA-Mosh-Curobo
0.671
0.629
0.471 0.108
GA-Ptd-Cumobo
0.190
0.173
CGN-Curobo 0.094
0.088
0,017
C.2 Imitation Learning Baselines
We explore to improve the performance of our imitation learning baselines. First, we show that the end-to-end deployment with re-grasping behavior does improve the performance by roughly 3% in terms of success rate, over the vanilla two-stage deployment (i.e., only seproach-then-rereee However, we also found that the re-grasping behavior also increases the computation time and c- space trajectory length significantly. The results of E2Elmit variants and TwoStagelmit variants are shown in Table 6.
Table 6: Results of imitation learning baselines with different inference mode.
Success Computation Time
C-Space Trajectory
E2EImit-MLP 0.082
58.59
38.42
TwoStagelmit-MLL
0.053
16.54
20.88
E2Elmit- Transformer
0.099
57.7
45.1
Sattaagellmi--TTasfoommne 0.065
16.67
10.30
In addition, we also evaluate the performance with twice amount of training data and with ACT [49] algorithm. The results shown in Table 7. Training with twice amount data and on more advanced algorithms do improve the performance. We leave it future work to improve the end- to-end imitation learning models on challenging fetching tasks.
Table 7: Results of imitation learning methods under different models and with more training data.
Success (Ix Data)
Success (2x Data)
TwoStagelmit-MLP 0.058
0.061
E2EImit-MLP
0.082
0.093
E2EImit-ACT
0.093
16

---

D Real Robot Experiment
D.1 Experiment Setup
We demonstrate that robot fetching from complex environment challenging problem in prac- tice. Here, implement the CGN-RRtConnect baseline and test on diverse set of objects and environments in order to compare performance between different scenes.
Figure 10: Hardware setup of our real world experiment. The Franka Emika Research 3 is placed in front of the scene and we use Realsense L515 pointing towards the scene capture RGB-D information.
Hardware and Evaluation Setup. For our hardware, we used the 7-DOF Franka Emika Research our robot arm, and Intel Realsense L515 LiDAR camera capture the RGB-D images of the scene [1]. In order to evaluate on diverse set of environments, tabletop, two distinct shelves and various baskets. Figure 10 shows our hardware setup and the example scenes are shown in Figure 7.
performed experiments on
We tested on a diverse set of objects our experiment. Figure 11 shows examples of the objects used our experiments.
Algorithm. With our real-world robot, the CGN-RE CGN-RRTConned baseline is implemented as follows.
1. Capture the RGB-D image of the scene and get the target object segmentation mask by prompt- ing Seg-Any.
Use the point-cloud and the mask to query CGN for candidate grasp poses on the target object. Use Movelt! [12] to search for motion to the pre-grasp poses based on the confidence scores. Move from pre-grasp pose grasp pose with linear motion from pilz's 's LIN planner [12]. Crop out the object from the point-cloud, add placeholder to the robot, and plan the motion the initial pose.
D.2 Experiment Results
Table 8 shows the detailed results of real-world experiment. Comparing between table-top and shelf cases, we see the % success significantly decreases in shelf scenes due to the increased difficulty of the environment. In addition, we also show that grabbing objects from baskets within each respective scene also results in lower % success due to physical constraints.
Failure Analysis. To further understand the behavior of the CGN-RRTCommnn baseline, broke down each unsuccessful attempt into four categories of failure:
1. No Grasp Poses (NGP), where CGN failed to find any grasps.
2. Motion Planning Failure (MPF), where CGN returned grasps, but RRT-Connect found valid trajectories.
17

---

OTEEZ
Figure 11: Examples of objects used in real world experiments.
# Success
Attempts
% Success
TableToop
52
38.5 %
YableTTop-Baske
16
18.8 %
Shelf
11
83
13.3 %
Shelf-Basket
16
6.25%
Total
35
167
21.0
Table 8: Results of real-world fetching experiment by environment types.
Invalid Grasp Pose (INV), where the robot attempts grab the object, but the grasp is unstable.
4. Collision Failure (CF), where the robot target object collidess/disruts the environment.
% Success
% NGP
% MPF
% INV
% CF
TableTopp
38.5
13.5
36.5
11.5
Table Toplekop-Baske
18.8
31.3
18.8
25.0
6.25
Shelf
13.3
15.7
31.3
12.1
27.7
Shelf-
6.25
37.5
43.8
6.3
6.3
Total
21.0
18.6
21.6
20.4
18.6
Table 9: Results of different failure types in each type of environment.
Table shows the failure types of all attempts each environment category. Comparing table top scenes to shelf scenes, shelves have a much higher failure rate due higher number of motion planning and collisions. This is attributed the difficulty of motion planning within shelf scene. In addition, basket scenes had higher rate of NGP failure compared to their normal counterparts, as the baskets produce occlusions that limit the pointcloud input into CGN. We also found that many of the perspectives in shelf and basket scenes caused CGN to output very few + low confidence grasp propositions, leading us to believe that CGN sensitive to perspective and produces better results on table top scenes.
Linitations. Due limited resources, only evaluate smaller number of fetching cases, comparing to the simulation benchmark.
18

---

E Additional Related Works
Predicting grasp poses for various novel objects has been an important long-standing research challenge in robotics [50]. It becomes more challenging to predict accurate and diverse grasp poses from noisy and partially observed sensory readings like depth maps and par- grasp pose synthesis has become a crucial solution paradigm to this problem [33], owing the power of neural networks to handle nigh-dimensional inputs and flexibility to various objects with different geometries and physical properties. To train the neural network, methods create and utilize large-scale object grasp pose datasets [41, 215. The valid grasp poses of each object are labeled with analytic metrics [3. 5, 2]. or with physics simulation [35, 1]. Notably, researchers have applied the grasp pose prediction models to build robust grasping systems that can clear various unknown objects from cluttered bins [5] and table-tops [1].
Grasp Pose Prediction.
tial point clouds [1, 5]. Recently, learning-baat
Grasp pose prediction models play crucial role in the baselines tested. We use the Contact- GraspNet [1] to predict 6D grasp poses from partial point clouds to command the motion generation module. However, as we will show in Section 6.2, the model is not powerful enough to solve the benchmark. Furthermore, having an accurate grasp pose is not the only bottleneck challenge to grasping objects from more challenging cluttered environments, e.g., shelves, cabinets, and drawers, that are erucial for applications like service robots [7].
Motion Generation for Robot Arm. To generate collision-free trajectories is one of the fundamen- tal problems for robot arm control. Given the obstacles of the environment, sampling -easpidg---adddddd motion planning algorithms like RRT [52] and its variants [43] are the common choice to command the arm to the target pose [12]. Meanwhile, optimization- based motion generation methods [13] have been proposed as an alternative to classical approach. CuRobo [13] has much higher efficiency than the sampling-basee -bsed motion planning algorithms, can be computed parallel on GPUs. How- ever, these methods assume known environment and obstacles. They do not account for the partial observation problem, which is common challenge for in-the-wild applications like home robots. To overcome this issue, [9, 53] propose to learn a neural collision checker for partially observed scenes and movable objects. After training on large-scale synthetic data, these models have shown promising results in tackling the partial observation problem in robot object rearrangement tasks.
However, in Section 6.2, we find that the partial observation problem still remains a challenge to existing motion generation methods in grasping. This suggests huge space for improvement in the motion generation methods to tackle real-world challenges.
Imitation Learning. Imitation learning [49, 54, 45] has become a promising approach to learning large-scale behavior models [36, 37] for robots. However, despite great effort from the community, researchers still lack enough data to train powerful large behavior models. Moreover, majority of the expert data [36, 55, 56, 57] collected on table-top environments and lack diversity in scene variation. With our procedural scenes and tasks, our simulation benchmark generate large quantity of diverse grasping demonstrations in various environments. Thus, our benchmark also serves as a platform for imitation learning research of large behavior models grasping. We provide a procedural demonstration synthetic data generator for diverse and abundant expert demos. Furthermore, Section 6.1, we find that combining imitation learning and the common grasping pipeline achieves SOTA performance on our benchmark, which suggests promising direction for new grasping systems.
19


Reviews:
[{"review_id": "5bXOGFUSBW", "reviewer": "Reviewer_XzpA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new simulated benchmark for robotic fetching tasks in cluttered environments, which requires a Franka robot arm to grasp a desired object and fetch it in diverse scenes, including a table, shelf, drawer, and basket. Despite the abundant prior simulated robotic manipulation benchmarks, this benchmark poses its unique challenges and anlyses: (1) procedurally generated 6K testing task instances to evaluate a policy's **generalization capability** to diverse scenes and objects, (2) dissecting a \"sense-plan-act\" pipeline to find out its major roadblocks, which are mainly due to *partial observability* and *grasp pose estimation*. This simulated benchmark can serve as a testbed for generalizable robotic manipulation.", "summary_of_recommendation": "This benchmark provides an important testbed for \"generalization\", which has been barely covered in the robotic manipulation community. Thus, I believe this benchmark will be a good contribution to further robot learning research. I suggest \"weak acceptance\" of this paper due to relatively weak imitation learning baselines, which may be able to show reasonably good performance in this benchmark.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "* The end-to-end imitation learning baselines seem too weak (less than 2% success rates). The authors could consider some latest approaches, such as Diffusion Policy, ACT, and OpenVLA. Moreover, the ablation studies are only conducted for planning-based approaches; but, the authors could consider including ablation studies for imitation learning approaches to provide more insights about the benchmark tasks for end-to-end learning. \n* In Table 2, \"Traj Length (Rad)\" is a bit confusing to me. It would be clear to use \"Traj Length in C-Space\" instead. Moreover, it would be interesting to see the \"Traj Length\" in terms of execution time or number of low-level control actions, which might be something around 600 (3M / 5K).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IEKmJrJIJm", "reviewer": "Reviewer_qTGA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present FetchBench, a simulation benchmark focused on diverse pick and place manipulation tasks. The benchmark consists of a set of procedurally generated scenes with 5.5k+ objects, a dataset of 27.5k successful grasp trajectories, and a study comparing analytical, learned, and hybrid grasping models on these tasks. The study shows that all of these approaches have significant room for improvement. The best approach is a hybrid approach which uses analytical grasping methods to reach and grasp objects, and imitation learning methods to manipulate the object post-grasping. A series of more fine-grained analyses identify specific bottleneck parts where each method struggles, which is insightful.", "summary_of_recommendation": "It is insightful to see that, contrary to a commonly held belief that pick and place tasks are “solved”, existing state-of-the-art approaches fail to solve these tasks effectively. I believe that this benchmark will be helpful to the community to help develop more effective approaches.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see the points for “areas for improvement” in the previous section. Furthermore, please add a discussion on RoboCasa [1] on the discussion on related work, which also includes pick and place tasks and imitation learning datasets for these tasks.\n\n[1] Nasiriany et al., RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. RSS 2024", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3zYDTJSfz", "reviewer": "Reviewer_vPSQ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new benchmark FetchBench, with diverse procedural scenes that integrate both grasping and motion planning challenges.", "summary_of_recommendation": "More details are needed and the benchmark has the potential to get scaled and diversified.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- See the reviews section.\n- How the material or other physical properties are distributed in the benchmark.\n- Can the robotic arm adjust its pose before the fetch, or it is the fixed base?\n- Would different robotic hardware affect?\n- When spawning the robotic arm, how to consider the action space of the robotic arm?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a new simulation benchmark for fetching. The authors added new experiments during the rebuttal, with results now using ACT and Diffusion Policy. The authors also did new experiments with different data sizes for imitation learning, as well as changing the robot type and gripper. There was some internal discussion about this benchmark among reviewers during the rebuttal. While a unanimous accept did not emerge from this discussion (especially due to potentially unrealistic assumptions like perfectly segmented point clouds), the consensus was that there is merit to the benchmark. It appears a good fit for the robot learning community, though there is overlap with some other benchmarks such as RoboCasa (https://robocasa.ai/). 

The authors should add their new rebuttal experiments to the finalized paper.

Strengths:
- The paper studies a novel application area for a benchmark. 
- While it is in simulation, which makes it easier to compare different algorithms from different research teams, it shows applicability in the real world.
- The design of the benchmark is sound overall, with procedural generation involved to test generalization. Also, it tests occlusions.

Weaknesses:
- The imitation learning results could probably be better than they are reported in the paper. For example, how about increasing the size of the demonstration data?
- The paper could also consider adding more imitation learning (and reinforcement learning) methods.
- It would be nice to have more focus on diverse robots and grippers.
Decision: Accept