GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
Authors: Pu Hua
Minghuan Liu
Annabella Macaluso
Yunfeng Lin
Weinan Zhang
Huazhe Xu
Lirui Wang
Venue: CoRL 2024 (2024)
Keywords: Generative Simulation; Robotics; Learning
TL;DR: We propose GenSim2, a scalable framework that uses foundation models for diverse, complex, realistic simulation tasks and data generation.

Abstract: Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.

GenSim2:
Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
Pu Hua" "u Minghuan 2.3+ Annabella Macaluso Yunfeng Lin
Weinan Zhang Huazhe Xu Lirui Wang
Institute for Interdisciplinary Information Sciences, Tsinghua University!,
UCSD2, Shanghai Jiao Tong University*, MIT CSAIL
https:// / . github. 1o/
Robotic simulation today remains challenging to scale up due to the human efforts required create diverse simulation tasks and scenes. Simulation- trained policies also face scalability issues many sim-to-real methods focus a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding L.MS with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long- horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RI. solvers that generalize within object categories. The pipeline can generate data for up 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, propose effective multi-task language-conditing conditionnd policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the gener- ated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively limited real data.
Abstract:
Keywords: Coding Large Language Models, Vision Language Models, Robotic Simulation, Multi-task Sim-to-Real Transfer, Articulated Object Manipulation
1 Introduction
Robot learning requires large amounts of interaction data and evaluation, which are expensive to acquire at scale in the real world. Robot simulation holds the promise of providing such data and verification in high diversity and efficiency across objects, tasks, and scenes. While the ability to simulate has led to many successes in Al across Gaming, Go, and Mathematical Proofs [2, 3, 4], there are two requirements for such path to be successful robotics: The data needs to scale in complexity without significant human efforts and the data needs be realistic enough transfer the real world. Previous works [5, 6, 7, 8, 9, 10, 11] have made significant progress in scalable simulation benchmarks in robotics and training policies the simulation data.
Foundation models [12), particularly generative models pre-trained on internet-scale data [13, 14, 15], have demonstrated impressive capabilities required for generating robot simulation tasks, such as coding [16], spatial reasoning [17], task semantics
[9], planning [18, 19], video prediction[20 21], and cost and reward understanding [22, 23). While foundation models have shown impressive capa- bilities output actions to solve robotic tasks directly in the real world [24], simulation provides low-cost and scalable platform to learn robust end-to-end policies. In addition to generating numer-
equal contribution. $ project lead. 8th Conference Robot Learning (CoRL. 2024), Munich, Germany

---

(A) Large-scale Task and Data Generation
(B) Multi-Task Training in Simulation
Scene:
Randoo Pose
Object: Random
Pointsloud Rändom Cropping
(C) Sim-to-Real Transfer
Figure I. GenSim2 introduces scalable task and data generation framework in SAPIEN [1] for articulation objects with multi-modal and reasoning L.LMs. The framework comprises three main stages: 1) (Top) We first generate large-scale robotics tasks and collect massive data with LLMs; 2) (Bottom-left) Then, we train a multi-task point dloud-based policy in simulation with imitation learning; 3) (Bottom-right) Finally, we zero-shot transfer the policy to the real world.
ous tasks, automatically solving them to produce demonstration data and transferring the learned skills to the real world present significant challenges. To address this, the data generation and policy learning pipelines must scale efficiently while minimizing the sim-to-real gaps.
In this work, we propose GenSim2 (Fig. 1), scalable framework that amplifies robotics data by generating diverse articulated and long-horizon simulation tasks and demonstrations. In addition to the semantic knowledge and compositional capabilities in language models, these tasks also re- quire extended 6-dof motions with contacts. To generate demonstrations for such tasks, we develop various solvers, including a keypoint-basee motion planner that generalizes across object categories and initial conditions. Moreover, we use the visual information a multi-modal LLM (MLLM) to iteratively generate and verify novel tasks, to the extent of 100 tasks to train multi-task policy.
GenSima tackles more challenging articulated and long- nng-horizon tasks beyond top-down pick-and- place [9]. These complex tasks require grounding and reasoning capabilities beyond previous LLM techniques. Therefore, GenS m2 is an agent pipeline featuring multi-modal foundation models (e.g. GPT-4V) and reasoning models (e.g. OpenAl ol) as well as generalized keypoint planners for such complex tasks. To fully utilize the generated data, we further propose sim-to-real native policy architecture, dubbed proprioceptive point-cloud transformer (PPT), with point-cloud observations and language conditioning. This effective architecture is designed for sim-to-real transfer with the large-scale data generated in simulation.
In our experiments, we have generated over 100 articulated object manipulation tasks with 35 objects (over 200 instances), including long-horizon tasks. Our task generation pipeline achieves 25% better success rates than previous works. After using an automated pipeline solve these tasks and generate task demonstrations, configurations and only suffer a 3% performance drop when tested unseen object instances. More importantly, by integrating the GenS im2 pipeline with the designed policy architecture, we achieve promising results across 8 real-world articulated object tasks, through such as zero-shot transfer and co-training with real-world data. This approach leads to a 20% improvement in policy performance compared to training exclusively on limited teleoperation data.
multi-task policy can jointly solve 24 tasks with random scene
In summary, show promising way reduce data collection efforts and solve real-world prob- lems through massive simulated data generation with MLLMs. Our contributions are as follows: (1)

---

We design robotic simulation task generation pipeline that explores the usage of internet-scale vi- sual and language knowledge embedded in coding multi-modal L.MMs (e.g. GPT-4V), to generate up to 100 articulated object manipulation tasks. (2) We propose a simulated demonstration generation pipeline that can generate high-quality data with minimal labeling efforts. The solution integrates smooth motions from model-based planners that have spatial and object-level generalization. (3) We develop novel policy architecture for seamless sim-to-real transfer that distills multi-task demon- stration efficiently and with strong sim-to-real performance.
2 Related Work
Task Generation in Robotic Simulation. Training and evaluation for roboties have greatly bene- fited from developments in robotic simulation such Mujoco, NVIDIA Isaac, Drake, and Sapien [25, 26, 27, 1]. With these tools, researchers and engineers have been able to develop benchmarks with tens to hundreds of unique, hand-designed robotic tasks [28, 29, 30, 31, 32, 33, 28]. However, the requirements of high-quality assets, scenes, and task design along with required verifications for task solvability and real-world transfer demand significant amount of human skill and effort, Recent works have explored methods such as domain randomizations [34, 7, 35, 36], procedural asset generation [8, 37] and text-to-3D synthesis [38, 39, 40, 41] to mitigate some of these efforts. Researchers also investigated how generative models can generate suites of robotic agent tasks [23, 42, 43, 44, 45, 9, 10, 11] generate interactive simulations directly from videos [20, 21J. In particular, GenSim [9] develops novel pipeline that generates over 100 simulation tasks utilizing LLMs, verifies these tasks, uses them to generate data, and trains multi-task policies on top of this data. Similarly, RoboGen [10] uses RL to solve complicated locomotion tasks and deformable ma- nipulation and RoboCasa [11] generates massive high- high-fidelity tasks for multiple embodiments in mobile manipulation tasks. In our work, we leverage multi-modal LLMs to generate 6-DOF robotic manipulation and long-horizon tasks at scale. Furthermore, we focus complex yet realistic tasks that can solved with different solvers and transferred to the real world.
Multi-Task Policy Learning with 3D Information. With the surge of robotic data, recent works have explored multi-task policy learning, usually conditioned on language inputs. In particular, policy learning often benefits from access explicit 3D information when doing generalized 6- DOF tasks [46, 47]. [48] uses RGB-D inputs and output pixel-level affordance map. [46] uses voxel as policy input information and output keyframe actions. [47, 49] uses point cloud as inputs the policies. [50] uses keypoints at the corners to manipulate boxes with dexterous hands. In our work, we focus on point-cloud transformers as multi-task policies for articulated object manipulation.
Rim-to-Real Transfer. While simulation provides scalable training data and evaluation runs, sim- to-real transfer [51, is a challenge and an active research area. RL-based controllers [53] leverage simulation environments for large-scale interactions learn robust behaviors. Previous works have also explored techniques such domain randomizations [54, 34], domain adaptations [55], and policy composition [56] to improve sim-to-real transfer. Our work can be viewed as a distillation process from the knowledge in foundation models such as MLLMs(14, 13], into policies, which can have a smaller sim-to-real "semantic" gap compared to rule-based simulation data.
Generating Tasks and Training Policies at Scale
The proposed GenSim2 framework executes a series of processes including task proposal (3. demonstration generation (3.2), and policy learning and transfer (3.3). We illustrate the pipeline in Fig. 2. Our work enhances GenSim [9] at the level of task complexity beyond top-down pick-and- place, which necessitates grounded multi-modal task designs, generic 6-dof motion planners and RI. solvers, and scalable policy architectures.
3.1 Task Proposal
Primitive Task. Our pipeline begins with proposing primitive tasks, namely tasks with single simple motion (e.g. opening a box), by prompting LLM to generate novel task, defined by its
3

---

Task Proposal
Data Generation
Multi-Task Training
Evaluation Transfor
UMA Motion
Demonsaat Collection
Spulia Object
MLLKK
Controlens 1pl, pAL
LLH
Simulation
desciptiicn
MILLM
"put a ball
In box"
"opan
Prescturtion
LLM
motione
molioro
Poliy Tranklg
Savask
RL Policy Leuereer
MLLM
Trak Decomptieiinn
Reword
Tranning
PPO
Figure 2. Overview of GenSim2 framework. The pipeline consists of (I) task proposal, (2) solver creation, (3) multi-task training, and (4) generalization evaluation and sim-to-real transfer.
task description (a short phrase or sentence), used assets and task code. We start from fixed asset library and small task library initialized with hand-designed example tasks. We parse them into the prompts for in-context learning. We query the LLM to discover applicable assets for creating new task. The LLM then takes in as input the generated task definition and outputs the corresponding task code. This code is finally compiled and ready for demonstration generation in Section 3.2.
Long-horizon Task. Moreover, we extend our pipeline to generate long-horizon tasks that consist of multiple steps of primitive tasks and involve manipulating articulated and rigid-body objects. Ex- amples include opening box, placing ball inside, and then closing the box. When generating a long-horizon task, we execute task decomposition process between task proposal and code im- plementation, to decompose the long-horizon task into several sub-tasks (primitive tasks). We use two distinct methods for generating long-horizon tasks: (1) Top-down: We directly generate a task in a curriculum, then decompose the task into several sub-tasks, each with a dedicated solver. (2) Bottom-up: We first generate primitive tasks and build up a task library. Then the LLM will be prompted to select tasks from the pre-built task library to compose new task. We have observed reasoning LLM to improve task proposal performance in this stage. After we have finished the task planning, we continue to generate solvers and demonstrations for the subtasks in succession. Fig. 9 demonstrates visualizations of some generated long- horiznn tasks.
3.2 Demonstration Generation
In this section, our goal is to create solver to generate demonstrations given a task (or sub-task) code. The objective of our generation pipeline is to collect large-scale, high- data for gen- eral 6-DOF manipulation task learning in the real-world setting, our task solver should meet the following requirements: 1) Universal for 6-DOF tasks without task-specific designs; 2) Robust to different scene configurations; 3) Fast to execute; 4) Deterministic with high success rates. While such solver is easy design for top-down pick-and-place, in which we only need specify some 2D position on the table waypoints and execute primitive "pick" and "place" actions, becomes challenging for more complex tasks such as general articulated object manipulation.
To address this challenge, we propose and investigate the planner config generation with a multi- modal L.M [14]. We also propose two types of task solvers: a kPAM-based motion planner and a complementary RL-based policy learner (see in Appendix A.3), to solve the generated tasks.
Motion Planner. kPAM[57] is keypoint-based planner proposed for category-level manipulation tasks. It defines a robotic task by an actuation pose, namely the homogencous transformation re- quired to manipulate the target object, and it addresses an optimization problem based on several keypoint-bor based constraints to get this pose. Following Wang et al. [33], we improve and extend kPAM to produce an object-centric trajectory of end-effector poses, termed actuation motions, to solve a generated task. We parse the constraints and actuation motions into parameterized config- urations that easily interpretable and codable. We provide illustration in Fig. 7 to show how the kPAM planner solves task.

---

cleess
MLLM
Solver Creation
Task Proposal
1: Constraints
Solver Creation 2: Actuation Motions
Actustion Mations
generate the solver config
Trajectory
Dheale 4.4
Postactarer::
Actustion Posa
Pejectio Semplig
Figure 3. Multi-modal Task Generation Pipeline. GenSim2 first is prompted to generate the task code, given few-shot examples and available assets. It then renders the scene image and the keypoint information of the task assets fed into the MLLM model to generate a planner config for the actuation pose. The actuation pose is then extended to actuation motions, which are fed into GPT-4V for inspections. In this example, the pipeline produces a task motion for opening the box.
Multi-modal Task Solver Generation. Compared to previous works that primarily utilize language prompts, our approach incorporates visual information and key points [58, 59, 60], serving as an explicit representation scene details (e.g.. object structures, affordance and spatial information). The pipeline of solver generation is depicted Fig. First, we execute the generated code and capture scene images. This visual data, together with object key points, are then used to prompt an MLLM generate the planner constraints. Subsequently, we visualize the actuation pose defined by these constraints. Finally, query the MLLM to generate the actuation motions. Optionally, we also introduce reject sampling to guide the MLLM in refining its previous outputs.
3.3 Policy Learning and Tranfer
Multi-Task Training. We design multi-task policy structure, denoted Proprioception Point cloud Transformer (PPT), illustrated in Fig. 4. Specifically. we handle three kinds of observations: point cloud, proprioceptive states, and language task description, all of which can be obtained from the real world. Each observation is separately tokenized via respective encoders and cross-attention, fused together in the shared latent space through transformer blocks, and post-processed into global condition tokens [61]. Given the global condition tokens, the policy predicts a sequence of actions through the policy head. In our implementation, we test against various policy heads such as MLPs, the transformer decoder [62], and the diffusson model [63]. See Appendix B.2 for more details.
Sim-to-Rest Transfer.
To transfer between simulation and real environments, we ignore the color information of the point cloud, and augment the point cloud observations of simulated demonstra- tions during training. This augmentation involves cropping, adding Gaussian noise as perturbations, and randomly dropping points. In the real-world setting, we process the actual point cloud using uniform sampling, farthest point sampling, and outlier removal. This processed clean point cloud serves input the multi-task policy during inference.
4 Experiment: Task and Data Generation
In this section, we aim to verify the feasibility of the task generation framework and investigate the effectiveness of the task and data generation pipeline.
Evaluation Criteria. We evaluate the task generation procedure mainly on two types of success rates: /) execution rate, the success rate of completing the whole pipeline without any error such as syntax or runtime emror; 2) solution rate, the success rate of solving the generated task.
Ablation Study. In this section, we conduct a careful ablation study on each component of GenSim2. The results, depicted Fig. 13, are summarized as follows:
Types of LLMs: We test multi-modal LLM, reasoning LLM, and vanilla LLM on solution rate in Fig. 5 left. Without visual data as input, the performance of the pipeline deteriorates with signifi-
5

---

Simulation
Real World
Transformer
Action Sequence
Language Tokenizer
Cloud Encoder
MLP
Action Head
Leplop
Langage instruction
Pointcloud
Propriccepiion
The proposed Proprioception Point-cloud Transformer (PPT) policy architecture maps language, point cloud, and proprioception inputs in shared latent space for action prediction. The policy action head supports various architectures from diffusion [63] to transformer decoder [62].
Figure 4.
0.78
Mull-modal LLM
0.8
1-shol
0.8
0.78
GenSim2 wlo CoT
Reasoning LLM
3-shol
0.7
Success Rate
LLM
Success Rate
5-shol
Success Rate
0.6
0.6
0.5
0.8
0.44
0.4
0.36
0.4
0.4
0.18
0.2
0.1
Figure 5. Ablation study on components of our generation pipeline. All results are based on no less than 10 generations. Left) We use various types of LLMs for solver generation and find that multi-modal LLM (GPT-4V) outperforms the others. Middle) We find that the performance of our method will increase, as we increase the maximum iteration for reject sampling. Right) We observe that splitting solver generation into a prompt chain surpasses generating the whole solver config.
cantly lower success rates in task generation under both criteria. The performance drops because the constraints for actuation pose not be correctly generated as it requires details for object structure. Meanwhile, the actuation motions also suffer from large errors because the vanilla LLM has no ac- cess to spatial relationships between objects. This highlights the crucial role visual information in the success of GenSim2. as it provides essential details about the scene such as spatial relationships and the appearance of objects. Moreover, reasoning capability also improves vanilla LLM's score on solver generation. We hypothesize that the reasoning trace of "think" before coding, based on the task and config structure reduce hallucinations in generating kPAM constraints. See Appendix E.2 and E.3 for more details.
Multi-shot Reject Sampling: In Fig. 5 middle, we have visualized the solution rates of our pipeline under different maximum iterations of multi-shot rejection sampling. The results indicate that our pipeline with multi-shot rejection sampling surpasses the version without it, highlighting the impor- tance of self-reflection in L.MMs. Additionally, allowing L.MS more opportunities refine their answers through iterations leads improved performance. We also explore the use of rejection sampling from GPT-4V, which takes solver config visualizations input and determines their effec- tiveness, in our pipeline. Due to the domain gaps of visual perception and understanding capabilities of MLLMs, particularly in processing 3D robotic scenes ((64, 65]), the MILM can generate irrele- vant responses. More powerful models and advanced prompting and finetuning methods are likely to improve these in future work.
We observe that dividing solver generation into prompt chain (first generating constraints and then actuation motions) yields better results (over 30%) than directly outputting the complete solver at once in Fig. 5 right. This approach allows for focused, sequential processing of components, leveraging outputs from earlier stages for subsequent ones.
Chain-of ohtuugh Prompt Design:
Comparison with Baseline. We compare the pipeline of GenSim2 using kPAM motion planner with another sourced framework RoboGen[10], which encompasses a large number of long- horizon articulated tasks. For primitive tasks, we consider the sub-tasks within a long- morgoorizon task

---

Table 1. Comparison with RoboGen task generation success rates. GenSim2-B represents the
bottom-up long-horizon task proposal method, while GenSim2-T and GenSim2-T (ol) represent top-down methods using GPT-4 and OpenAl-ol respectively.
Type
Primitive
Long- Long-horizon
Method
GenSim2
RoboGen
GenSim2-B
GenSim2-T
GenSim2-T (ol)
RohoGen
Execution
0.94
0.94
1.00
0.83
0.87
0.76
Solution
0.78
0.58
0.68
0.54
0.60
0.43
0.8
0.8
0.6
0.88
0.48
0.46
Success Rate
Success Rate
0.56
Success Rate
0.29
0.19
Tasks
propros PPT-Tri PP1-Tet RGii-Tee
Figure 6. Multi-task training results. All results are evaluated with 20 episodes per task. Left)) As the number of training tasks increases, the policy performance decreases first and then increases. Middle) We respectively ablate the input modalities leveraged our policy architecture on tasks. Right) We find that policies exhibit strong object-level generalization. For each architecture, the left and right bars illustrate the success rates during the training and testing phases, respectively.
in RoboGen for comparison. For long-horizon tasks, compare our top-down methods (labeled as GenSim2-T and GenSim2-T (ol), utilizing GPT-4 and OpenAT-ol task decomposers, respec- tively) and bottom-up method (GenSim2-B) with RoboGen's 's long-horizon tasks.
The results, presented in Tab. 1, indicate that our method exceeds the baseline in both primitive and long-horizon task settings. The underlying reason approach's implementation of fine-grained motion planner generation, whereas RoboGen directly generates task-specific reward function and relies on reinforcement learning for task training, which can be inherently more fragile. Addition- ally, the bottom-up approach surpasses the top-down approach in both metries. This advantage stems from the former's reliance on composing existing tasks, where most failures occur during the chain- ing of sub-tasks, whereas the latter necessitates generating each sub-task from scratch. Furthermore, enhanced reasoning capabilities of OpenAl ol contribute to more logical task decompositions compared vanilla GPT-4, resulting in superior performance across both evaluation criteria.
Experiment: Multi-Task Policy Training and Transfer
In this section, show how the generated data by GenSim2 can be used by multi-task imitation learning and sim-to-real transfer.
5.1 Multi-task Training in Simulation
Training. With the tasks and data generated by our pipeline, we train multi-task policy (with 382M parameters) across different numbers of tasks and test its generalization to new scenarios. In Fig. left, we jointly train a different number of LLLMggeneraeed tasks and test on 4 original tasks under the low data regime such as 10 demos per task. Interestingly, adding more tasks will first drop the performance and then increase it by virtue of scaling. Please refer to Appendix B for
details.
Generalization. After training, we test the generalization ability of our policy to unseen object instances, compared with policy with RGB inputs. We split the instances into train/test sets. Eval- uated results are shown in Fig. right, where we find that the success rates of the PPT architectures only drop by less than 3% on unseen instances, whereas the RGB policies reduce by quite bit. These results indicate that by generating data with object-level and spatial-level variance domain randomization, along with the pointcloud poittcloud-based policy architecture design, the trained policy can acquire generalization in both aspects, which lays the foundation for further sim-to-real transfer.

---

Real Experiments: We observe that the multi-tass task policy, trained simulation data, can perform robustly on various unseen real-world objects through sim-to-real transfer.
Figure 7.
Table 2. Evaluation on a set of real-world tasks using one multi-task policy. We report the per-task success rate of 10 episode evaluations with real-world tasks. Sim-only represents policy trained purely with 100 simulation data generated by GenSim2. Real-only represents policy trained with 10 real-world data collected by teleoperation. Combined represents policy co-trained with combined simulation and real-world data.
Training Data
aptop
apoop
OpenSafe
CloseSafe
CloieDrawer
SwingRucket
OpenBox
CloseBox
Average
Real-only
0.5
0.0
0.2
0.4
1.0
0.5
0.2
0.1
0.363
Sin-only
0.7
0.1
0.3
0.8
0.5
0.425
Combined
0.8
0.7
0,3
0.6
1.0
0.8
0.0
0.4
0.575
5.2 Real-World Experiments
To evaluate the quality of the data collected simulation and how CenSim2 helps in real-robot tasks, we test sim-to-real and co-training on the generated data over several tasks. As for the setup, use the Franka Research robot arm with modified, deformable TPU parallel gripper for easier grasping. The robot work cell is equipped with three RealSense D435 cameras: one wrist-mounted and two externally facing the scene. Each captures an RGB-D observation which is combined and processed into point cloud to be used observations. More details are listed in Appendix C.
We perform experiments on 8 real-world tasks, collecting 100 demonstrations for each task sim- ulation, along with an additional 10 real-world demonstrations via teleoperation. We assess the quality and utility of the generated data across three distinct training setups: (1) using only simula- tion data, (2) using only real-world data, and (3) using combination of both. The results of these evaluations are presented in Tab. 2. Our findings can be summarized follows: (a) data generated by GenS i m2 enables effective zero-shot sim-to-real transfer, with the resulting policy outperform- ing one trained solely on limited real-world data; and (b) when co-trained with real-world data, the data generated by GenS im2 significantly enhances the policy performance even by 20% in absolute values and 50% in relative scale. These results underscore the potential of large-scale, high-quality data generation, like GenSim2, reduce the burden of extensive real- world data collection while improving policy effectiveness for real-world tasks.
6 Limitations and Discussions
Our proposed method has several limitations. Due to the lack of "robotic centric" knowledge, such as 3D spatial understanding, foundation models like GPT-4V still face hallucination issues in creating meaningful tasks and successfully coding them. Additionally, human involvement, though minimal, still required to generate these complex manipulation tasks. Finally, we have only considered 6-dof tasks in zero-shot sim-to-real transfer with limited point cloud observations.
7 Conclusion and Future Work
In this work, we propose GenSima task and demonstration generation framework that utilizes multi-modal foundation models to generate up 100 robotic simulation tasks at scale, such long- horizon manipulation with articulated objects. We ablate on different task solvers and generation components, as well as propose multi-task pointcloud -aased policy architecture that distills gen- erated demonstrations from simulation transfer to the real world. Future works include expanding task complexity and diversity through advanced muulti-mdal agents and 3D asset generation. Future works can explore advanced sim-to sim-to-real methods for complex tasks with multiple embodiments.

---

Acknowledgments
We want thank Professor Xiaolong Wang for his kind support and discussion of this project. We thank Yuzhe Qin and Fanbo Xiang for their generous help in Sapien development. We thank Mazeyu Ji for his help with real-world experiments. This work is partly supported by the Amazon Greater Boston Tech Initiative and Amazon PO No. 2D-06310236.
References
[1] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang. L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference Computer Vision and Putlern Recognition (CVPR), June 2020.
[2] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dkebiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint arXiv:19/2.066 2019.
[3] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 350(7676):34-359,, 2017.
[4] T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong. Solving olympiad geometry without human demonstrations. Nature, 62777995) 2024.
[5] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint 1:X1v:1910.07113, 2019.
[6] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 62097976),982-987, 2023.
[7] K. Fang, T. Migimatsu, A. Mandlekar, L. Fei-Fei, and J. Bohg. Active task randomization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks. arXiv preprint arXiv:2211.061344 2022.
[8] M. Deitke, E. VanderBilt. A. Herrasti, L. Weihs, K. Ehsani, J. Salvador, W. Han, E. Kolve, A. Kembhavi, and R. Mottaghi. Procthor: Large-scale embodied ai using procedural genera- tion. Advances in Neural Information Processing Systems, 35:5982-5999 2022.
[9] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang- Gensim: Generating robotic simulation tasks via large language models. In The Twelfth Inter- national Conference on Learning Representations, 2023.
[10] Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01453 2023.
[11] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.
[12] R. Bommasani, D. A. Hudson, Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bersttein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 202.000..07555 07258, 2021.
[13] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, Yu, R. Soricut, Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.

---

[14] OpenAl. Gpt-4 technical report, 2023.
[15] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr J. Tay- lor, T. Luhman, E. Luhman, C. Ng. R. Wang, and A. Ramesh. Video genera- tion models as world simulators. 2024. URL https: /ooppaiii...mmmmmmmm com/ research/
vi video--eneratin inmrraiiiinnee ooooooommmmdddoee s-rellsass aa--as-woore woolddssimulttrr
[16] B. Rozière, J. Gehring, Gloeckle, S. Sootla, I. Gat, X. Tan, Y. Adi, Liu, T. Re- mez, J. Rapin, arXiv:2308. /2950, 2023.
Code llama: Open foundation models for code. arXiv preprint
[17] W. Huang, Wang. Zhang. Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:230..0597., 2023.
[18] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Tex{2mtioon From natural language instructions feasible plans. arXiv preprint arXiv:2303.1215:. 2023.
[19] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05600, 2022.
[20] D. Valevski, Y. Leviathan, M. Arar, and S. Fruchter. Diffusion models are real-time game engines, 2024. URL ht https:: arwaa.ivvv...o. wwg./v.rrg//ooooooooooooo htt//bb//24000 14837.
[21] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning. 2024.
[22] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models, arXiv preprint arXiv:2310.12931) 2023.
[23] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-guns guided robot skill acquisition. arXiv preprint arXiv:2307. 14535, 2023.
[24] Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 9493-9500, IEEE, 2023.
[25] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[26] R. Tedrake and the Drake Development Team. Drake: Model-basee design and verification for robotics, 2019. URL https:://rraa mit edu.
[27] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gau-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
[28] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, What matters learning from offline human demonstrations
Y. Zhu, and R. for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[29] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin,
D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
10

---

[30] B. Liu, Y. Zhu, C. Gao. Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong rohot learning. Advances in Neural Information Processing Systems, 36, 2024.
[31] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robol learming, pages 1094-1100. PMLR, 2020.
[32] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robolics and Automation Letters, 5(2):3019-3026, 2020.
[33] L. Wang, K. Zhang, A. Zhou, M. Simchowitz, and R. Tedrake. Robot fleet learning via policy merging. arXiv preprint arXiv:2310.013622 2023.
[34] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 JEEE/RSI international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[35] Z. Chen, S. Kiami, A. Gupta, and Kumar. Genaug: Retargeting behaviors to unseen situa- tions via generative augmentation. arXiv preprint arXiv:2302.06671. 2023.
[36] F. Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilis- tic inference for robotics simulators. arXiv preprint arXiv: 1906.01780 2019.
[37] L. Makatura, M. Foshey, B. Wang, F. HähnLein, P. Ma, B. Deng. M. Tjandrasuwita, A. Spiel- berg, C. E. Owens, P. Y. Chen, et al. How can large language models help humans in design and manufacturing? arXiv preprint urXiv:2307.13377 2023.
[38] H. Jun and A. Nichol. Shap-e: Generating conditional 3d implicit functions, arXiv preprint
20050433,..02463, 2023.
[39] A. Nichol, H. Jun, P. Dhariwal, Mishkin, and M. Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751/ 2022.
[40] B. Poolc, A. Jain, T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[41] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint orXiv:2302.115333 2023.
[42] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandku- mar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[43] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[44] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-
11530. IEEE, 2023.
[45] J. Duan, W. Yuan, W. Pumacay, Y. R. Wang, K. Ehsani, D. Fox, and R. Krishna. Manipulate- anything: Automating real-world robots using vision-language models. arXiv preprint arXiv:2406. 18915, 2024.
[46] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.

---

[47] L. Wang, Y. Xiang, W. Yang. A. Mousavian, and D. Fox. Goal Goal-auxiliary actor-critic for 6d robotic grasping with point clouds. In Conference on Robot Learning, pages 70-80. PMLR, 2022.
[48] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[49] Y. Ze. G. Zhang. K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954 2024.
[50] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[51] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737-744. IEEE, 2020.
[52] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: International Conference on Robotics and Automation (ICRA), pages 8973-8979. IEEE, 2019.
Adapting simulation randomization with real world experience. In 20/9
[53] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[54] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[55] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized to to-cnnoni adaptation networks. In Proceedings the IEEE/CVF conference on computer vision and patlern recognition, pages 12627-12637, 2019.
[56] L.. Wang, J. Zhao, Y. Du, E. H. Adelson, and R. Tedrake. Poco: Policy composition from and for heterogeneous robot learning. arXiv preprint arXiv:2402.02511, 2024.
[57] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132- 157. Springer, 2019.
[58) F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mask- visual prompting. arXiv preprint arXiv:2403.03174, 2024.
[59] N. Di Palo and E. Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403. 19578. 2024.
[60] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024.
[61] L. Wang, X. Chen, and K. H. Jialiang Zhao. Scaling proprioceptive-visual learning with het- erogeneous pre-trained transformers. In Neurips, 2024.
[62] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-graines grainadind bimanual manipulation with low-cost hardware. arXiv preprint a3X00:2304.13755, 2023.
12

---

[63] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 27X17 2023.
[64] X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Kr- ishna. Blink: Multimodal large language models see but not perceive. arXiv preprint ar0iv:2404. 12390, 2024.
[65] D. Liu, X. Dong, R. Zhang, X. Luo, Gao, X. Huang, Gong, and Z. Wang. 3daxiesprompts: Unleashing the 3d spatial task capabilities of gpt-4v. arXiv preprint arXiv:2312.09738, 2023.
[66] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00,:00777.06377 2017.
[67] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https: / //oi. wrg/10........ /10. 5281/ zenodo. 5143773. If you use this software, please cite as below.
[68] G. Qian, Y. Li, H. Peng. J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[69] G. Yan, Y.-H. Wu, and X. Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint 240.:24003.4115.. 2024.
[70] Y. Ze, G. Yan, Y.-H. Wu, Macaluso, Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang- Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284-301 PMLR, 2023.
[71] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
13

---

Appendix
A Task Generation Details
A.1 Asset Statisties
We summarize the information of the articulated asset library we used in this project in Table 4. To ease the workload of the L.M in understanding the manipulatable parts of Articulation, we only reserve one unfixed (prismatic revolute) joint in each object and fix the others. For rigid body objects, we build dataset adapted from the YCB object dataset in Maniskill2. We have modified the scale and joint limits of the objects to make the scene proper.
A.2 Task Generation Statisties
In this section, investigate the statistics of the tasks in the generation process. We have generated 100 tasks based on the asset library described in Appendix 1. including 50 primitive tasks and 50 long-horizon tasks. Notice that we consider tasks with the same object class but different instances as the same tasks. Despite the tasks we have generated, these tasks do not include all assets in the asset library and the asset library can also be expanded to contain more object classes and instances.
A.3 Task Solver Details
kPAM Planner. kPAM[57] is a keypoint- based planner proposed for category-level manipulation lasks, and we follow the roadmap proposed in Wang et al. [33] to improve kPAM and apply it in our articulation tasks. It generally defines a robotic task by the homogeneous transformation required to manipulate the target object. kPAM discovers such transformation by solving an optimization problem built on several keypoint-based constraints. After kPAM implicitly defines the end-effector pose get contact with the object, which call the actuation pose, need to build the trajectory to approach the actuation pose and subsequently complete the task. To this end, pre-actuation and post-actuation motions are introduced, both of which include a series of waypoints to guide the end- effector motions (such as making a turn or pushing by moving forward). Together with the actuation pose, an object-centric trajectory of key poses is designed to solve an articulation manipulation task. We provide an illustration of leveraging kPAM planner to solve task in Fig.
RL Learner. In addition to planners, also leverage reinforcement learning (RL) learner [10] as an alternative solver for the generated manipulation tasks, in which we prompt LLM to produce a reward function for a target task. After the task proposal stage, we we're provided a task definition and task code generated by the LLM which is used as context for LLM to generate an executable reward function. The LLM is supplemented with a small library of curated reward examples as well as an API consisting of reward functions corresponding to various predefined components that can refer to, such as the distance between the end-effector and the object, the joint positions of certain articulated objects. If successful generation is achieved, we append the reward into the task code and execute the final learning code using Proximal Policy Optimization (PPO)[66) and fix a default set of hyperparameters for all tasks.
In summary, we propose the kPAM motion planner and RL policy learner as complementary meth- ods for solving generated tasks. For kPAM, since it is defined by well-formulated optimization problem regarding fine-grained constraints and costs, the output motions are much smoother and more natural than motions from other learning- beasedngbbased methods. Moreover, kPAM is inherently gen- eralizable to different scene configurations and object instances because the constraints and actuation motions are all "object-centric" elements, which will be further exploited for domain randomization in Section 3.3. Furthermore, kPAM is fast to execute (~> 2 seconds planning time given a configura- tion) and robust to multiple runs. Despite the strength of kPAM, RL provides advantages in creative solution proposals for tasks that are too complex or ambiguous for kPAM to define proper con- straints, especially for those involving thin objects or contact-rich manipulation. In this project, we give priority to the kPAM planner in most cases because it produces more realistic and generalizable motions, and leaves those that can not be solved for the RL Iearner.
14

---

Constraints
Actuation Pose
Actuation Motions
(p1. p2, coincident),
Pre-actuation:
(v1, v2, parallel),
(Translate. x, -0.05, (Translate, 2, 4.15)
(v3, v2, orthogonal)
Post-actuation:
(Rotate, 30)
Pre actuation
Post-ectuation
Motion
Motion
Figure 8. Illustration of KPAM Planner. We show how leverage the KPAM planner to solve the task OpenBox. First, constraints are defined to make the gripper get contact with the box lid. Then based the this actuation pose, some actuation motions are assigned to complete the motion of opening a box.
Secure gold safe
Initial scene
"Open the safe"
"Grasp the gold"
"Place the gold into the safe"
"Close the safe"
Prepare breakfast
"Open the Microwave"
"Put the steak in the bowl"
"Put the bowl into the microwave"
"Close the microwave" "Press the toaster"
Figure 9. Illustration of Generated Long-horizon Tasks. We show two example tasks generated by our method, including securing gold in safe (top) and preparing breakfast (bottom). The former contains 4 sub-tasks and is solved by chaining motion planners of all sub-tasks The latter contains 7 sub-tasks and is teleoperated due to its complexity.
A.4 Human Verification
We measure the human efforts required in GenS m2, including labeling keypoints for the motion planner and multi-shot rejection sampling in task solver generation.
Keypoint Labeling. We set up a protocol for keypoint labeling and annotated 1-3 keypoints for each asset. We measured the time spent on 10 objects and the average time is 8.2s per task.
Rejection Sampling. While task solver generation inevitably involves some degree human in- tervention, have made significant efforts minimize this effort. To evaluate the efficiency of our approach, we conducted user study measuring the time required for human user to generate a novel task and corresponding solver using GenSim2, and compared it with the time needed for manual task design. The results are presented Tab. We consider the performance one of the project's authors (User 1) as expert proficiency and additionally involve two other users who are unfamiliar with our pipeline. The findings indicate that our method significantly reduces task creation time by 50%, and over 90% of the required time involves waiting for LLM to response. Furthermore, the reduced performance gap between experts and beginners suggests that our pipeline requires less technical expertise, making it accessible to users with varying levels of familiarity.
15

---

Table Task creation time measurement. The results are all based on tasks. The average task creation time is around 4 minutes for GenSim2.
User 1 (Expert)
User 2
User
Method
GenSim2
Manual
GenSim2
Manual
GenSim2
Manual
Average
159.4s
463.8s
241.0s
778.6
292.0s
727.4s
A.5
Demonstration Generation Statistics
In this section, assess the efficiency and quality of demonstration generation using our kPAM- based motion planner. Recalling our goal to use generation pipeline as data amplifier to enhance the generalization and sim-to-real transfer ability of simulation policy, the generated data should be diverse and of high quality for data augmentation or domain randomization.
To this end, we evaluate the robustness of our kPAM solver random scene configurations and object instances. For primitive tasks, we add uniform noise in [-0.1m, 0.1m] the and y positions of the object, as well random rotation in [-30%, 30 to the yaw axis. For some hard long-horizon tasks, in which the articulation is so small that it may be struggling to place something inside, we fail to generate successful demonstrations with normal randomization, so we reduce the scale of noise by half. For articulations, we randomly select an instance adapted from PartNet-Mobility and add noise to its default joint position.
We test the success rates of generating demonstrations for 24 single-step and 15 long-horizon tasks, and report them task by task in Tab. 5. For each task, run the demo collection process 5 times, and each time the kPAM solver is executed for 50 episodes with both spatial and object randomization. We compute the mean and standard deviation of the 5 runs on the success rate.
16

---

Table 4. Statistics of assets used in GenSim2
Object
Num
Joint Type
Asset Description
rotaep_
44
revolute
A laptop fixed on the table, with lid connected by revolute joint that can be opened and closed
stapler_move
3
prismatic prismatic
A stapler that can be moved on the table
window_push
2
A window with a frame connected by a prismatic joint that can be pushed left and right A suitcase that can be moved on the table
suitcase_move
3
prismatic
oven
4
revolute
An oven fixed on the table, with door connected by revolute joint that can be opened and closed
drawer
20
prismatic
A drawer fixed on the table, connected with its body
by prismatic joint that can be opened and closed
box_move movee
prismatic
A box that can be moved on the table
kitchen.pon
3
prismatic
A kitchen bot fixed on the table, connecting its lid and body with a prismatic joint
bucket_lift
5
prismatic
A bucket that can be lifted from the table
trash_can
3
revolute
A trash can fixed on the table, with lid connected by revolute joint that can be opened and closed A stapler fixed on the table, with its handle connected to its body by revolute joint that can be pressed
stapler press
3
revolute
bottle
2
revolute
A bottle fixed on the table, with a top connected to its body by prismatic joint
bag_move move
2
prismatic
A bag that can be moved on the table
dishwasher
3
revolute
A dishwasher with door connected by a revolute joint that can be opened and closed
muicasc_rotate
4
revolute
A suitcase fixed on the table, with lid connected by revolute joint that can be opened and closed
toaster move
10
prismatic
A toaster that can be moved on the table
bag_swing
2
revolute
A bag fixed on the table, with strap connected by revolute joint
toilet
2
revolute
A toilet fixed on the table, lid connected by revolute joint
door
2
revolute
A door with its frame connected by revolute joint that can be opened and closed
coffee_machine
2
prismatic
A coffee machine fixed on the table, with button con- nected by a prismatic joint to pressed
faucet
13
revolute
A faucet fixed on the table, with handle connected by a revolute joint that can be turned on and off
washing machine
2
revolute
A washing machine fixed on the table, with door con- nected by a revolute joint
switch bucket swing
2
revolute revolute
A switch with frame connected by revolute joint
26
A bucket fixed on the table, with a handle connected
by revolute joint
safe move toaster_press
2
prismatic prismatic
A safe that can be moved on the table
10
A toaster fixed on the table, with button connected by prismatic joint
window_rotate refrigerator
2
revolute revolute
A window with a frame connected by a revolute joint
2
A refrigerator fixed on the table, with a door connected
by revolute joint that can be opened and closed
laptop_move safe_rotate
3
prismatic revolute
A laptop that can be moved on the table
15
A safe fixed on the table, with a door connected by revolute joint that can be opened and closed
bag lift microwave
2
prismatic revolute
A bag that can be lifted from the table
10
A microwave fixed on the table, with door connected by revolute joint that can be opened and closed
box_rotate
13
revolute
A box fixed the table, with lid connected by
revolute joint that can be opened and closed A bucket that can be moved on the table
muve
2
prismatic
17

---

Table Success Rates of demonstration generation on different tasks
Task Name
Task Type
Success Rate
OpenBox
Primitive
0.84 0.07
CloseBox
Primitive
0.94 + 0.03
OpenLaptop
Primitive
0.76 0.03
CloseLaptop
Primitive
0.95 + 0.01
TumonFaucce
Primitive
0.67 0.05
TunoOfFaucet
Primitive
0.72 0.03
OpenDrawer
Primitive
0.80
PushDrawer
Primitive
0.87 + 0.06
SwingBucketHanddle
Primitive
0.89 0.03
ResstraasterLeeee
Primitive
0.96 + 0.03
RotateMiicrowaeeDon
Primitive
0.92 = 0.01
CloseSafe
Primitive
0.80 = 0.04
OpenSafe
Primitive
0.62 + 0.03
Push ToasterF
Primitive
0.99 +: 0.01
CloseSuitcaseLid
Primitive
0.82 + 0.05
SwingSuitcaseLidOppr
Primitive
0.81 = 0.06
RelocateSuitcase
Primitive
0.80 + 0.04
LiftBBuckttUprigh
Primitive
0.76 0.05
MoveBagForward
Primitive
0.72 + 0.05
CloseMicrowave
Primitive
0.53 0.04
SwingDoorOpen
Primitive
0.79 + 0.03
ToggleDoorClose
Primitive
0.80 + 0.05
Primitive
0.82 + 0.03
Primitive
0.75 0.06
MiceGolfBalllntoDrre
Long-horizon
0.56 0.09
Long-horizon
0.53 = 0.08
PlaceLemonIntoDraweee
Long-horizon
0.53 + 0.06
PlaceSottBalllttDDaawaa
Long-horizon
0.58 0.06
PropAppleenntoDraweer
Long-horizon
0.58 + 0.05
StachCuplnBox
Long-horizon
0.40 0.09
PlaceGolrBalllntoooxx
Long-horizon
0.58 + 0.06
Pun
Long-horizon
0.48 + 0.08
Itore
Long-horizon
0.62 + 0.03
Inopooooe
Long-horizon
0.64 + 0.04
StoreLemonnnneefiigrratooo
Long-horizon
0.48 0.07
PlaeeTaackerBootttooRdiiiinngg
Long-horizon
0.26 +.09
SecureGoldInaafe
Long-horizon
0.21 0.05
FillMug WithWater
Long-horizon
0.13 + 0.04
18

---

B Multi-Task Training Details
B.1 Task Settings
We gradually add the number of training tasks from 4 to 24 in Section 5.1 and test the scaling ability of our framework on the original 4 tasks. We provide detailed task list here to better clarify our task settings:
4 tasks: OpenBox, CloseBox, OpenLaptop, CloseLaptop:
10 tasks: 4 tasks OpenDrawer, PushDrawerClose, SwingBucketHandle, LiftBucketUpright, PressToasterLeeve
PushToastteForrwrdd;
15 tasks: 10 tasks MoveBagForward, OpenSafe, CloseSafe, RotateMicrooavveDorr CloseMi-
crowave;
20 tasks: 15 tasks CloseSuitcasel.id, SwingSuiicasell.ddOpnn. http:/acccce.cccooooeeecoccataae TurnOnFaccet, TuroOiFaauct;
24 tasks: 20 tasks SwingDoorOpen, ToggleDoorClose, CloseRefrigerrttrDDor, OpenRefriger- atorDoor.
B.2 Architecture Implementation
We implemented a multi-modal policy architecture, as illustrated in Fig. 4, which includes trans- former policy stem, an action head, and different encoders for modeling various types of observa- tions as tokens.
Specifically, we take the CLIP [67] tokenizer to encode the task instruction, which is frozen during training; pre-trained PointNext [68] to encode the point cloud, which is fine-tuned during training; an MLP for encoding the proprioception states, which is trained from seratch. The transformer conducts self-attention over all tokens, and then we post-process the tokens for different action heads. For example, we compute the mean pooling all the tokens as the global condition of the diffusion and MLP head; for the transformer, we compute cross-attention between the modeled token and of position embeddings get the final action sequence. Regarding the PointNext encoder, utilize the pre-trained model on ScanObjectNN Classification' as it does not include color information, making it easier for downstream sim-to-real transfer. Note that the diffusion head and the transformer head both model and predict action sequences, but the MLP head only works for single-step action.
B.3 Policy Architecture Comparison against Baselines.
We first evaluate the Proprioception Pointcloud Transformer (PPT) in solving multiple tasks. We train policy with demonstrations of 10 tasks the commonly used manipulation benchmark, RLBench [32]. In particular, we train PPT for 250 epochs on data collected using the same setup as in Yan et al. [69]. Results shown in Tab. 6 performance best performances over 6/10 tasks improved performance over recent competitive baseline methods such as PerAct [46] and GNFactor [70] with a significant reduction required camera views and pre-trained feature representations. Note that GNFactor [70] requires rather hassles to set up for real-world experiments, it requires getting features from the neural radiance field [71]. On the contrary, our proposed PPT architecture only requires point cloud, languages, and robot sensor states, which are much easier to obtain with real- world robots.
B.4 Additional Experiments for Object-Levve Generalization
After multi-task training, we test the generalization ability our policy on unseen object instances. We chose tasks using assets with more than instances and 10 instances to respectively construct
/https:/ / /rrve. wwwgle..com/tiii//eeeeeeeeeeeeee 7100018IIPuppshhat
AAgpp ( (NNiVqIA_
19

---

Table 6. Multi-Task Performance RLBench. We evaluate 25 episodes for each checkpoint on 10 tasks across seeds and report the success rates (%) of the final checkpoints. Our method outperforms the most competitive baseline PerAct [46] and GNFactor [70] over 6/10 tasks.
turn tup
drag stick
open fridge
sweep to dusipan
phone on
place
slide
put
Methods
drawer
grill
buse
wine
block
safe
PerAct
57.3
14.7=6.1
1.066
19.5
4.066.
17.3=14.0
98. T=
6.7+6.1
29 3+32
48 0=262
GiFFactor
48.0
68.0aas6
2.7
12.0aa6.
61.3
96. 30.0a66
8.0.
30.7-61 21.326
PPT (ours)
68 0so 10
89.3+0.0 0e
10. 1+0.10
22. 7=0.00
30.1=0000
57.3=0022
11.000.04
29.3+0.00
21.300.00
52.0+0.00
Generalization to Unseen Objects
0.8
Success Rate
0.65
0.59
0.6
0.49
0.46
0.4
0.32
0.32
0.31
0.23
0.21
0.2
0.18
0.17
0
Tasks Train
5 Tasks Test
15 Tasks Train
15 Tasks Test
Diflusion
Transformer
Figure 10. Results for object-level generalization test. We show that our policies with various head architectures can generalize to unseen objects.
two training sets of 5 tasks and 15 tasks. During training, we only leverage data generated from 90% of the instances and leave the remaining 10% for testing. The initial poses of the objects are randomly initialized in both procedures mentioned.
We find that the success rates only drop by around 3% on unseen instances, as shown in Fig. 10. These results indicate that by generating data with object-level and spatial-level variance as domain randomization, our policy can acquire generalization both aspects, which lays the foundation further sim-to-real transfer.
C Real- World Details
We selected real-world tasks to evaluate the multi-task policy trained on generated data from our pipeline. The OpenLaptop task requires the robot fully open a par- tially closed laptop. similarly CloseLaptop closes the lap- top lid. The OpenSafe and CloseSafe performed similar opening and closing motions on an articulated safe door. The CloseDrawer task required the robot to reach and close an open drawer. The generated SwingBucker task required the robot push a handle perpendicular its current position. The OpenBox task required the robot to open box lid from partially closed position and the CloseBox task required the robot to fully close box lid.
Fertic Serrondy
Canera
Figure 11. Real-world Setup.
As we didn't utilize a discrete action space such as keyframes for real-world experiments, real- time control was important. We achieved fast inference speeds with our multi-task policy with 0.1s inference latency on NVIDIA 3080 GPU. We parallelized and synchronized point cloud processing from 3 Intel Realsense D435 cameras in order prevent additional latency. Please view Fig. 1l to see the robot workcell described. Processing the point clouds involved uniform sampling
20

---

Table Real-world Generalizability Experiments. We transfer a multitask policy from sim to real and evaluate 15 episodes for each task and object and report the success rates below.
Task
OpenBox
OpenLaptop
CloseDrawer
Asset
Small
Medium
Long
Large
Toy
Real
Short
Tall
Average
Success Rate 46.7%
60.0%
40.0%
40.0%
80.0%
73.3%
80.0%
53.3%
61.7%
Real World 3D Observation: Point clouds of different objects in the real world. Cat- egory of objects from left to right: laptop, safe, box, and bucket. Color is shown for easier under- standing, however, color is not used input the multi-task policy.
Figure 12.
step for time efficiency, farthest point sampling, and outlier removal to clean and simplify the point cloud for successful inference during Sim-to-Real transfer.
C.1 Generalization on Unseen Objects
In addition, we also evaluate how well the multi-task policy could generalize to different objects within each task. We run an additional set of experiments for the OpenBox, CloseDrawer, and OpenLaptop tasks, on varying objects. Thanks to the training scale in simulation and the design of PPT policy architecture, various unseen objects, as shown in Tab. 7. In addition, the performances on different objects within the same task are comparable, demonstrating the robust real-world generalization achieved by our policy.
method can achieve over 60% success rate multiple tasks with
C.2 Co-Training Experiment Details
To conduct additional experiments on differ- ent multi-task policies incorporating real-world data we incorporated teleoperation setup us- ing an HTC Vive controller and base station to track 6-DOF hand movements which were mapped to control the robot arm's pose. To in- terface with the HTC Vive controller we used the (triad-openvr package) and SteamVR. We implemented position-velocity controller for the robot to track given poses from the con- troller. Using this teleoperation system, we col- lected 10 demonstrations for each generated task, collecting state and observation data for each trajectory. We co-trained a multi-task policy using 100 episodes of simulation data per task and the 10 collected demonstrations of real data per task.
Figure 13, Comparison simulation (left) vs real (right) pointcloud of laptop.
D Comparison with Previous Works
Different from previous efforts including GenSim (top-down manipulation tasks), RoboGen (dy- namic tasks), and Robocasa (mobile manipulation tasks with multiple embodiments), GenSim2 fo- cuses on 6-DOF realistic tasks that allow task and high-quality data generation at scale, which really
21

---

Table Comparison several released generation frameworks robotics simulation.
Methods
GenSim2 (ours)
GenSim[9]
RoboGen 10)
[oboCasa[11]
Task Type
Articulation
'Top-donn From scratch
Articulation
Articulation
Data Generation
From scratch
From scratch
Teleoperated
Efficiency
Fast
Fast
Slow (RI.)
Slow (leleoperation)
Data Transferability
High
High
Low (RL)
Medium
Sim-to-Real
7ero-shot/co-train
7ero-shot/finctune
None
Co-train
benefits solving real-world tasks. However, contrast to GenSim2, GenSim is different from the task level, which primarily deals with simpler top-down pick-place tasks; RoboGen different from the demonstration generation method, which leverages RL for manipulation tasks and thus produces unrealistic and untransferable data for real-world tasks; RoboCasa is different from the pipeline, which requires large amounts of human teleoperation to collect source data. Moreover, our experi- ments on taking advantage of generated simulation data for real-world transfer show that we achieve a success rate 42.5% for zero-shot sim2real transfer and 57.5% for transfer via co- co-training with both simulation and real data, while RoboCasa, which conducts similar sim2real experiments as ours, only achieves score lower than 25% by co-training. We summarize features of generated data of the aforementioned previous works in Tab. 8.
Furthermore, our task and data generation method features multimodal LLMs as well as gener- alizable planner to solve such complex tasks. In our experiment, we verify that the multi-modal task solver generation in this system can greatly improve the task generation success rates by 60%, outperforming previous work by 20%, and the planner used is also friendly and convenient for data generation.
22

---

E Prompt Examples
E.1
Prompt Templates
In this section, demonstrate some examples of the prompt templates we have used to query LLM.
The prompt templates are shown as follows:
Prompt: Task Proposal
You are an expert in creating robotic simulation environments and tasks. You are given some artic- ulated assets for example. Please come up with creative use of the gripper to manipulate single Articulation. Note that the simulation engine does not support deformable objects accurate colli- sion models for contacts. Moreover, the robot can only execute simple trajectory of one motion.
Here are all the assets. Please try to come up with tasks using only these assets.
Here are some examples good tasks. Try learn from these structures but avoid overlapping with them.
Here had example task instances with reasons. Try avoid generating such tasks.
reasons:
Please describe NEW task in natural languages and explain its novelty and challenges. Note:
Do not use assets that are not the list above.
Do not repeat the tasks similar to the good examples or the already generated tasks.
The task needs to obey physics and remain feasible.
Do
similar tasks with the
"assets-used"" set.
All assets are on the table when initialized.
Do not place objects on small objects.
Only one Articulation can be loaded.
The task contains simple trajectory of one motion.
The task should have clear goal, e.g. "open/close" instead of "adjust position".
Before the next step, please check the generated task bad task shown in the above examples and meets all the criteria as stated above. Specifically, if the task **only** contains a simple tra- jectory of one motion, and should have **clear*** goal. Explain in detail, and get a conclusion. If the task is had task, regenerate new one.
Then, format the answer in Python dictionary with keys "task-name" and value type string, "task- description" (one short phrase), value type string with lower-case and separated by hyphens, "assets-used" and value type list of strings, and "success-criteria" (choose from "articulated "articulated, clossedd "distance articulated "iiiiulated "distance ceataaccegriiper..c wrwppgrrpppprr rigiibhody" and "dis- tance gripper articulated")) and value type list of strings. Try to be creative possible.
Please remember not to add any extra comments to the Python dictionary. Let's think step by step.
Prompt: Task Decomposition
You expert in creating robotic simulation environments and tasks. A robot with 2- finger gripper is used all the robotic simulation environments and tasks. In each task, there exactly one Articulation and one rigid body object that you can manipulate. You will be given long- hing-horizon tasks with each task including at least 2 sub-tasks. Each sub-task can only include simple motion such moving the gripper some object, opening closing the gripper fin- gers, or interacting with certain Articulations by its prismatic/revolute joints.
Please come up with decomposition of the given horizon task to get several sub-tasks.
Some rules of such decomposition are listed here:
23

---

1. Each long-horizon task can not include over 5 sub-tasks, and usually 3-4 are enough.
2. Each sub-task should only include one simple motion as mentioned.
3. Each sub-task(except "grasp" and "ungrasp") should be presented in the format of Python dictionary with keys "lask-name" and value type string with lower-case and separated by hy- phens, "task "isk-ddesrrpption (one specific sentence) value type string, "assets-ure uuedss and value type list with necessary asset(s) in the current sub-task, and "success-critenii (choose from "ar- ticulated_open", and "distance_grue"
srcccullatd_closedd
sattanncearticulatee rigidboddy" dustanceerriperrriiddddy
wwt.rggrppeperrrcuiiiie articulated"") and value type list of strings.
4. Each sub-task should have only
used in the task.
5. motion opening or closing the gripper fingers included the whole task, it should be listed as a separate sub-task, whose "task-name" should strictly be "grasp" or "ungrasp" respec- tively and should be the only key in the dictionary.
Here an example of the dccomposition of the following long-horizon task "..."
# Sub-task
# Sub-task
# Sub-task
# Sub-task 4
Now please start to generate sub-task decomposition of the following new task:
Prompt: Code Generation
Now will provide you with some reference code you can write code the task NTAE
The generated code should follow the same structure the reference and call similar functions. Do not use libraries, extra functions, properties. arguments, assets that you don't know. Remember to import used functions from the corresoondigg files as the example task codes. For the Articulation. use "self.articulatoo to refer to it, which should be the same as the "assets- used" of the task.
For the objects used, you only have pass the corresponding parameter (c.g, articulator) and its name (string format, c.g., 'box') the '_iint...' function as shown above codes, and the base class will automatically load them.
Please comment on the code to explain what each piece does and why it's written that way.
Now write the code for the task "TASK NAME TEMPLATE" in the Python code block.
Prompt: kPAM Solver Stage
You expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.
The task solution config contains the necessary positions, parameters, and keypoints for existing trajectory optimization algorithm to solve a feasible solution. It mainly contains two parts, con- straints and pre/post-actuation motions:
(1) The constraints are used to ensure the gripper is in contact with the object and to implicitly define certain actuation pose.
(2) The pre- pre-actuation motions are used move gripper the actuation pose, while the post- actuation motions are used to complete the task after the actuation pose.
Here is the task description.
24

---

Here are all the available keypoint names for the used manipulator and asset and their descriptions.
Here some examples of the constraint part of some configs.
Note that, in the constraint list, you need to define different items of constraint to define an actua- tion pose for the task. There are some pre-defined types of constraints you can use:
(1) point2point_constne connttain::: This constraint is used ensure two keypoints ("keypoint__nmm and "target keypoint name", respectively on the tool and object) are in contact.
(2) frame axis parallet: This constraint is used to ensure two axes (respectively on the tool and ob- ject) are parallel. The axis on the tool is defined by unit vector from "axis_from ferponkypoonttttmmmmmmmmmmmmoom _amee he..kkeppitttnaaaa
while the axis on the object is defined by "target.axis" [0,1,0] or [0,0,1] which is in the coordinate frame of "target.axis. aar_sstaasfffaee (world or object). (3) frame_axe _wa.aaiiiitttoa orthhgona:: This constraint is used to ensure two axes (respectively on the tool and object) are orthogonal. The axis on the tool is defined by unit vector from "axis, frxisfrrm__ee kenpkieypoonaaatt _aame to "axis to keypoint_ name"" while the axis on the object is de- fined by Marget_axis"" "[1,00
or
[0.1,0] [0,0,1] which is in the coordinate frame of "tar- framee "rame"(worlk) or object). aeta
get axis
(4) keypoint axis parallel: This constraint is used to ensure two axes (respectively on the tool and object) are parallel. The axis on the tool is defined by unit vector from "axis_from_ ftrom/eypooittttmmmm
"axis_to ketis.tt.eeoiittttaa.. another unit vector from "target_axis, from_kcypoint, name" to arrget_axistt..kypooii (5) grppoint_xisoorthogom
while the axis the object is defined by
This constraint used to ensure two axes (respectively on the tool and object) are orthogonal, The axis on the tool is defined by unit vector from "axis. from keypoint name" to "axis to keypoint name", while the axis the object is defined by another unit vector from "target axis from keypoint name" to "target axis to keypoint name".
The tolerance is used to define the tolerance of the constraint.
The aeee-iiner_ppoduee is used to define the inner product between the two axes. For exam- ple, if you want to ensure two axes are parallel and in the same direction, you can set the tar- it-inner-produce to 1.0. If you want ensure two axes parallel and of opposite directions, you can set the target_inner, productt to -1.0. you want to ensure two axes be orthogonal, you can set the target_ .product to 0.0.
Usually, you need to define one point2point.constrain to ensure contact and several axis con- straints to adjust the actuation pose.
Now please first generate the constraint part for task "TASK_NAME NAMENNLLLLEETLTTTTTTT in the same config format as the above. Do not use terms that you have not seen before. The output should be in the YAML format with no extra text.
Prompt: kPAM Solver Stage
You are an expert in solving robotic tasks by providing some motion plans and coding task solu- tion configs for a 2-finger rohot arm. Now please salve the newly generated task by generating the task solution config.
The task solution config contains the necessary positions, parameters, and keypoints for existing trajectory optimization algorithm to solve a feasible solution. It mainly contains two parts, con- straints and pre/post-actuation motions:
(1) The constraints are used to ensure the gripper is in contact with the object and to implicitly define certain actuation pose. An actuation pose means the key frame that the robot arm manipu- lates the object, usually representing the moment when the gripper gets contact with the object. (2) The pre-actuation motions are used to move the gripper to the actuation pose, while the post- actuation motions are used to complete the task after the actuation pose.
Here is the task description.
Here are all the available keypoint names for the used manipulator and asset and their descriptions.
25

---

Here are some examples the pre/post-actuation actuattio part some task solution configs.
Now please generate SOLVER_TRIAL different pre/post-actuation motions for task "TASK NAME NAME. TEMPLATE" following the same config format shown above based on the con- straint part that is generated previously. The pre/post-actuation motions of difterent solutions can be diverse. but their task names should be the same.
Do not use terms that you have not seen before. The output should be in the YAMI. format with
text.
motions can he achieved by using different axes translation. The diversity pre-actuation and post-actuation motions are relative to the actuation pose and the Notice that the translation motions are represented in coordinates relative to the manipulator base.
Let's think step by step, and try your best to understand the job.
26

---

E.2 Example for Multi-modal LLM Response
We provide an example conversation when generating solver config for the task "close-drawer" with a molti-modal LLM, and compare with the response generated by vanilla LLM. Some of the prompt details are omitted, and please refer to Appendix E for the complete prompts.
Example for Multi-modal LLM Response
Input:
You are an expert in solving robotic tasks by coding task solution configs. Now plcase solve the newly generated task by generating the task solution config.
Here is the task description.
"lask-name": "close-drawer", daask-description.. "push drawer to closed position",
"asscts-used"; ["drawer"], "success-criteria": |"articulated closed")
The initial scene of this task visualized in the first uploaded image. The frame axes of gripper and the object are respectively visulized in the second and third images, where red, green blue axes represent X, Y and axes respectively.
Now please first generate the constraint part for task "close-drawer" in the same config format as the above.
Output:
task name: CloseDrawer
aname: Articulated
mool_keypoint.nams [tool_head, tool stii, tool_side]
bojjcc.kcypoint.name... jarticulated.ohieet.hedddd
constraint Jist:
keypoint_namee tool_hear
target target.keypone keypointtnnaae name: articulated. object heaa head
tolerance: 0.0001
type: point2point_constee cconstriit
wxis.from_keypoint.co
thad
sxis_to_keypoint_nne tool_side
target_axis: [1.0, 0, 0]
27

---

target_axis_frai frame: object
tolerance: 0.01
target_inner inner-prodduct
type: frame_axis catnneaxxisppppaallll parallel
axis. from keypoint name: head
axis to keypoint name:
name:: tool
rrget_axiis [0, 0, 1.0]
target axis frame: object
tolerance: 0.01
type: frame_axis caann__xxis_ortthee oothoonna
Input:
The initial scene of this task is visualized in the first uploaded image. The actuation pose of this task is visualized the second uploaded image. The frame axes the gripper and the object are respectively visualized in the third and fourth images, where red, green, and blue axes represent the X, Y, and axes respectively.
Now please generate different pre/post-actuation motions for task "close-drawer" following the same config format shown above based on the constraint part that was generated previously. The pre/post-actuation motions of different solutions can be diverse, but their task names should be the same.
Output:
task_nane mask: CloseDrawer category Articulated
# Pre-actuation motions for the manipulator.
pre aetacctttiion motions:
["translate_x" -0.1]
["translate__z, -0.15]
# Post-actuation motions for the manipulator.
post actuation motions:
["translate x", 0.1]
Success!
28

---

Example for Vanilla LLM Response
Input:
You expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.
Here the task description.
nams-,
"task-descriptioon:: "push drawer to closed position",
"assets-used"; ["drawer"], "success-criteria" ("articulated closed"]
Now please first generate the constraint part for task celose-drawerf in the same config format as the above.
Output:
task CloseDrawer category name: Articulated
(ool_keyponntnanee list: [ead, tool.tail, tool_side] bojjett.kypoonttnnmm..ii seffeullted_object.hheee head]
constraint list:
keypoint name: tool head
target keypoint_nanee name: wowpulattedooooee...ooee head
tolerance: 0.0001
type: wwtnutppoint.oossrraat
oxis.from.. www.ffrommepyooo keypoint name: tool_side
axis_to tet.too_eypoiittnne name: tool_tail
target axis: [0, 1.0,
Visualization of Actuation Pose
target axis frame: world
tolerance: 0.01
type: frame_axis, paraalel
axis_from_kie fcypoit name:: thad
axis tiistto_kyypontt _nam:: tool_tail
target_axis: [0, 0, 1.0]
target_axis, framee: world
tolerance: 0.01
target inner wewiiinnneeeiuue product:
type: frame axis orthogonal
Fail to generate correct constraints!
29

---

E.3 Example for Reasoning LLM Response
We provide an example conversation when generating a solver config for the task "close-safe" with reasoning L.L,, and compare with the response generated by vanilla L.M.. Some of the prompt details are omitted, and please refer Appendix for the complete prompts. We highlight some limitations of applying vision language foundation models robotic simulation today, but expect further progress, such finetuning APIs, would be able to correct some of these failure modes.
Example for Reasoning LLM Response
Input:
You expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.
Here the task description.
"lask-name": "close-safe",
"task-descriptioon:: "Close the safe door", "assets-used": ["safe_rotate"" "success crattesse: ("articulated.closed""
Now please first generate the constraint part for task "close-safe" in the same config format as the above.
Output: (Reasoning parts tracked with blue)
Step Understand the Task Keypoints
Task: Close the safe door.
Manipulator Keypoints:
tool_head: Middle point between the two finger tips.
tool tail: Middle point between the two finger bases.
tool side: Tip of the right finger.
Articulated Object Keypoints:
articulated -ojject__tttt_ticee head: Middle point of the safe handle.
articulated www.ooooeeeecstteeeeeeeeeee base: Point inside the safe to drop objects.
Step 2:
Plan Feasible Constraints
We need to define constraints that can be satisfied and are different from the previous attempts. Constraints typically include:
Pnint-to-Point Constraint: Ensure contact hetween keypoint the tool and keypoint the object.
2. Axis Constraints: Orient the tool relative to the object to facilitate the action.
Config
Constraint Point-to- to-Point Constraint
Purpose: Ensure the right finger tip contacts the safe handle. Implementation:
keypoint__amee tool poie
target keypoint kergot.k name: articulated obijeet head
tolerance: 0.0001
type: poin@ppoint.constrintt
Constraint Frame Axis Parallel Constraint
Purpose: Align the gripper's ' Z-axis with door's closing direction.
30

---

Assumptions:
The door closes along the object's X-axis ((1, 0, 0)).
"The gripper's Z-axis is from tool_tail to tool hea_.
Implementation:
axis, from wwwmfrrooppppppee kwypoient_naaaa nnae:: tool_tai) tail
axis tos.to..d keypoint name: tool head
onget_axxis [1.0, 0, 0]
target.axis axis frame: object
tolerance: 0.01
target.inner.rron -ronuutt 1.0
type: frame trame_axis_perr aaaaallllllel
Constraint Frame Axis Parallel Constraint
Purpose: Align the gripper's 's X-axis with door's hinge axis.
Assumptions:
The door's 's hinge axis is along the object's Z-axis ((O, 0, ID.
'The gripper's X-axis is from tool to tool_side.
Implementation:
axis frxis.from_ai /wrommmcypoitt.nne name: tool_head
axis t...tookeyyiiitt name: tool_side
target axis: [O, O, 1.0]
target axis frame: object
tolerance: 0.01
target.inner.pros 1.0 type: frame parallel
Config
task name: CloseSafe
category name: Articulated
keol.keypoint.eo
www.eeetooooaeeeeeelee name_list. [tool_ tail, tool_side]
keypoitt name list: [articulated_obee object.heaae articulated.objer objectt htwp:laeed.ooeeee.iiid.ee...e inw.eeeeiiieeeeieeee basc]
constraint
# Ensure the right finger tip contacts the safe handle
keypoint_name tool osi_e
target keypoint name: articulated object head tolerance: 0.0001
type: cointapoit
# Align gripper's Z-axis with the door's closing direction fros_frrm__tf wetomn_cppootttnaat name: ttoll
axis to kcypoint name: tool head
target_axis: [1.0, 0, 0]
target_axis axiss frame: object
tolerance:
target.inne producc:: 1.0
type: frame axis parallel
# Align gripper's X- X-axis with the door's hinge axis axis_from soro__kepoiit_nnaa name:: toal_head axis to keypoint name: tool side
target axis: [0, 0, 1.0]
target axis frame: object
tolerance: 0.01
larget_inner, ppoducct:: 1.0
type: frame axis_paralled
Summary
Right finger tip contacts the safe handle, with gripper's Z-axis aligned the door's closing direction, and X-axis aligned with the door's hinge axis.
31

---

These configurations are different from the previous ones and aim to provide feasible solutions for closing the safe door.
Input:
Now please generate 3 different pre-actuation motions for task "close-safe" following the same config format shown above based the constraint part that is generated previously. The pre-actuation motions different solutions can diverse, but their task names should be the same.
Output:
task name: CloseSafe
category name:
Articulated
# Pre-actuation motions for the manipulator.
# Each motion is represented in the mmode, valuel format.
# Mode:
translate_x, translate.yy translate.2 .z, rotate.
# Value: distance in meters angle radians.
pre pre_actuation motions:
["tramslate z", -0.11
[lift gripper above handle to approach from ahove]
Input:
Now please generate different post-actuation motions for task "closc-safe" following the same config format shown above based on the constraint part that is generated previously. The post-actuation motions different solutions can diverse, but their task names should be the same.
Output: (Reaxoning parts tracked with blue)
Step 1:
Understand the Task
We need to generate three different post-actuation motion plans for the task"close-safe". The post- actuation motions the actions the manipulator performs after reaching the actuation pose plete the task closing the safe door.
Step 2: Consider Possible Motions
Since the goal is to close the safe door, we need to decide how the manipulator move to achieve this, using only the allowed motion set:
Option I: Push the door forward close it.
Step 3: Define the Motions According to the Robot's Coordinate System
Forward Direction:
Along the red axis; useful for pushing actions.
Backward Direction: Opposite the red axis; useful for pulling actions.
Left Direction: Along the green axis; can used to swing the door closed if hinges that way.
Right Direction: Opposite to the green axis.
-Up/Down Directions: Along the blue axis; may not be as relevant for closing a door but can be included for completeness.
Step 4: Generate Motion Plans
Config 1: Push the door closed hy moving forward. Post-actuation motions: ["move-forward"]
32

---

Step S:
Ensure Compliance with Constraints
Each motion plan uses only the allowed motions.
Motions are within the 3-step limit.
Dircctions defined according the robot's coordinate system.
Plans avoid collision with the table (e.g., no downward movements that might cause collision).
Step 6: Output the Configs in YAML Format
tase_ CloseSafe category_nana _nme:: Articulated
# Post-actuation motions for the manipulator. # Post-actuation motions should be in the format of python list. Each element in the list should be string with lower-case separated by hyphens. # post-actuation pose list cectuattio_motionn ["move- ["moor-forward"]
# [push door closed]
Success!
33

---

Example for Vanilla LLM Response
Input:
You expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.
Here the task description.
ntask, "close-safe", :tas--descrpption "Close the safe door", "assets-used"; ['safe rotate"], "success-criteria" l"articulated closed")
Now please first generate the constraint part for task celose-drawerf in the same config format as the above.
Output:
task CloseSafe
category name: Articulated
kool_keypoint list: [ead, tool_tail, tool_side] hiriuullaed_bbbet.hhhee headd]
constraint list:
# ensure the gripper's tool head is in contact with the safe handle keypoint name: tool head kerpointnniint.... name: name: articulated ceiiuulated.bbect.hheee.
tolerance: 0.0001
type: powntppoint.comsseeeeeee constraint
# ensure the gripper's top side is aligned with the safe handle's 's direction (assuming the handle the world's Y direction)
Visualization of Actuation Pose
axis from keypoint name: tool
axis to keypoint name: tool tail target_axis: [0, 1.0, 0]
frrgetaxiss.famee world
tolerance: 0.01
target_inner.proode pproduct:
type: frame axis saraalea
# ensure the gripper's 'ss front side is perpendicular
the world's Y direction
axis from hetn-keypoitttaatt name: tool.tai tail
axis to keypoint name: tool side
target_axis: [0, 1.0, 0]
target.axis.frame framm:: world
tolerance: 0.01
type: frame axis orthogonal
Fail to generate correct constraints!
34

---

E,4 Example Failure Cases
We provide some failure cases of different stages in this section.
Failure Case for Task Proposal
Input:
You expert in creating robotic simulation environments and tasks. You given articulated assets example. Please come up with creative use the gripper to manipulate single articulated object. Note that the simulation engine docs not support deformable objects or accurate collision models for contacts. Morcover, the robot can only execute simple trajectory of one motion.
Format the answer in a Python dictionary with keys "task-name" and value type string, "task- description" (one short phrase), and value type string with lower-case and separated by hyphens, "assets-used" and value type list of strings, and "success-criteria" (choose from "articulated open". "articulated_closedd""
"distance mistaace-griiper.c geipperrriiidbboop........ tance -eneeer articulatedd") and value type list of strings. Try to be as creative as possible.
"dis-
Output:
"lask-name": "unlock-safe". trask-description". "rotate the safe's handle to a horizontal position to simulate unlocking", "assets-used used"": ["safe rotate") "success-criteriaa" ("articulated_open"]
Fail to generate correct task!
Failure reason: The handle not rotatable.
Failure Case for Task Decomposition
Input:
are an expert in creating robotic simulation environments and tasks. You will given long- horizon tasks with each task including at least 2 sub-tasks. Each sub-task can only include one simple motion such as moving the gripper to some object, opening or closing the gripper fingers, or interacting with certain articulated objects by its prismatic/revollte joints.
Now please start to generate sub-task decomposition of the following task:
"task-name": "tid--upfoamma hup-foom-brickt
deserrpteri:::: "Pick up the foam brick, open the microwave door, place the brick inside, and close the door.".
"asscts-used": ["microwave", "foam "foam_brick"]
35

---

Output:
# Sub-task
"lask-name": "reach-foam-brick"
"task-descriptionn:: "reach the foam brick and make the foam brick between gripper fingers", "assets-used"; ["foam_brick" puccess-criteriaa" ("distance- grippeer.
# Sub-task 2
"task-name": "grasp"
# Sub-task 3
"task-name";
"task-description". : "open the microwave door by interacting with its revolute joint", "assets-used": ["microwave"]]
"success-criteria": ["articulated.openl
# Sub-task 4
{
"task-name": "place-isside-mirrovavv""""
"task-descriptionn" "move the gripper with the foam brick inside the microwave", "assets-used": ["microwave"],
"success-criteriaa: ("distance articulated rigidbody"]
# Sub-task 5
"task-name": "ungrasp" {
# Sub-task 6
{
"task-name": gelos--microwave-dooms
daakkddssriptioon.: "close microwave door by interacting with revolute joint". "asscts-used": ["microwave"],
"success-criteriaa: ["articulated closed")
Fail generate correct task decomposition!
Failure reason: Door should be open before the object is grasped.
Failure Case for Solver Creation
Input:
You are an expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.
Here is the task description.
{
"task-name": "
trakkddsscrpttion" : "open the microwave door".
36

---

"assets-used": |"microwave"l, "success-criteriaa: ["articulatcd.open""
The initial of this task is visualized in the first uploaded image. The frame axes of the gripper and the object are respectively visulized in the second and third images, where red, green blue axes represent X, Y and axes respectively.
Now please first generate constraint part task "close-drawer" in the same config format as the above.
Output:
name: OpenMicrowave category name: Articulated
kool_keypoint_nans Lww.eeiiitttaammmeeeeeeeeeee Jist: [ead, tool. toolsidee]
keypoitk name list: aurticulated.ojjett..eee
constraint Jist:
keypoint_name: tool _eead
articulated_oojjer _object_head _ead
tolerance: 0.0001
type: point2point constraint
www.ffrmm.eyooittt.c thad
keeyooot nnam:: tool.tail
target_axis: [0, 1.0]
Visualization of Actuation Pose
target_axis_frame: world
tolerance: 0.01
target larget_inner inner product:
type: frame saineeaaxissam saane__axss.othhoonaa
axis frii..from.kee aerommeeypint.naae name: head
axis.to keypoint. name: tool_side
keme::nt_namme
target.axis: [0. 1.0, 0]
target_axis_ framme: object
tolerance: 0.01
seaee_inner__roodue
type: frame axis parallel
Fail generate correct solver config!
Failure reason: The corresponding actuation pose cannot complete the task.
37


Reviews:
[{"review_id": "lqALuBQZVS", "reviewer": "Reviewer_Wb5Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes GenSim2 a LLM based task generation pipeline that can generate realistic articulated object tasks capable of sim-to-real transfer.", "summary_of_recommendation": "Weak Accept because this paper presents a sound system for end-to-end LLM task generation to sim-to-real transfer; however, the technical novelty is rather limited and many components take a lot of engineering to get to work. Furthermore, its reliance on human feedback is not highlighted enough in the paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Additional Question/Comments:\n1. In the real-world tasks, some generalization over unseen objects, scenes would be nice to demonstrate that the policies learned in simulation can have good generalization. \n\n2. Typo in line 230.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TiPvcpZwsk", "reviewer": "Reviewer_i12f", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work presents GenSim2 a framework to enable sim2real transfer. GenSim2 leverages LLM and VLMs for designing tasks in simulation, RL and motion planners to obtain demonstrations, and multi-task training, to aid generalization and transfer to real-world applications.", "summary_of_recommendation": "A convincing method for sim2real transfer, with minor presentation flaws which are expected to be solvable by the authors.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* What is the simulator used? This should be clearly stated in the text (and Figure I caption as well)\n* What is the task library adopted? How is this composed/learned?\n* List of presentation flaws:\nLine 30, GenSim2 1, GenSim2 (Figure 1)\nLine 49, 25 -> 25%\nLine 110, \"language and visual reflections\" -> I think a citation to [1] or similar work would be necessary\nParagraph from line 112 to 125, I think the bottom-up approach is very similar to [2] and a citation to it or to similar work would be necessary\nLine 196, pcd not defined (pointcloud?)\nLine 199, 4.4 -> Section 4.4\nLine 230, LLM -> the LLM\nLine 231, missing Section (??)\n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al\n\n[2] Voyager: An Open-Ended Embodied Agent with Large Language Models, Wang et al\n\n**Post rebuttal update**:\n\nApologies for the tardiness in the response. I have gone through the extensive rebuttal provided and updated my score to a \"Strong Accept\". \n\n**Additional ablation studies**: I appreciate the authors' efforts in adding this, as it helps to understand what are the sources of performance improvement in their system.\n\n**Task library**: I think a more detailed description of the task library should be included in the paper, as clarified by the authors in their response.\n\nOverall, given the additional clarifications and the extensive amount of additional content presented by the authors, I think the paper would be a nice addition to the conference, and thus I think it should be accepted.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zOKOCLJ3Jh", "reviewer": "Reviewer_9Mkg", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work (GenSim2) presents an extension of GenSim, a method for scaling robot task creation using large language models (LLMs). The extensions address increased task complexity, improved task solvers for demonstration data collection, and training a multi-task transformer model. First, the iterative top-down and bottom-up approaches enable the generation of long-horizon tasks involving articulated objects. Second, the integration of a kPAM planner with reinforcement learning (RL) policy learning enhances the efficiency and effectiveness of task-solving for demonstration data collection. Third, the proposed multi-task transformer model is trained to distill the demonstration data into a single policy, demonstrating sim-to-real transfer through the careful selection of observation inputs. The paper shows promising results in generating long-horizon tasks more effectively than previous methods, creating 50 single-step tasks and 50 multi-step tasks. Additionally, the proposed approach solves these tasks and distills multiple tasks into a single policy. Notably, the distilled multi-task policy exhibits effective sim-to-real transfer, primarily due to the use of point cloud observations, which has a small sim-to-real gap.", "summary_of_recommendation": "While the work is interesting and addresses the challenge of task generation, it is fairly incremental, and it is not entirely clear if the improvements over GenSim are significant enough to warrant a completely new paper. The tasks remain relatively simple and similar. Additionally, it is unclear how much human input is required for task generation and solving (e.g., human-specified kPAM key points), especially given that the primary motivation for using LLMs in task generation is to reduce the burden on humans in the process.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See review comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose an LLM-based task data generation system for robots, together with a 3d-based sim-to-real pipeline. They particularly focused on realistic, 6-dof robot tasks. The work itself seems like a notable improvement over several similar works in this area. Their approach does leverage human expertise when creating tasks, which has advantages and disadvantages, but is interesting compared to the other related works. They also show some ability to generate long-horizon tasks.

During the rebuttal period, they provided a very detailed followup, with information clearly comparing them to related work and showing the ability of models trained using their method to generalize, as well as adding substantial extra discussion that addressed many of the core concerns raised by the reviewers.

Strengths:
	• Paper is mostly well-written and convincing
	• Generates lots of tasks beyond just pick and place - great step towards solving data problems in robotics
	• Good sim-to-real learning pipeline
Weaknesses:
	• Need for human intervention
Generalization is greatly improved from rebuttal, but some concerns remain
Decision: Accept