GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs

Authors: pu  hua
minghuan  liu
annabella  macaluso
yunfeng  lin
weinan  zhang
huazhe  xu
lirui  wang
shanghai jiao tong university
mit  csail
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-24 11:46:34

Abstract: Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.

Main paper: 
 introduction: 
 Robot learning requires large amounts of interaction data and evaluation, which are expensive to acquire at scale in the real world. Robot simulation holds the promise of providing such data and verification in high diversity and efficiency across objects, tasks, and scenes. While the ability to simulate has led to many successes in AI across Gaming, Go, and Mathematical Proofs [2, 3, 4] , there are two requirements for such a path to be successful in robotics: The data needs to scale in complexity without significant human efforts and the data needs to be realistic enough to transfer to the real world. Previous works [5, 6, 7, 8, 9, 10, 11] have made significant progress in scalable simulation benchmarks in robotics and training policies on the simulation data.Foundation models [12] , particularly generative models pre-trained on internet-scale data [13, 14, 15] , have demonstrated impressive capabilities required for generating robot simulation tasks, such as coding [16] , spatial reasoning [17] , task semantics [9] , planning [18, 19] , video prediction [20, 21] , and cost and reward understanding [22, 23] . While foundation models have shown impressive capabilities to output actions to solve robotic tasks directly in the real world [24] , simulation provides a low-cost and scalable platform to learn robust end-to-end policies. In addition to generating numer- GenSim2 introduces a scalable task and data generation framework in SAPIEN [1] for articulation objects with multi-modal and reasoning LLMs. The framework comprises three main stages: 1) (Top) We first generate large-scale robotics tasks and collect massive data with LLMs; 2) (Bottom-left) Then, we train a multi-task point cloud-based policy in simulation with imitation learning; 3) (Bottom-right) Finally, we zero-shot transfer the policy to the real world. ous tasks, automatically solving them to produce demonstration data and transferring the learned skills to the real world present significant challenges. To address this, the data generation and policy learning pipelines must scale efficiently while minimizing the sim-to-real gaps.In this work, we propose GenSim2 (Fig. 1 ), a scalable framework that amplifies robotics data by generating diverse articulated and long-horizon simulation tasks and demonstrations. In addition to the semantic knowledge and compositional capabilities in language models, these tasks also require extended 6-dof motions with contacts. To generate demonstrations for such tasks, we develop various solvers, including a keypoint-based motion planner that generalizes across object categories and initial conditions. Moreover, we use the visual information in a multi-modal LLM (MLLM) to iteratively generate and verify novel tasks, to the extent of 100 tasks to train a multi-task policy.GenSim2 tackles more challenging articulated and long-horizon tasks beyond top-down pick-andplace [9] . These complex tasks require grounding and reasoning capabilities beyond previous LLM techniques. Therefore, GenSim2 is an agent pipeline featuring multi-modal foundation models (e.g. GPT-4V) and reasoning models (e.g. OpenAI o1) as well as generalized keypoint planners for such complex tasks. To fully utilize the generated data, we further propose a sim-to-real native policy architecture, dubbed proprioceptive point-cloud transformer (PPT), with point-cloud observations and language conditioning. This effective architecture is designed for sim-to-real transfer with the large-scale data generated in simulation.In our experiments, we have generated over 100 articulated object manipulation tasks with 35 objects (over 200 instances), including 50 long-horizon tasks. Our task generation pipeline achieves 25% better success rates than previous works. After using an automated pipeline to solve these tasks and generate task demonstrations, our multi-task policy can jointly solve 24 tasks with random scene configurations and only suffer a 3% performance drop when tested on unseen object instances. More importantly, by integrating the GenSim2 pipeline with the designed policy architecture, we achieve promising results across 8 real-world articulated object tasks, through such as zero-shot transfer and co-training with real-world data. This approach leads to a 20% improvement in policy performance compared to training exclusively on limited teleoperation data.In summary, we show a promising way to reduce data collection efforts and solve real-world problems through massive simulated data generation with MLLMs. Our contributions are as follows: (1) Multi-Task Policy Learning with 3D Information. With the surge of robotic data, recent works have explored multi-task policy learning, usually conditioned on language inputs. In particular, policy learning often benefits from access to explicit 3D information when doing generalized 6-DOF tasks [46, 47] . [48] uses RGB-D inputs and output pixel-level affordance map. [46] uses voxel as policy input information and output keyframe actions. [47, 49] uses point cloud as inputs to the policies. [50] uses keypoints at the corners to manipulate boxes with dexterous hands. In our work, we focus on point-cloud transformers as multi-task policies for articulated object manipulation.Sim-to-Real Transfer. While simulation provides scalable training data and evaluation runs, simto-real transfer [51, 52] is a challenge and an active research area. RL-based controllers [53] leverage simulation environments for large-scale interactions to learn robust behaviors. Previous works have also explored techniques such as domain randomizations [54, 34] , domain adaptations [55] , and policy composition [56] to improve sim-to-real transfer. Our work can be viewed as a distillation process from the knowledge in foundation models such as MLLMs [14, 13] , into policies, which can have a smaller sim-to-real "semantic" gap compared to rule-based simulation data. 
generating tasks and training policies at scale: 
 The proposed GenSim2 framework executes a series of processes including task proposal (3.1), demonstration generation (3.2), and policy learning and transfer (3.3) . We illustrate the pipeline in Fig. 2 . Our work enhances GenSim [9] at the level of task complexity beyond top-down pick-andplace, which necessitates grounded multi-modal task designs, generic 6-dof motion planners and RL solvers, and scalable policy architectures.Primitive Task. Our pipeline begins with proposing primitive tasks, namely tasks with a single simple motion (e.g. opening a box), by prompting an LLM to generate a novel task, defined by its task description (a short phrase or sentence), used assets and task code. We start from a fixed asset library and a small task library initialized with hand-designed example tasks. We parse them into the prompts for in-context learning. We query the LLM to discover applicable assets for creating a new task. The LLM then takes in as input the generated task definition and outputs the corresponding task code. This code is finally compiled and ready for demonstration generation in Section 3.2.Long-horizon Task. Moreover, we extend our pipeline to generate long-horizon tasks that consist of multiple steps of primitive tasks and involve manipulating articulated and rigid-body objects. Examples include opening a box, placing a ball inside, and then closing the box. When generating a long-horizon task, we execute a task decomposition process between task proposal and code implementation, to decompose the long-horizon task into several sub-tasks (primitive tasks). We use two distinct methods for generating long-horizon tasks: (1) Top-down: We directly generate a task in a curriculum, then decompose the task into several sub-tasks, each with a dedicated solver.(2) Bottom-up: We first generate primitive tasks and build up a task library. Then the LLM will be prompted to select tasks from the pre-built task library to compose a new task. We have observed reasoning LLM to improve task proposal performance in this stage. After we have finished the task planning, we continue to generate solvers and demonstrations for the subtasks in succession. Fig. 9 demonstrates visualizations of some generated long-horizon tasks.In this section, our goal is to create a solver to generate demonstrations given a task (or sub-task) code. The objective of our generation pipeline is to collect large-scale, high-quality data for general 6-DOF manipulation task learning in the real-world setting, our task solver should meet the following requirements: 1) Universal for 6-DOF tasks without task-specific designs; 2) Robust to different scene configurations; 3) Fast to execute; 4) Deterministic with high success rates. While such a solver is easy to design for top-down pick-and-place, in which we only need to specify some 2D position on the table as waypoints and execute primitive "pick" and "place" actions, it becomes challenging for more complex tasks such as general articulated object manipulation.To address this challenge, we propose and investigate the planner config generation with a multimodal LLM [14] . We also propose two types of task solvers: a kPAM-based motion planner and a complementary RL-based policy learner (see in Appendix A.3), to solve the generated tasks.Motion Planner. kPAM [57] is a keypoint-based planner proposed for category-level manipulation tasks. It defines a robotic task by an actuation pose, namely the homogeneous transformation required to manipulate the target object, and it addresses an optimization problem based on several keypoint-based constraints to get this pose. Following Wang et al. [33] , we improve and extend kPAM to produce an object-centric trajectory of end-effector poses, termed actuation motions, to solve a generated task. We parse the constraints and actuation motions into parameterized configurations that are easily interpretable and codable. We provide an illustration in Fig. 7 to show how the kPAM planner solves a task. Multi-modal Task Solver Generation. Compared to previous works that primarily utilize language prompts, our approach incorporates visual information and key points [58, 59, 60] , serving as an explicit representation of scene details (e.g., object structures, affordance and spatial information). The pipeline of solver generation is depicted in Fig. 7 . First, we execute the generated code and capture scene images. This visual data, together with object key points, are then used to prompt an MLLM to generate the planner constraints. Subsequently, we visualize the actuation pose defined by these constraints. Finally, we query the MLLM to generate the actuation motions. Optionally, we also introduce reject sampling to guide the MLLM in refining its previous outputs.Multi-Task Training. We design a multi-task policy structure, denoted Proprioception Point cloud Transformer (PPT), as illustrated in Fig. 4 . Specifically, we handle three kinds of observations: point cloud, proprioceptive states, and language task description, all of which can be obtained from the real world. Each observation is separately tokenized via respective encoders and cross-attention, fused together in the shared latent space through transformer blocks, and post-processed into global condition tokens [61] . Given the global condition tokens, the policy predicts a sequence of actions through the policy head. In our implementation, we test against various policy heads such as MLPs, the transformer decoder [62] , and the diffusion model [63] . See Appendix B.2 for more details.Sim-to-Real Transfer. To transfer between simulation and real environments, we ignore the color information of the point cloud, and augment the point cloud observations of simulated demonstrations during training. This augmentation involves cropping, adding Gaussian noise as perturbations, and randomly dropping points. In the real-world setting, we process the actual point cloud using uniform sampling, farthest point sampling, and outlier removal. This processed clean point cloud serves as input to the multi-task policy during inference. 
experiment: task and data generation: 
 In this section, we aim to verify the feasibility of the task generation framework and investigate the effectiveness of the task and data generation pipeline.Evaluation Criteria. We evaluate the task generation procedure mainly on two types of success rates: 1) execution rate, the success rate of completing the whole pipeline without any error such as syntax or runtime error; 2) solution rate, the success rate of solving the generated task.Ablation Study. In this section, we conduct a careful ablation study on each component of GenSim2. The results, depicted in Fig. 13 , are summarized as follows:Types of LLMs: We test multi-modal LLM, reasoning LLM, and vanilla LLM on solution rate in Fig. 5 left. Without visual data as input, the performance of the pipeline deteriorates with signifi- Ablation study on components of our generation pipeline. All results are based on no less than 10 generations. Left) We use various types of LLMs for solver generation and find that multi-modal LLM (GPT-4V) outperforms the others. Middle) We find that the performance of our method will increase, as we increase the maximum iteration for reject sampling. Right) We observe that splitting solver generation into a prompt chain surpasses generating the whole solver config. cantly lower success rates in task generation under both criteria. The performance drops because the constraints for actuation pose can not be correctly generated as it requires details for object structure. Meanwhile, the actuation motions also suffer from large errors because the vanilla LLM has no access to spatial relationships between objects. This highlights the crucial role of visual information in the success of GenSim2, as it provides essential details about the scene such as spatial relationships and the appearance of objects. Moreover, reasoning capability also improves vanilla LLM's score on solver generation. We hypothesize that the reasoning trace of "think" before coding, based on the task and config structure can reduce hallucinations in generating kPAM constraints. See Appendix E.2 and E.3 for more details.Multi-shot Reject Sampling: In Fig. 5 middle, we have visualized the solution rates of our pipeline under different maximum iterations of multi-shot rejection sampling. The results indicate that our pipeline with multi-shot rejection sampling surpasses the version without it, highlighting the importance of self-reflection in LLMs. Additionally, allowing LLMs more opportunities to refine their answers through iterations leads to improved performance. We also explore the use of rejection sampling from GPT-4V, which takes solver config visualizations as input and determines their effectiveness, in our pipeline. Due to the domain gaps of visual perception and understanding capabilities of MLLMs, particularly in processing 3D robotic scenes ( [64, 65] ), the MLLM can generate irrelevant responses. More powerful models and advanced prompting and finetuning methods are likely to improve these in future work.Chain-of-thought Prompt Design: We observe that dividing solver generation into a prompt chain (first generating constraints and then actuation motions) yields better results (over 30%) than directly outputting the complete solver at once in Fig. 5 right. This approach allows for focused, sequential processing of components, leveraging outputs from earlier stages for subsequent ones.Comparison with Baseline. We compare the pipeline of GenSim2 using a kPAM motion planner with another open-sourced framework RoboGen [10] , which encompasses a large number of longhorizon articulated tasks. For primitive tasks, we consider the sub-tasks within a long-horizon task Table 1 . Comparison with RoboGen on task generation success rates. GenSim2-B represents the bottom-up long-horizon task proposal method, while GenSim2-T and GenSim2-T (o1) represent top-down methods using GPT-4 and OpenAI-o1 respectively. in RoboGen for comparison. For long-horizon tasks, we compare our top-down methods (labeled as GenSim2-T and GenSim2-T (o1), utilizing GPT-4 and OpenAI-o1 as task decomposers, respectively) and bottom-up method (GenSim2-B) with RoboGen's long-horizon tasks.The results, presented in Tab. 1, indicate that our method exceeds the baseline in both primitive and long-horizon task settings. The underlying reason is our approach's implementation of fine-grained motion planner generation, whereas RoboGen directly generates a task-specific reward function and relies on reinforcement learning for task training, which can be inherently more fragile. Additionally, the bottom-up approach surpasses the top-down approach in both metrics. This advantage stems from the former's reliance on composing existing tasks, where most failures occur during the chaining of sub-tasks, whereas the latter necessitates generating each sub-task from scratch. Furthermore, the enhanced reasoning capabilities of OpenAI o1 contribute to more logical task decompositions compared to vanilla GPT-4, resulting in superior performance across both evaluation criteria. 
experiment: multi-task policy training and transfer: 
 In this section, we show how the generated data by GenSim2 can be used by multi-task imitation learning and sim-to-real transfer.Training. With the tasks and data generated by our pipeline, we train a multi-task policy (with 382M parameters) across different numbers of tasks and test its generalization to new scenarios. In Fig. 6 left, we jointly train a different number of LLM-generated tasks and test on 4 original tasks under the low data regime such as 10 demos per task. Interestingly, adding more tasks will first drop the performance and then increase it by virtue of scaling. Please refer to Appendix B for more details.After training, we test the generalization ability of our policy to unseen object instances, compared with a policy with RGB inputs. We split the instances into train/test sets. Evaluated results are shown in Fig. 6 right, where we find that the success rates of the PPT architectures only drop by less than 3% on unseen instances, whereas the RGB policies reduce by quite a bit. These results indicate that by generating data with object-level and spatial-level variance as domain randomization, along with the pointcloud-based policy architecture design, the trained policy can acquire generalization in both aspects, which lays the foundation for further sim-to-real transfer. We observe that the multi-task policy, trained on simulation data, can perform robustly on various unseen real-world objects through sim-to-real transfer.Table 2 . Evaluation on a set of real-world tasks using one multi-task policy. We report the per-task success rate of 10 episode evaluations with 8 real-world tasks. Sim-only represents policy trained purely with 100 simulation data generated by GenSim2. Real-only represents policy trained with 10 real-world data collected by teleoperation. Combined represents policy co-trained with combined simulation and real-world data. To evaluate the quality of the data collected in simulation and how GenSim2 helps in real-robot tasks, we test sim-to-real and co-training on the generated data over several tasks. As for the setup, we use the Franka Research 3 robot arm with a modified, deformable TPU parallel gripper for easier grasping. The robot work cell is equipped with three RealSense D435 cameras: one wrist-mounted and two externally facing the scene. Each captures an RGB-D observation which is combined and processed into a point cloud to be used as observations. More details are listed in Appendix C.We perform experiments on 8 real-world tasks, collecting 100 demonstrations for each task in simulation, along with an additional 10 real-world demonstrations via teleoperation. We assess the quality and utility of the generated data across three distinct training setups: (1) using only simulation data, (2) using only real-world data, and (3) using a combination of both. The results of these evaluations are presented in Tab. 2. Our findings can be summarized as follows: (a) data generated by GenSim2 enables effective zero-shot sim-to-real transfer, with the resulting policy outperforming one trained solely on limited real-world data; and (b) when co-trained with real-world data, the data generated by GenSim2 significantly enhances the policy performance even by 20% in absolute values and 50% in relative scale. These results underscore the potential of large-scale, high-quality data generation, like GenSim2, to reduce the burden of extensive real-world data collection while improving policy effectiveness for real-world tasks. 
limitations and discussions: 
 Our proposed method has several limitations. Due to the lack of "robotic centric" knowledge, such as 3D spatial understanding, foundation models like GPT-4V still face hallucination issues in creating meaningful tasks and successfully coding them. Additionally, human involvement, though minimal, is still required to generate these complex manipulation tasks. Finally, we have only considered 6-dof tasks in zero-shot sim-to-real transfer with limited point cloud observations. 
conclusion and future work: 
 In this work, we propose GenSim2, a task and demonstration generation framework that utilizes multi-modal foundation models to generate up to 100 robotic simulation tasks at scale, such as longhorizon manipulation with articulated objects. We ablate on different task solvers and generation components, as well as propose a multi-task pointcloud-based policy architecture that distills generated demonstrations from simulation transfer to the real world. Future works include expanding task complexity and diversity through advanced multi-modal agents and 3D asset generation. Future works can explore advanced sim-to-real methods for complex tasks with multiple embodiments. 
 
 Appendix: We summarize the information of the articulated asset library we used in this project in Table 4 . To ease the workload of the LLM in understanding the manipulatable parts of an Articulation, we only reserve one unfixed (prismatic or revolute) joint in each object and fix the others. For rigid body objects, we build a dataset adapted from the YCB object dataset in Maniskill2. We have modified the scale and joint limits of the objects to make the scene proper.In this section, we investigate the statistics of the tasks in the generation process. We have generated 100 tasks based on the asset library described in Appendix A.1, including 50 primitive tasks and 50 long-horizon tasks. Notice that we consider tasks with the same object class but different instances as the same tasks. Despite the tasks we have generated, these tasks do not include all assets in the asset library and the asset library can also be expanded to contain more object classes and instances.A.3 Task Solver Details kPAM Planner. kPAM [57] is a keypoint-based planner proposed for category-level manipulation tasks, and we follow the roadmap proposed in Wang et al. [33] to improve kPAM and apply it in our articulation tasks. It generally defines a robotic task by the homogeneous transformation required to manipulate the target object. kPAM discovers such transformation by solving an optimization problem built on several keypoint-based constraints. After kPAM implicitly defines the end-effector pose to get contact with the object, which we call the actuation pose, we need to build the trajectory to approach the actuation pose and subsequently complete the task. To this end, pre-actuation and post-actuation motions are introduced, both of which include a series of waypoints to guide the endeffector motions (such as making a turn or pushing by moving forward). Together with the actuation pose, an object-centric trajectory of key poses is designed to solve an articulation manipulation task.We provide an illustration of leveraging kPAM planner to solve a task in Fig. 8 .RL Learner. In addition to planners, we also leverage a reinforcement learning (RL) learner [10] as an alternative solver for the generated manipulation tasks, in which we prompt an LLM to produce a reward function for a target task. After the task proposal stage, we're provided a task definition and task code generated by the LLM which is used as context for a LLM to generate an executable reward function. The LLM is supplemented with a small library of curated reward examples as well as an API consisting of reward functions corresponding to various predefined components that it can refer to, such as the distance between the end-effector and the object, or the joint positions of certain articulated objects. If successful generation is achieved, we append the reward into the task code and execute the final learning code using Proximal Policy Optimization (PPO) [66] and fix a default set of hyperparameters for all tasks.In summary, we propose the kPAM motion planner and RL policy learner as complementary methods for solving generated tasks. For kPAM, since it is defined by a well-formulated optimization problem regarding fine-grained constraints and costs, the output motions are much smoother and more natural than motions from other learning-based methods. Moreover, kPAM is inherently generalizable to different scene configurations and object instances because the constraints and actuation motions are all "object-centric" elements, which will be further exploited for domain randomization in Section 3.3. Furthermore, kPAM is fast to execute (∼ 2 seconds planning time given a configuration) and robust to multiple runs. Despite the strength of kPAM, RL provides advantages in creative solution proposals for tasks that are too complex or ambiguous for kPAM to define proper constraints, especially for those involving thin objects or contact-rich manipulation. In this project, we give priority to the kPAM planner in most cases because it produces more realistic and generalizable motions, and leaves those that can not be solved for the RL learner. . Illustration of Generated Long-horizon Tasks. We show two example tasks generated by our method, including securing gold in a safe (top) and preparing breakfast (bottom). The former contains 4 sub-tasks and is solved by chaining motion planners of all sub-tasks. The latter contains 7 sub-tasks and is teleoperated due to its complexity.We measure the human efforts required in GenSim2, including labeling keypoints for the motion planner and multi-shot rejection sampling in task solver generation.Keypoint Labeling. We set up a protocol for keypoint labeling and annotated 1-3 keypoints for each asset. We measured the time spent on 10 objects and the average time is 8.2s per task.Rejection Sampling. While task solver generation inevitably involves some degree of human intervention, we have made significant efforts to minimize this effort. To evaluate the efficiency of our approach, we conducted a user study measuring the time required for a human user to generate a novel task and corresponding solver using GenSim2, and compared it with the time needed for manual task design. The results are presented in Tab. 3. We consider the performance of one of the project's authors (User 1) as expert proficiency and additionally involve two other users who are unfamiliar with our pipeline. The findings indicate that our method significantly reduces task creation time by 50%, and over 90% of the required time involves waiting for LLM to response. Furthermore, the reduced performance gap between experts and beginners suggests that our pipeline requires less technical expertise, making it accessible to users with varying levels of familiarity. In this section, we assess the efficiency and quality of demonstration generation using our kPAMbased motion planner. Recalling our goal to use our generation pipeline as a data amplifier to enhance the generalization and sim-to-real transfer ability of a simulation policy, the generated data should be diverse and of high quality for data augmentation or domain randomization.To this end, we evaluate the robustness of our kPAM solver to random scene configurations and object instances. For primitive tasks, we add uniform noise in [-0.1m, 0.1m] to the x and y positions of the object, as well as a random rotation in [-30 • , 30 • ] to the yaw axis. For some hard long-horizon tasks, in which the articulation is so small that it may be struggling to place something inside, we fail to generate successful demonstrations with normal randomization, so we reduce the scale of noise by half. For articulations, we randomly select an instance adapted from PartNet-Mobility and add noise to its default joint position.We test the success rates of generating demonstrations for 24 single-step and 15 long-horizon tasks, and report them task by task in Tab. 5. For each task, we run the demo collection process 5 times, and each time the kPAM solver is executed for 50 episodes with both spatial and object randomization. We compute the mean and standard deviation of the 5 runs on the success rate. We gradually add the number of training tasks from 4 to 24 in Section 5.1 and test the scaling ability of our framework on the original 4 tasks. We provide a detailed task list here to better clarify our task settings:• 4 tasks: OpenBox, CloseBox, OpenLaptop, CloseLaptop;• 10 tasks: 4 tasks + OpenDrawer, PushDrawerClose, SwingBucketHandle, LiftBucketUpright, PressToasterLever, PushToasterForward;• 15 tasks: 10 tasks + MoveBagForward, OpenSafe, CloseSafe, RotateMicrowaveDoor, CloseMicrowave;• 20 tasks: 15 tasks + CloseSuitcaseLid, SwingSuitcaseLidOpen, RelocateSuitcase, TurnOnFaucet, TurnOffFaucet;• 24 tasks: 20 tasks + SwingDoorOpen, ToggleDoorClose, CloseRefrigeratorDoor, OpenRefriger-atorDoor.We implemented a multi-modal policy architecture, as illustrated in Fig. 4 , which includes a transformer policy stem, an action head, and different encoders for modeling various types of observations as tokens.Specifically, we take the CLIP [67] tokenizer to encode the task instruction, which is frozen during training; a pre-trained PointNext [68] to encode the point cloud, which is fine-tuned during training; an MLP for encoding the proprioception states, which is trained from scratch. The transformer conducts self-attention over all tokens, and then we post-process the tokens for different action heads. For example, we compute the mean pooling of all the tokens as the global condition of the diffusion and MLP head; for the transformer, we compute cross-attention between the modeled token and a set of position embeddings to get the final action sequence. Regarding the PointNext encoder, we utilize the pre-trained model on ScanObjectNN Classification1 , as it does not include color information, making it easier for downstream sim-to-real transfer. Note that the diffusion head and the transformer head both model and predict action sequences, but the MLP head only works for single-step action.We first evaluate the Proprioception Pointcloud Transformer (PPT) in solving multiple tasks. We train one policy with demonstrations of 10 tasks on the commonly used manipulation benchmark, RLBench [32] . In particular, we train PPT for 250 epochs on data collected using the same setup as in Yan et al. [69] . Results shown in Tab. 6 performance best performances over 6/10 tasks improved performance over recent competitive baseline methods such as PerAct [46] and GNFactor [70] with a significant reduction in required camera views and pre-trained feature representations. Note that GNFactor [70] requires rather hassles to set up for real-world experiments, as it requires getting features from the neural radiance field [71] . On the contrary, our proposed PPT architecture only requires point cloud, languages, and robot sensor states, which are much easier to obtain with realworld robots.After multi-task training, we test the generalization ability of our policy on unseen object instances. We chose tasks using assets with more than 20 instances and 10 instances to respectively construct Table 6 . Multi-Task Performance on RLBench. We evaluate 25 episodes for each checkpoint on 10 tasks across 3 seeds and report the success rates (%) of the final checkpoints. Our method outperforms the most competitive baseline PerAct [46] and GNFactor [70] We find that the success rates only drop by around 3% on unseen instances, as shown in Fig. 10 . These results indicate that by generating data with object-level and spatial-level variance as domain randomization, our policy can acquire generalization in both aspects, which lays the foundation for further sim-to-real transfer. As we didn't utilize a discrete action space such as keyframes for real-world experiments, realtime control was important. We achieved fast inference speeds with our multi-task policy with 0.1s inference latency on an NVIDIA 3080 GPU. We parallelized and synchronized point cloud processing from 3 Intel Realsense D435 cameras in order to prevent additional latency. Please view Fig. 11 to see the robot workcell described. Processing the point clouds involved a uniform sampling In addition, we also evaluate how well the multi-task policy could generalize to different objects within each task. We run an additional set of experiments for the OpenBox, CloseDrawer, and OpenLaptop tasks, on varying objects. Thanks to the training scale in simulation and the design of our PPT policy architecture, our method can achieve over 60% success rate on multiple tasks with various unseen objects, as shown in Tab. 7. In addition, the performances on different objects within the same task are comparable, demonstrating the robust real-world generalization achieved by our policy. To conduct additional experiments on different multi-task policies incorporating real-world data we incorporated a teleoperation setup using an HTC Vive controller and base station to track 6-DOF hand movements which were mapped to control the robot arm's pose. To interface with the HTC Vive controller we used the (triad-openvr package) and SteamVR. We implemented a position-velocity controller for the robot to track given poses from the controller. Using this teleoperation system, we collected 10 demonstrations for each generated task, collecting state and observation data for each trajectory. We co-trained a multi-task policy using 100 episodes of simulation data per task and the 10 collected demonstrations of real data per task.Different from previous efforts including GenSim (top-down manipulation tasks), RoboGen (dynamic tasks), and Robocasa (mobile manipulation tasks with embodiments), GenSim2 focuses on 6-DOF realistic tasks that allow task and high-quality data generation at scale, which really Zero-shot/co-train Zero-shot/finetune None Co-train benefits solving real-world tasks. However, in contrast to GenSim2, GenSim is different from the task level, which primarily deals with simpler top-down pick-place tasks; RoboGen is different from the demonstration generation method, which leverages RL for manipulation tasks and thus produces unrealistic and untransferable data for real-world tasks; RoboCasa is different from the pipeline, which requires large amounts of human teleoperation to collect source data. Moreover, our experiments on taking advantage of generated simulation data for real-world transfer show that we achieve a success rate of 42.5% for zero-shot sim2real transfer and 57.5% for transfer via co-training with both simulation and real data, while RoboCasa, which conducts similar sim2real experiments as ours, only achieves score lower than 25% by co-training. We summarize features of generated data of the aforementioned previous works in Tab. 8.Furthermore, our task and data generation method features multimodal LLMs as well as a generalizable planner to solve such complex tasks. In our experiment, we verify that the multi-modal task solver generation in this system can greatly improve the task generation success rates by 60%, outperforming previous work by 20%, and the planner used is also friendly and convenient for data generation.In this section, we demonstrate some examples of the prompt templates we have used to query LLM. The prompt templates are shown as follows:Prompt: Task ProposalYou are an expert in creating robotic simulation environments and tasks. You are given some articulated assets for example. Please come up with a creative use of the gripper to manipulate a single Articulation. Note that the simulation engine does not support deformable objects or accurate collision models for contacts. Moreover, the robot can only execute a simple trajectory of one motion. =========================== Here are all the assets. Please try to come up with tasks using only these assets. ... =========================== Here are some examples of good tasks. Try to learn from these structures but avoid overlapping with them. ... =========================== Here are some bad example task instances with reasons. Try to avoid generating such tasks. ... reasons: ... =========================== Please describe a NEW task in natural languages and explain its novelty and challenges. Note:-Do not use assets that are not in the list above.-Do not repeat the tasks similar to the good examples or the already generated tasks.-The task needs to obey physics and remain feasible.-Do not create similar tasks with the same "assets-used" set.-All the assets are on the table when initialized.-Do not place objects on small objects.-Only one Articulation can be loaded.-The task contains a simple trajectory of one motion.-The task should have a clear goal, e.g. use "open/close" instead of "adjust position".Before the next step, please check if the generated task is a bad task shown in the above examples and meets all the criteria as stated above. Specifically, if the task **only** contains a simple trajectory of one motion, and should have a **clear** goal. Explain in detail, and get a conclusion. If the task is a bad task, regenerate a new one.Then, format the answer in a Python dictionary with keys "task-name" and value type string, "taskdescription" (one short phrase), and value type string with lower-case and separated by hyphens, "assets-used" and value type list of strings, and "success-criteria" (choose from "articulated open", "articulated closed", "distance articulated rigidbody", "distance gripper rigidbody", and "distance gripper articulated") and value type list of strings. Try to be as creative as possible.Please remember not to add any extra comments to the Python dictionary. Let's think step by step.You are an expert in creating robotic simulation environments and tasks. A robot arm with a 2finger gripper is used in all the robotic simulation environments and tasks. In each task, there is exactly one Articulation and one rigid body object that you can manipulate. You will be given long-horizon tasks with each task including at least 2 sub-tasks. Each sub-task can only include one simple motion such as moving the gripper to some object, opening or closing the gripper fingers, or interacting with certain Articulations by its prismatic/revolute joints.Please come up with a decomposition of the given long-horizon task to get several sub-tasks. Some rules of such decomposition are listed here:1. Each long-horizon task can not include over 5 sub-tasks, and usually 3-4 are enough.2. Each sub-task should only include one simple motion as mentioned.3. Each sub-task(except "grasp" and "ungrasp") should be presented in the format of a Python dictionary with keys "task-name" and value type string with lower-case and separated by hyphens, "task-description" (one specific sentence) and value type string, "assets-used" and value type list with necessary asset(s) in the current sub-task, and "success-criteria" (choose from "articulated open", "articulated closed", "distance articulated rigidbody", "distance gripper rigidbody", and "distance gripper articulated") and value type list of strings. 4. Each sub-task should have only one asset used in the task. 5. If the motion of opening or closing the gripper fingers is included in the whole task, it should be listed as a separate sub-task, whose "task-name" should strictly be "grasp" or "ungrasp" respectively and should be the only key in the dictionary. =========================== Here is an example of the decomposition of the following long-horizon task ". Remember to import used functions from the corresponding files as the example task codes. For the Articulation, use "self.articulator" to refer to it, which should be the same as the "assetsused" of the task.For the objects used, you only have to pass the corresponding parameter (e.g, articulator) and its name (string format, e.g., 'box') in the ' init ' function as shown in above codes, and the base class will automatically load them. Please comment on the code to explain what each piece does and why it's written that way. Now write the code for the task "TASK NAME TEMPLATE" in the Python code block.Prompt: kPAM Solver Stage 1You are an expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.The task solution config contains the necessary positions, parameters, and keypoints for an existing trajectory optimization algorithm to solve a feasible solution. It mainly contains two parts, constraints and pre/post-actuation motions:(1) The constraints are used to ensure the gripper is in contact with the object and to implicitly define a certain actuation pose.(2) The pre-actuation motions are used to move the gripper to the actuation pose, while the postactuation motions are used to complete the task after the actuation pose.Here is the task description.Here are all the available keypoint names for the used manipulator and asset and their descriptions. ... =========================== Here are some examples of the constraint part of some configs. ... =========================== Note that, in the constraint list, you need to define different items of constraint to define an actuation pose for the task. There are some pre-defined types of constraints you can use:(1) point2point constraint: This constraint is used to ensure two keypoints ("keypoint name" and "target keypoint name", respectively on the tool and object) are in contact.(2) frame axis parallel: This constraint is used to ensure two axes (respectively on the tool and object) are parallel. The axis on the tool is defined by a unit vector from "axis from keypoint name" to "axis to keypoint name", while the axis on the object is defined by "target axis"([1,0,0] or [0,1,0] or [0,0,1]) which is in the coordinate frame of "target axis frame"(world or object).(3) frame axis orthogonal: This constraint is used to ensure two axes (respectively on the tool and object) are orthogonal. The axis on the tool is defined by a unit vector from "axis from keypoint name" to "axis to keypoint name", while the axis on the object is defined by "target axis"([1,0,0] or [0,1,0] or [0,0,1]) which is in the coordinate frame of "target axis frame"(world or object). ( 4) keypoint axis parallel: This constraint is used to ensure two axes (respectively on the tool and object) are parallel. The axis on the tool is defined by a unit vector from "axis from keypoint name" to "axis to keypoint name", while the axis on the object is defined by another unit vector from "target axis from keypoint name" to "target axis to keypoint name".(5) keypoint axis orthogonal: This constraint is used to ensure two axes (respectively on the tool and object) are orthogonal. The axis on the tool is defined by a unit vector from "axis from keypoint name" to "axis to keypoint name", while the axis on the object is defined by another unit vector from "target axis from keypoint name" to "target axis to keypoint name". The tolerance is used to define the tolerance of the constraint.The target inner product is used to define the inner product between the two axes. For example, if you want to ensure two axes are parallel and in the same direction, you can set the target inner product to 1.0. If you want to ensure two axes are parallel and of opposite directions, you can set the target inner product to -1.0. If you want to ensure two axes to be orthogonal, you can set the target inner product to 0.0.Usually, you need to define one point2point constraint to ensure contact and several axis constraints to adjust the actuation pose. =========================== Now please first generate the constraint part for task "TASK NAME TEMPLATE" in the same config format as the above. Do not use terms that you have not seen before. The output should be in the YAML format with no extra text.Prompt: kPAM Solver Stage 2You are an expert in solving robotic tasks by providing some motion plans and coding task solution configs for a 2-finger robot arm. Now please solve the newly generated task by generating the task solution config.The task solution config contains the necessary positions, parameters, and keypoints for an existing trajectory optimization algorithm to solve a feasible solution. It mainly contains two parts, constraints and pre/post-actuation motions:(1) The constraints are used to ensure the gripper is in contact with the object and to implicitly define a certain actuation pose. An actuation pose means the key frame that the robot arm manipulates the object, usually representing the moment when the gripper gets contact with the object.(2) The pre-actuation motions are used to move the gripper to the actuation pose, while the postactuation motions are used to complete the task after the actuation pose. =========================== Here is the task description. ... =========================== Here are all the available keypoint names for the used manipulator and asset and their descriptions. ... ===========================Here are some examples of the pre/post-actuation part of some task solution configs. ... =========================== Now please generate SOLVER TRIALS different pre/post-actuation motions for task "TASK NAME TEMPLATE" following the same config format shown above based on the constraint part that is generated previously. The pre/post-actuation motions of different solutions can be diverse, but their task names should be the same. Do not use terms that you have not seen before. The output should be in the YAML format with no extra text. The diversity of the motions can be achieved by using different axes for translation. Notice that the pre-actuation and post-actuation motions are relative to the actuation pose and the translation motions are represented in coordinates relative to the manipulator base.Let's think step by step, and try your best to understand the job.We provide an example conversation when generating a solver config for the task "close-drawer" with a multi-modal LLM, and compare it with the response generated by a vanilla LLM. Some of the prompt details are omitted, and please refer to Appendix E for the complete prompts.Input:You are an expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.......Here is the task description. { "task-name": "close-drawer", "task-description": "push a drawer to a closed position", "assets-used": ["drawer"], "success-criteria": ["articulated closed"] } =========================== ...... Success!We provide an example conversation when generating a solver config for the task "close-safe" with a reasoning LLM, and compare it with the response generated by a vanilla LLM. Some of the prompt details are omitted, and please refer to Appendix E for the complete prompts. We highlight some limitations of applying vision language foundation models to robotic simulation today, but we expect further progress, such as finetuning APIs, would be able to correct some of these failure modes.Input:You are an expert in solving robotic tasks by coding task solution configs. Now please solve the newly generated task by generating the task solution config.......Here is the task description. { "task-name": "close-safe", "task-description": "Close the safe door", "assets-used": ["safe rotate"], "success-criteria": ["articulated closed"] } =========================== ...... Now please first generate the constraint part for task "close-safe" in the same config format as the above.Step 1: Understand the Task and Keypoints -Task: Close the safe door.-Manipulator Keypoints:-tool head: Middle point between the two finger tips.-tool tail: Middle point between the two finger bases.-tool side: Tip of the right finger. -Articulated Object Keypoints:-articulated object head: Middle point of the safe handle.-articulated object inside base: Point inside the safe to drop objects.Step 2: Plan Feasible Constraints We need to define constraints that can be satisfied and are different from the previous attempts. Constraints typically include: 1. Point-to-Point Constraint: Ensure contact between a keypoint on the tool and a keypoint on the object.2. Axis Constraints: Orient the tool relative to the object to facilitate the action. These configurations are different from the previous ones and aim to provide feasible solutions for closing the safe door.Input:...... Now please generate 3 different pre-actuation motions for task "close-safe" following the same config format shown above based on the constraint part that is generated previously. The pre-actuation motions of different solutions can be diverse, but their task names should be the same........ Input:...... Now please generate 3 different post-actuation motions for task "close-safe" following the same config format shown above based on the constraint part that is generated previously. The post-actuation motions of different solutions can be diverse, but their task names should be the same.......Step 1: Understand the Task We need to generate three different post-actuation motion plans for the task"close-safe". The postactuation motions are the actions the manipulator performs after reaching the actuation pose to complete the task of closing the safe door.Step 2: Consider Possible Motions Since the goal is to close the safe door, we need to decide how the manipulator can move to achieve this, using only the allowed motion set:-Option 1: Push the door forward to close it.Step 3: Define the Motions According to the Robot's Coordinate System -Forward Direction: Along the red axis; useful for pushing actions.-Backward Direction: Opposite to the red axis; useful for pulling actions.-Left Direction: Along the green axis; can be used to swing the door closed if it hinges that way.-Right Direction: Opposite to the green axis.-Up/Down Directions: Along the blue axis; may not be as relevant for closing a door but can be included for completeness.Step 4: Generate Motion Plans Config 1: Push the door closed by moving forward.-Post-actuation motions: ["move-forward"]Step 5: Ensure Compliance with Constraints -Each motion plan uses only the allowed motions.-Motions are within the 3-step limit.-Directions are defined according to the robot's coordinate system.-Plans avoid collision with the table (e.g., no downward movements that might cause a collision).Step 6: Output the Configs in YAML Format Success!We provide some failure cases of different stages in this section.Failure Case for Task ProposalYou are an expert in creating robotic simulation environments and tasks. You are given some articulated assets for example. Please come up with creative use of the gripper to manipulate a single articulated object. Note that the simulation engine does not support deformable objects or accurate collision models for contacts. Moreover, the robot can only execute a simple trajectory of one motion....... Format the answer in a Python dictionary with keys "task-name" and value type string, "taskdescription" (one short phrase), and value type string with lower-case and separated by hyphens, "assets-used" and value type list of strings, and "success-criteria" (choose from "articulated open", "articulated closed", "distance articulated rigidbody", "distance gripper rigidbody", and "distance gripper articulated") and value type list of strings. Try to be as creative as possible............."task-name": "unlock-safe", "task-description": "rotate the safe's handle to a horizontal position to simulate unlocking", "assets-used": ["safe rotate"], "success-criteria": ["articulated open"] } Fail to generate a correct task! Failure reason: The handle is not rotatable.You are an expert in creating robotic simulation environments and tasks. You will be given longhorizon tasks with each task including at least 2 sub-tasks. Each sub-task can only include one simple motion such as moving the gripper to some object, opening or closing the gripper fingers, or interacting with certain articulated objects by its prismatic/revolute joints. ...... Now please start to generate a sub-task decomposition of the following new task:"task-name": "tidy-up-foam-brick", "task-description": "Pick up the foam brick, open the microwave door, place the brick inside, and close the door.", "assets-used": ["microwave", "foam brick"] } ......https://drive.google.com/drive/folders/1A584C9x5uAqppbjNNiVqlA_ 7uOOOlEII?usp=sharing 


Decision: The authors propose an LLM-based task data generation system for robots, together with a 3d-based sim-to-real pipeline. They particularly focused on realistic, 6-dof robot tasks. The work itself seems like a notable improvement over several similar works in this area. Their approach does leverage human expertise when creating tasks, which has advantages and disadvantages, but is interesting compared to the other related works. They also show some ability to generate long-horizon tasks.

During the rebuttal period, they provided a very detailed followup, with information clearly comparing them to related work and showing the ability of models trained using their method to generalize, as well as adding substantial extra discussion that addressed many of the core concerns raised by the reviewers.

Strengths:
	• Paper is mostly well-written and convincing
	• Generates lots of tasks beyond just pick and place - great step towards solving data problems in robotics
	• Good sim-to-real learning pipeline
Weaknesses:
	• Need for human intervention
Generalization is greatly improved from rebuttal, but some concerns remain

Reviews: review: title: title: Paper review
paper_summary: paper_summary: This paper proposes GenSim2 a LLM based task generation pipeline that can generate realistic articulated object tasks capable of sim-to-real transfer.
main_review: main_review: Strengths:
1. This paper presents a scalable solution to the important topic of simulation task generation; GenSim2 is compelling with its capability to generate 100+ simulation tasks that go beyond the top-down pick-and-place tasks that prior works are able to generate. 

2. This paper designs an effective sim-to-real pipeline with a 3D transformer architecture; this is well executed to show that the generated tasks are usable. 

3. The paper is well-written and easy to follow.

Weaknesses:
1. This paper is quite similar to a prior works RoboGen, and on the surface level, it's not clear the novelty of this work in comparison to RoboGen. While there are implementation differences, such as using kPAM solver for demonstration generation (which is rather specific to the task categories this paper considers), the overall system pipeline is rather quite similar.

2. The approach is limited by what is simulatable or available in simulation. That is, this paper takes pre-existing assets and then design the algorithm around them, e.g., the use of kPAM planner which is limited to  object-centric displacement tasks. In contrast, concurrent works like RoboGen or RoboCasa do consider generative new assets. 

3. The best performing GenSim2 variant requires human feedback on the generated tasks; this limits the scalability of the approach. Without human feedback, GenSim2 variants do not outperform RoboGen on single-step tasks. On long-horizon tasks, only the variants with human feedback are considered. I think this limitation is rather important, but it's not mentioned in the introduction or the limitation section.

Overall, I think GenSim2 is a sound system but with some important limitations and its comparison against prior works can be strengthened.
questions: questions: Additional Question/Comments:
1. In the real-world tasks, some generalization over unseen objects, scenes would be nice to demonstrate that the policies learned in simulation can have good generalization. 

2. Typo in line 230.
limitations: limitations: 3
review_summary: review_summary: Weak Accept because this paper presents a sound system for end-to-end LLM task generation to sim-to-real transfer; however, the technical novelty is rather limited and many components take a lot of engineering to get to work. Furthermore, its reliance on human feedback is not highlighted enough in the paper.
title: title: A solid and functional work with some presentation flaws
paper_summary: paper_summary: The work presents GenSim2 a framework to enable sim2real transfer. GenSim2 leverages LLM and VLMs for designing tasks in simulation, RL and motion planners to obtain demonstrations, and multi-task training, to aid generalization and transfer to real-world applications.
main_review: main_review: ## Strengths
* **Problem**: given the increasing ability of LLM and VLM in designing tasks, reward functions and policies in simulation, I believe works like this will be important in order to leverage such abilities for sim2real transfer
* **Method**: the framework presents a robust pipeline that should enable effective sim2real transfer with minimal human intervention. The overall methodology and the results are convincing  
## Weaknesses
* **Complexity**: the system itself is complex and made of several components, and variables. It is hard to understand which components are contributing to the overall success of the system. The authors presented some ablations, but these don't exhaustively cover all the variables involved (e.g. the simulator, the LLM, the RL policy learner, the motion planner, etc). It would be great if additional ablations could be provided 
* **Presentation flaws**: there are some presentation flaws, such as missing information, typos and missing references. I would recommend the authors revise their paper (also considering the suggestions provided in the Questions for rebuttal section)
questions: questions: * What is the simulator used? This should be clearly stated in the text (and Figure I caption as well)
* What is the task library adopted? How is this composed/learned?
* List of presentation flaws:
Line 30, GenSim2 1, GenSim2 (Figure 1)
Line 49, 25 -> 25%
Line 110, "language and visual reflections" -> I think a citation to [1] or similar work would be necessary
Paragraph from line 112 to 125, I think the bottom-up approach is very similar to [2] and a citation to it or to similar work would be necessary
Line 196, pcd not defined (pointcloud?)
Line 199, 4.4 -> Section 4.4
Line 230, LLM -> the LLM
Line 231, missing Section (??)

[1] Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al

[2] Voyager: An Open-Ended Embodied Agent with Large Language Models, Wang et al

**Post rebuttal update**:

Apologies for the tardiness in the response. I have gone through the extensive rebuttal provided and updated my score to a "Strong Accept". 

**Additional ablation studies**: I appreciate the authors' efforts in adding this, as it helps to understand what are the sources of performance improvement in their system.

**Task library**: I think a more detailed description of the task library should be included in the paper, as clarified by the authors in their response.

Overall, given the additional clarifications and the extensive amount of additional content presented by the authors, I think the paper would be a nice addition to the conference, and thus I think it should be accepted.
limitations: limitations: 2
review_summary: review_summary: A convincing method for sim2real transfer, with minor presentation flaws which are expected to be solvable by the authors.
title: title: Utilizing LLMs for Robot Task Generation and Learning
paper_summary: paper_summary: This work (GenSim2) presents an extension of GenSim, a method for scaling robot task creation using large language models (LLMs). The extensions address increased task complexity, improved task solvers for demonstration data collection, and training a multi-task transformer model. First, the iterative top-down and bottom-up approaches enable the generation of long-horizon tasks involving articulated objects. Second, the integration of a kPAM planner with reinforcement learning (RL) policy learning enhances the efficiency and effectiveness of task-solving for demonstration data collection. Third, the proposed multi-task transformer model is trained to distill the demonstration data into a single policy, demonstrating sim-to-real transfer through the careful selection of observation inputs. The paper shows promising results in generating long-horizon tasks more effectively than previous methods, creating 50 single-step tasks and 50 multi-step tasks. Additionally, the proposed approach solves these tasks and distills multiple tasks into a single policy. Notably, the distilled multi-task policy exhibits effective sim-to-real transfer, primarily due to the use of point cloud observations, which has a small sim-to-real gap.
main_review: main_review: The paper is clear and easy to follow, effectively motivating the challenge of automatic task generation and extending it towards long-horizon tasks.

However, a number of components of the work require clarification or additional experimentation. For example:

- **Importance of Visual Reflection**: Section 3.1 mentions that both language and visual reflections of the LLM are used to verify the task, but section 4.1 talks about humans doing the reflection. It might be better to clarify in the methods section that the reflection/feedback can either be provided by humans or LLMs.

   - **Confusion about _GenSim2 w/oV_ results**: The configuration labeled "GenSim2 w/oV" is unclear. The paper describes it as GenSim2 without the vision API of GPT-4, but it is not specified whether this is an ablation of GenSim2-HF or GenSim2-VF. If it is an ablation of GenSim2-VF, it is unclear why removing vision (as in GenSim2 w/oV) would improve performance compared to GenSim2-VF.

- **No compelling examples of Long-Horizon Tasks**: Although generating long-horizon tasks is a key feature of this work, none of the supplementary materials include examples of such tasks. Providing examples would help illustrate the capabilities of the proposed method.

- **Pre-actuation and Post-action Motion**: The paper states that these motions are defined as a series of waypoints. Does each waypoint represent a 6D end-effector pose? How many of such waypoints are specified for each task? Additional details, such as the typical number of waypoints required, should be included in the appendix to provide further clarity.
questions: questions: See review comments.
limitations: limitations: 3
review_summary: review_summary: While the work is interesting and addresses the challenge of task generation, it is fairly incremental, and it is not entirely clear if the improvements over GenSim are significant enough to warrant a completely new paper. The tasks remain relatively simple and similar. Additionally, it is unclear how much human input is required for task generation and solving (e.g., human-specified kPAM key points), especially given that the primary motivation for using LLMs in task generation is to reduce the burden on humans in the process.