HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning
Authors: Zhi Jing
Siyuan Yang
Jicong Ao
Ting Xiao
Yu-Gang Jiang
Chenjia Bai
Venue: NeurIPS 2025 poster ()
Keywords: Bimanual Manipulation
Dexterous Manipulation
Humanoid Robots
LLM Planning
Primary area: deep_learning
TL;DR: This paper presents HumanoidGen, an automated task creation and demonstration collection framework that leverages atomic dexterous skills and LLM reasoning to generate relational constraints.

Abstract: For robotic manipulation, existing robotics datasets and simulation benchmarks predominantly cater to robot-arm platforms. However, for humanoid robots equipped with dual arms and dexterous hands, simulation tasks and high-quality demonstrations are notably lacking. Bimanual dexterous manipulation is inherently more complex, as it requires coordinated arm movements and hand operations, making autonomous data collection challenging. This paper presents HumanoidGen, an automated task creation and demonstration collection framework that leverages atomic dexterous operations and LLM reasoning to generate relational constraints. Specifically, we provide spatial annotations for both assets and dexterous hands based on the atomic operations, and perform an LLM planner to generate a chain of actionable spatial constraints for arm movements based on object affordances and scenes. To further improve planning ability, we employ a variant of Monte Carlo tree search to enhance LLM reasoning for long-horizon tasks and insufficient annotation. In experiments, we create a novel benchmark with augmented scenarios to evaluate the quality of the collected data. The results show that the performance of the 2D and 3D diffusion policies can scale with the generated dataset. Project page is https://openhumanoidgen.github.io.

HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning
Zhi Jing
Siyuan Yang
Jicong Ao
Ting Xiao
Yu-Gang Jiang'
Chenjia Bai
Fuudan University
Institute of Artificial Intelligence (TeleAI), China Telecom
3 University of Science and Technology of China
*East China University of Science and Technology
Abstract
For robotic manipulation, existing roboties datasets and simulation benchmarks pre- dominantly cater to robot-arm platforms. However, for humanoid robots equipped with dual arms and dexterous hands, simulation tasks and high-quality demonstra- tions are notably lacking. Bimanual dexterous manipulation is inherently more complex, as it requires coordinated arm movements and hand operations, making autonomous data collection challenging. This paper presents HumanoidGen, an automated task creation and demonstration collection framework that leverages atomic dexterous operations and LLM reasoning to generate relational constraints. Specifically, we provide spatial annotations for both assets and dexterous hands based on the atomic operations, and perform an LLM planner to generate chain of actionable spatial constraints for arm movements based on object affordances and scenes. To further improve planning ability, we employ a variant of Monte Carlo tree search to enhance LLM reasoning for long-horizon tasks and insufficient annotation. In experiments, we create novel benchmark with augmented scenarios to evaluate the quality of the collected data. The results show that the performance of the 2D and 3D diffusion policies can scale with the generated dataset. Project page is https: /ttp::///pppeeeee hptp///aonnmninee.... github.ie io.
Introduction
The long-term. goal of embodied manipulation is achieve human-like manipulation capabilities across versatile scenes and tasks [1, 2]. Humanoid robots, with their human-like morphology, offer universal platform capable of leveraging bimanual coordination and dexterous hand manipulation [3, 4], potentially enabling more complex tasks than conventional robot arms. Bimanual dexterous manipulation is inherently intricate due to the requirement for coordinated arm movements and hand operations, which in turn makes the collection of the demonstration dataset more challenging.
The existing data generation pipeline for humanoid robots primarily depends on teleoperation systems via Virtual Reality (VR) [5, 6] and exoskeleton devices [7, 8, 9]. However, teleoperation requires the deployment numerous real-world objects and scenes, and also requires proficient operational skills from human operators. This makes it difficult for the dataset to cover diverse real-world tasks and scenarios. Consequently, existing real-world datasets such as Open-X [10, 11, 12, 13] consist mainly of single-arm manipulation data. To address this issue, many researchers have turned to various simulators for data acquisition, such as SAPIEN [14], IsaacSim [15], and MuJoCo [16]. the context of bimanual manipulation, simulated benchmarks such as Aloha [17]. PerAct2 [18], and RoboTwin [19] utilize dual-arm or wheeled robots for task creation and data collection. These efforts are limited
Correspondence to: Chenjia Bai (baicj (chinntelecoommcc) Equally leading organizations
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SpatialAnnotation Auntttasiiii
Demonstration Generatoon
Scene Scaling
Hand Aznotalion
Layout Scaling
Stzle
Task: Open Drawer
Iniiad
Initall State
Scene G Generation
CBBBaccpropagation
cdon
(1) Select
Cumrnn State
2ooks
Primiire Scane
Attilbutes
ledk
Description
Prshitited Action -e...
(2) Expand
Policy Evaluation
L.M
State
LLM
Exe Code
Joua
Pount Cloud
Closs Box
Pop
Black Strngg
Consraitalp
DPS
(Ade(A
Sucoss Critaraa)
Figure I: The overview of HumanoidGen. It includes spatial annotations, scene generation, constraint generation, -CCTS-enhanced reasoning, data collection, scene scaling, and policy evaluation.
in terms of task diversity and generally do not involve dexterous hands. Similarly, HumanoidBench [20] and BiGym [21] use humanoid robots perform loco-manipulation tasks but depend on VR or reinforcement learning (RL) policies for data collection, which would be costly considering that humanoid robots require coordinated control of arms and dexterous hands. Therefore, more efficient approach is to allocate rich interactable assets in simulation for task creation and develop fully autonomous paradigm for data collection. Taking inspiration from LLM--riven planning methods [22, 23], our goal is to generate code-form planning to complete humanoid manipulation tasks, aiming to produce high-quality demonstrations that can scale efficiently.
In this paper, we propose HumanoidGen, an LLM-based framework capable of generating diverse manipulation tasks and collecting scalable demonstrations for humanoid robots. For task creation, an LLM planner is prompted to generate code for environment setup and success criteria based on language descriptions and the wealth of 3D assets. gather demonstrations for each task, we first identify atomic operations for dexterous hands (such as pinch and grab), then we adopt LLMs to perform task decomposition for long-horizon horizon tasks, generating spatial rational constraints for arm movements. Specifically, we give spatial annotations for assets, and the constraints are defined through geometrical relationships on contact points and the function axis for both the arm and entities. Then we conduct I.LM-based code generation by translating planning into executable code-form constraints. off fff-ff trajectory optimizer applied subsequently to solve the constraints to get the arm and hand movements. To further enhance reasoning efficiency in complex tasks with long constraint chains, we incorporate a variant of Monte Carlo tree search (MCTS) for better test-time reasoning. This approach strengthens the reasoning ability of LLMs, especially when the required spatial annotations are absent.
Building HumanoidGen, we develop a comprehensive benchmark called HGen-Bence for bi- manual dexterous manipulation. In our setup, the Unitree H1-2 humanoid robot equipped with Inspire hands serves the robotic platform, and SAPIEN [14] as the simulation engine for data collection. HGen-Benn consists of 20 tasks of varying difficulty levels. For each task, a chain of constraints that determines arm movements and hand operations is generated, taking into account the spatial relationship between the robot and the objects. Then the actions are obtained by solving the constrained optimization problem. We execute these actions in the simulator and record the successful trajectories as demonstrations. We evaluated the success rate in trajectory generation with randomized scene configurations. The results show that MCTS significantly improves the reasoning ability of LLMs for long- horizon tasks and insufficient annotations. We train both 2D and 3D diffusion policies on the generated data, and the results show that the performance of the policy improved continuously as more demonstrations are incorporated.
2 Method
HumanoidGen is automated framework for scene generation, demonstration collection, and data generalization for bimanual dexterous manipulation, aiming to provide high-quality demonstrations over diverse scenarios to facilitate data scaling and policy learning. The overview of our framework

---

Pauring
Storage Point
Axis
Storage Axis
Pouring
Storage
Point
Point
Stomge
Function
Grasp Point
Pinch Point
Press
Attach
Approach
Parallel
Point
Point
Axis
Axis
Axis
Figure 2: The spatial annotations, including key points and key axes for assets and hands, as well as the atomic operations of hands that include grasp, pinch, and press.
is shown in Fig. 1. (i) As preparation, the assets and dexterous hands are meticulously annotated with spatial information, allowing flexible and reliable LLM-based planning. In scene generation, the LLM planner aims to generate an environment setup with code-form configuration based on asset, scene, and task descriptions.
(ii) Based on the generated scenes and pre-defined hand atomic operations, the LLM proceeds to generate the planning code of chain of spatial constraints for subsequent data collection. (iii) For tasks with we employ MCTS with introspective exploration to enhance the reasoning ability of LLMs. (iv) Then, we collect demonstrations by executing the planning with augmented scenarios to enhance data diversity. These demonstrations are utilized to construct a humanoid manipulation benchmark for policy evaluation.
planning and insufficient annotations,
2.1 Spatial Annotation and Scene Generation
Spatial Annotation Preparation. For efficient LLM planning with atomic hand operations, conduct meticulous annotations, including key points and key axes for both assets and dexterous hands. As shown in Fig. 2, we categorize annotations into three types as follows.
Hand Atomic Operation. The annotations of atomic operations indicate how the dexterous hands can interact with assets. For smooth operation execution, we perform the annotation for each atomic operation on both dexterous hands. As shown in Fig. 2, the key point and the axis vary in different atomic operations. When annotating the operation axes, specifically categorized them into three types according to the finger movement of the atomic operation: approach axes, attach axes, and parallel axes. Take 'pinch' as example, (i) its approach axis specifies the direction from which the dexterous hand should approach its pre- pre-pinching pose. This ensures proper alignment for the pinching operation execution. (ii) The attach axis indicates the finger movement direction during the operation execution, pointing from the thumb tip to the index finger tip for 'pinch'. (iii) The parallel axis, perpendicular to the first and oriented parallel to the palm plane, typically specifies the object's rotation axis during manipulation. This helps specify the rotation axis when manipulating an articulated object.
Asset Inherent Information. Asset inherent information indicates how the asset performs its own functionality. Usually, they are the points and axes where the asset exerts its functions, which are used to indicate the interaction between objects. These points and axes vary according to the functionality of the asset. For example, in Fig. 2, the cup has two possible functions: pouring and storage, which correspond to different annotations on the point and the axis, respectively.
Asset Operation Annotations. Asset operation annotations define the points and axes that indicate how various atomic operations can be applied interact with it. Importantly, atomic operations do not have one-to-one correspondence with assets: (i) Different operations can be performed on the same asset. For instance, both grasping and pinching can apply to cup. (ii) The same operation keypoints on both the upper and lower parts of a drawer. (iii) A single keypoint may support multiple operations, such the endpoint of box lid, which allows both grasping and pinching (Fig. 2).
asset
different key points the same asset. As shown in Fig. 2, grasping can target
With annotations, we can abstract the scene information into a of key points and key axes in the local frames of objects and hands separately. This facilitates the LLM's spatial and relational understanding of the scene and tasks, providing the foundation for defining the constraints of various atomic operations. We also note that the recent approach [19] adopts Stable Diffusion to simplify the annotation process for similar assets, which can be integrated with our method. Implementation details and evaluation results provided in Appendix A. and Appendix C.2.
3

---

Action Frame
Object Frame
Block Storage
Point Constraints
Task Description
Scone Layout
LLM
Axis Constraints
Poini Constraints
Ascct
Grusp Drawer
Pull Dower
Left
Hand
Initial
Pre Pre-Grrasp
Grasp
Open
Step
Right Hand
Initial
Pre-Pinch
Move
Pinch
Move
Keep
Pinch cube
Handover
Move Pre-Ginap
Storage
Open Hand
Move
Open Hand
Pinch
Move (L)
Move (L)
Group (1.)
Move
Dynamic Collision Avoidance
Active Collision Avoidance
Collision Point Cloud
Figure 3: An illustration of the generated plan for the task block storage. The LLM is prompted with task description, scene layout, and asset attributes generate step sequence. Fach step expressed using atomic operation, along with its corresponding annotations. During plan execution, (i) from step 0 5, the left hand pulls out the drawer by grasping its handle, while the right hand simuitaneously pinches and lifts the cube. (ii) From step 6 9, the left hand takes the cube from the right hand. (iii) From step
15, the left hand places the cube into the drawer and pushes the drawer back by grasping its handle. The LLM avoids collisions that would occur from directly moving the pinching pose by planning a collision firiiiioo--reeeeeee method during steps 6 7, demonstrating active collision avoidance. Additionally, the LLM proactively generates code account for potential collisions with the drawer when performing free motion in the compact workspace, as illustrated in the bottom part.
Scene Generation. Based on the annotations, an LLM planner is prompted to generate tabletop- level task scenes and configurations according to the task description. To achieve this, we take the asset library information, scene details, and task requirements as the prompt, then the LLM planner generates task-setup code, determining the categories, quantities, placement poses, and other attributes of the selected assets. In addition, we employ LLMs to generate task success criteria to guide the execution process and determine when the task is completed. For instance, in the cup pour task, the LLM must infer the size of cubes contained in the cup based on the cup's size, and assign plausible positions for the cup, the cubes, and the target bowl. The success criterion defined as all cubes being successfully placed inside the bowl.
2.2 LLM-based Task Planning
We design automated generation method generate scripts of constraint chains, which are used to collect demonstrations with the help of trajectory optimizers. Our method significantly differs from the conventional methods, which use LLMs only for high-level task plan generation and require wide array of predefined low-level operations [24]. Specifically, our method includes task decomposition, relational action constraints, and collision avoidance to enhance planning abilities.
Task Decomposition. As shown Fig. 3, the LIM planner decomposes the long-horizon task into action sequence S {Si, S2, Sn} with steps according to the task description, where

---

each step S; = indicate hand operations and arm movements respectively. (i) For hand operation. the LLM selects A; from the atomic library A Aband = { (Apinch, Agrasp, Aopen) at each step. (ii) For arm movements, we define two types of constraint: the goal pose constraint Cj and the path constraint Cruth Using these two constraints, the motion planning problem can be transformed into a constrained optimization problem, denoted as M, = [f(CC )s f(Cpath)), where the constraint C, is inferred by the LLM based on contextual atomic operations and functional reasoning related to movement, and f()) is a constraints solver detailed in Appendix A.2. By employing task decomposition, the long-horizon task can be decomposed into atomic operations and constraint definition, effectively reducing the reasoning complexity while enhancing planning efficiency.
includes left and right parts, with Sy E {A,, Mt). where A, and M;
Relational Action Constraints. Compared with conventional grippers, dexterous hands require more sophisticated motion constraints to accommodate diverse hand gestures and contact interactions. To systematieally describe these relationships, we introduce set of dynamic coordinate frames Fast that represent the contextual action space of each hand. Each frame F = (PI.... .P;; U1..... , Um) in Fact consists of key points Pa R and axes 0y E RR. The composition of Fact evolves dynamically according to the manipulation state: for instance, the left-hand action frame Fust initially includes only {Fi). which defines the geometric properties of the hand itself, and extends to {Fr, Fo} once a rigid grasp with object is established. Each constraint C E {Cruhh C specifies a geometric relation between an element of Faet and one from the global frame set Faul = Foty U Fhand U Fwodds These relations can take the form of point or axis correspondences, such as coincidence, parallelism. or orthogonality, allowing flexible encoding of motion intents. For example, in a grasping action (step 2 in Fig. 3), goal constraint may enforce point coincidence and directional aligoment, Pggnspp = p
and u hand || "appoouchh During the subsequent pulling phase (step 3), a path constraint Crath "approach
(ybend || > is applied to ensure that the hand maintains stable gesture throughout the motion.
This formulation explicitly encodes the spatial logic underlying dexterous manipulation, enabling consistent and adaptive control across complex, multi-stage tasks.
Collision Avoidance. Ensuring effective collision avoidance during atomic operations is a key challenge. We propose two solutions: (i) Active Collision Avoidance. The LLM planner generates atomic operation scripts that proactively incorporate collision avoidance behavior. This ensures the feasibility of the atomic operations, especially when coordinate bimanual manipulation is necessary. For example, in block stack, the left hand retracts from the stacking area to make space for the right hand. (ii) Dynamic Collision Management. The LLM planner dynamically manages collision checks to enable in-contact manipulation. Specifically, we maintain an object-ignoring list and let the LLM dynamically adjust it when generating scripts, thereby providing guidance for successful low-level trajectory optimization and execution. This is crucial for in-contact articulated object manipulation, whereas existing methods ignore this aspect or rely on manual processing in atomic operation programming. For example, in the block storage task illustrated in Fig. 3, the LLM adds the drawer to the list to ignore the contact between the robot hand and the drawer when pulling the drawer out. After that, the drawer is removed from the list, enabling subsequent collision avoidance. This approach seamlessly integrates in-contact manipulation with free-space motion, significantly enhancing the framework's 's ability to handle complex long horizoo dexterous tasks.
2.3 Enhancing Reasoning with MCTS
When performing long-horizon tasks or encountering insufficiently annotated objects. LLMs lack sufficient prompts for reliable reasoning, which often leads to failed planning. To address this issue, we employ tree search to enhance the ability of multi-step reasoning in task planning. Specifically, Combine-Resune (STCR) mechanism that abstracts a planning
we propose a novel Segment- Segment-Truncate search tree from LLMs outputs. By integrating MCTS exploration and exploitation strategies, the tree is iteratively expanded to derive an executable solution.
STCR mechanism. Inspired by abstractions in proving problems [25, 26]. STCR aims to abstract executable code into a tree to define nodes and branches, which includes four steps. (i) Segment. By matching critical functions, the planning code is segmented into multiple steps according to the granularity of the execution. The segmentation is inferred by LLMS following a similar granularity as described in the task decomposition of $2.2, denoted as S = (S,, ). (ii) Truncate. The truncation is performed at the point where an error occurs. We execute the segmented steps sequentially until a code formatting error action execution failure occurs (c.g., move or grasp
5

---

failed). We truncate the steps at the point of failure, discard the steps after the erroneous step, and retain the valid segments as Sreman {S. - i 1}. where Sy. failed. (iii) Combine. We merge atomic operations with consistent intent in Stemain to form new execution sequence Stemain = (S}, S ..... ) where k' k 1. Operations such as grasping and pinching are implemented by multiple steps. These steps are abstracted into single execution units during tree search. For instance, if S represents a grasping action containing two move steps, then Si = [A M2,, M3. A denotes the combined action sequence. In the task tree, each combined action sequence represents an executable branch, and its terminal state serves as a new node in the tree structure. (iv) Resume. For newly created nodes, the code of the executed sequence, the error code, and the scene state information are stored, which will be resumed when this node is selected in expansion.
Interactive Task Planning via MCTS. Based on the tree built by STCR, we adopt MCTS as the tree search algorithm, consisting Selection, Expansion, and Backpropagation steps. Different from standard MCTS methods, we incorporate simulation steps into the expansion process to provide execution results of the nodes. The details are given in Appendix A.4.
Scene Scaling. To enhance the data diversity, we perform room-level scene scaling that allows demonstrations to contain diverse scenes using demonstration scripts generated from table-level task scenes. We compute the transformation matrix between the coordinate systems of the original and new scenes to align the two scenes. The details are given in Appendix A.3. By hamnessing diverse assets in RoboCasa [24], we can generalize demonstrations across over 120 scenes. This substantially diversifies the dataset's s task scene distribution, ensuring broad coverage of potential scenarios.
3
The HumanoidGen Benchmark
Closs Lapttp
Open Laptrp
Close Boy
Open Drawer
Handover And Smrrage
Cup Four
Figure 4: HGen-Bench includes various dexterous bimanual manipulation tasks of varying difficulty. We provide different observation information and deploy the tasks in home scene.
Based on our framework, we construct a benchmark, HGen-Bench, for humanoid manipulation. We designed 20 tasks performed using Inspire hands mounted on the arms of the Unitree H1-2 humanoid robot, which has 7 DoFs for each arm and 6 DoFs for each hand, resulting in a 26-dimensional action space. The examples are given in Fig. 4. In front of the robot, we place table and construct scenes with various objects, including small items such as blocks of various shapes and colors, articulated objects such laptops and drawers, and everyday items such bottles, cups, and plates.
We design atomic operations fully exploit the dexterity of the hands and define task difficulties from easy to hard settings. We also incorporate bimanual coordination tasks to leverage the capabilities of dual dexterous hands, as well long-horizon tasks that require geometric reasoning from the LLM planner. To enhance variability, we randomize the initial position, pose, and joint angles of articulated objects within a certain range. We provide RGB and depth images from six camera views, located on both wrists, a first-person perspective, and three third third-person views. The details are given in Appendix B.
4 Related Works
L.LM-based Data Generation. Leveraging foundation models to automatically generate diverse tasks and scenes is promising for scaling robotic data sets with minimal human effort. RoboGen [22]

---

is generative robotic agent that generates scene components and configurations with language descriptions, and the skills are learned by optimizing L-M-eeeerated reward functions. Gensim [2.]] adopts L.MSs to generate codes to build scenes, simulations, and expert demonstrations. Gensim2 [27] further considers long-term and articulated tasks beyond aacc-end- tasks and incorporates reasoning models for planning. However, these approaches predominantly center on single-arm robots, whereas our work addresses bimanual dexterous manipulation tasks, which are critical importance for humanoid robots. Regarding expert data collection, LLMs have been leveraged to generate functional control programs for grasp-oriented tasks [28, 29] or articulated object manipulation [30, 31]. In contrast, our method enables code-form planning by framing the manipulation task as sequence of constraints, thereby eliminating the need for human intervention in defining the functions. Although Rekep [32], OmniManip [33], and RoboTwin [19] utilize spatial reasoning based key points and axes, they have limitations in handling bimanual dexterous tasks.
Bimanual Manipulation. Bimanual dexterous manipulation is promising in solving complex tasks through the coordinated operation of two arms. However, this area faces several challenges, including data scarcity [34, 35], enlarged action spaces [9], diverse collaboration modalities [36, 37J, and the intricacies of dexterous hand control [15]. Recent advancements have led to the development of simulation benchmarks for dual-arm systems [21, 19] and humanoid robots [20]. Nonetheless, these benchmarks exhibit limitations in task and scene diversity and typically exclude dexterous hands. These problems motivate us to develop automatic mechanisms for constructing bimanual environments and generating demonstrations, with the aim of broadening the skills and scenarios encompassed within bimanual datasets. For bimanual policy learning, previous methods rely on human-object interaction [38, (3), geometric constraints [40], and arm-movement primitives [41, 42]. In contrast, we do not consider these priors and directly train the 2D and 3D diffusion policies [43, 44), enabling a direct assessment of the effectiveness of the data collection paradigm.
Datasets and Benchmarks. Collecting real-world demonstrations is promising for acquiring realistic environmental observations and encompassing the target scenarios robots. Recent representative datasets, including RT-1 [45]. RH-20T [12], DROID [46]. Bridge data [13]. Open X-Embodiment
[10]. RoboMind [11]. and Agibot World [47], have collected numerous manipulation tasks on specific hardware platforms. However,
focus humanoid robots, the embodiment differences make most of the data unsuitable for directly training bimanual dexterous policies. Although methods such as physically interpretable action space [48] and latent action space [2, 49] have been proposed, the adaptation cross-embodiment data remains an open research challenge. Additionally, although several data augmentation techniques have been introduced [50, 51], the collection of large- scale real-world data remains costly. On the simulation front, benchmarks such as ManiSkill [52], SIMPLER [53], Robo Twin [19], RoboCasa [24], and Garmentlab [54] provide extensive assets and task configurations, while we focus on auto-create task variations and data collection for humanoid robots.
Experiments
We conducted the following three groups of experiments: (1) a comprehensive evaluation of demon- stration generation and execution performance of our framework compared to Robotwin; (2) an effectiveness evaluation of MCTS in enhancing the demonstration generation process; and (3) validity evaluation of the collected demonstration data training bimanual dexterous manipulation policies.
In addition these main experiments, further analyses and evaluations are provided in the Ap- pendix C, including real-world experiments (Appendix C.1), automatic asset annotation evaluation (Appendices C.2), additional challenging dexterous manipulation tasks (Appendix C.3), resource and efficiency analysis of HumanoidGen (Appendix C.4), comparison with existing generation frameworks (Appendix C.5), and comparison across different large models (Appendix C.6).
5.1 Evaluation of Data Generation and Execution
Experimental Setup. To better show the superiority of our framework, we designed 20 different tabletop manipulation tasks and conducted quantitative evaluation of the demonstration generation and execution capability of our framework. We divide the 20 tasks into 4 groups based on the following factors: the number of arms used, the length of task horizons, and the complexity of collision scenarios. This categorization allows for more detailed evaluation of the frameworks'
7

---

Generation Success Rate 19)
Rabatwin (dextorous hand)
HumanoidGsen
100
75
50
open drawer
0
cosse drawer
open box easy
close box easy
open laptop easy close laptop easy open box hard close box hard
open laptop hard close laptop hard
cup pour easy empty cup place
dual bettles pick easy dual battles pick hard
block handova
blocks stack easy blocks stack hard
handover and storage
pyramid stack
handover and storape cooparation
Single amm free motion space
Single complex collisinn scenarios
Bimenuel, space
Dimenusl
long honzon, complex colision scenaros
Figure 5: Experiment results of demonstration generation and execution capability. The tasks are categorized into 4 groups, with names based on the number of robotic arms required and their relative difficulty levels. The features of task categories are noted in parentheses.
performance. We compare our method to RobotTwin, where we modify the official implementation to add dexterous hands and provide additional annotations enable dexterous atomic operations. Besides, we prompt the L.LM not to perform proactive dynamical collision management during inference, which reflects the default setup in PobbotwinUUiig the DeepSeek-R1 [55], generate and execute the demonstration scripts with both frameworks and compare the final success rates.
Experimental Results. As shown in Fig. 5, HumanoidGen demonstrates superior performance across all dexterous manipulation tasks, achieving an average success rate of over 50%. Except for some bimanual long- horizon tasks that involve higher complexity, most task types achieve success rate above 75%. These results highlight the effectiveness of our framework, making automatic and efficient demonstration generation feasible in bimanual dexterous tasks. In comparison, Robotwin shows comparable performance on single-arm and bimanual short-horizon tasks, indicating that the spatial annotations related dexterous atomic operations introduced in $2.1 are effective in allowing the dexterous hand to participate in automated demonstration execution for simpler scenarios.
Notably, we observe that HumanoidGen outperforms Robotwin in long- horizoorizon and complex collision tasks, such as the close & open box in single-arm tasks and the blocks stack easy & hard in bimanual tasks. Further analysis shows that HumanoidGen dynamically manages collision avoidance during inference, handling the collision scenarios flexibly. As an example, in the task handover and storage, the collisions with the drawer contact pulling operation, while such collisions are taken into account when planning a path around the drawer retrieve the cube with the right hand. Another example is the close box task, where the robot hand circumvents the box lid with collision-awareeesss to reach an intermediate pose. Subsequently, these collisions permitted during the flipping operation to complete the closure. These examples demonstrate the potential of our dynamic collision management approach enabling efficient data collection for long-horizon tasks involving complex collision scenarios.
appropriately ignored to facilitate trajectory planning for in-
5.2 Effectiveness Evaluation of MCTS
Experimental Setup. To evaluate the effectiveness of MCTS in enhancing the demonstration generation process of HumanoidGen, we selected three tasks from 15.1: blocks stack easy, blocks stack hard, pyramid stack, and added a single-arm task, block stack single. In contrast to $5.1, we increased the task complexity to rigorously evaluate the performance of LLM in scenarios with insufficiently annotated objects and long-horizon tasks. Specifically, we introduced two key challenges: (i) removing all operation annotations for cubes, forcing the LLM to infer relevant constraints solely from cube poses; (ii) simplifying task descriptions to provide minimal guidance. For example, in the blocks stack hard task, the LLM instructed only stack cubes into a pile without considering the order of operations and target positions, allowing highly uncertain execution. Building upon this setup, we compared the reasoning success rates and token consumption per execution between MCTS and non-MCTS strategies using DeepSeek-Ri [55] across these four tasks. Additionally, we analyzed the diversity of generated plans to assess the exploratory capabilities of each approach.
8

---

Method
Success (9))
Token consumptioo (K)
Stack Single
Non-MCTS
63.3 4M44
15.3 1:000 19.3 8.7
Herrec
MCTS
98.3 2.30
Liant Consciaaniin candtarrd laco MCTIA
Blocks Stack Easy
peramic stace (PoTt)
Non-MCTS
1:1
MCTSsaa
83.3 5.56
21.6 1:775
MCTS
95,0
22.8 9.133
Blocks Stack Hard
MCTS
MCTS
78.3 2:2
69.9 + 21244
MCTS
98.3 a 2.35
78.3 3:3175
Pyramid Stack
Non-MCTS
16.2
MCTS
76.7 4.71
80.0 + 31772
MCTSSa12
90.0 a 4.9
89.6 4:401
Table I: The evaluation results of applying different numbers of max MCTS exploration steps N and non-MCTS in four tasks.
Figure The variation of the count of distinct successful plans for MCTS and non-MCTS with the count of successful plans.
Experimental Results. As shown in Tab. 1, MCTS can significantly improve the reasoning ability of LLM with minimal additional token consumption. This demonstrates that during the reasoning process of MCTS, LLM can effectively utilize the information of tree nodes to infer the correct task steps. For example, in the 'pyramid stack' task, LLMs are prone to reasoning errors such as unreasonable selections of stacking positions and collisions of two arms in the stacking area. When exploring nodes where such errors occur, MCTS enables LLMs to analyze the causes of errors based on past experiences, correct the execution methods, and continue execution. Further, the results demonstrate that the exploration strategy of MCTS is highly cost-effective, as evidenced across all tasks. For instance, in the block stack single task, 26% increase in token consumption led 55% improvement in the reasoning success rate. Specifically, the average token consumption per successful execution method decreased from 24. to 19.63K, reducing approximately 20% token consumption. In addition, MCTS can enhance the diversity of generated plans. Fig. 6 illustrates the cumulative number of different plans generated by MCTS and non-MCTS methods as a function of successful planning counts. Among the 20 success solutions, MCTS obtains 2 12 distinct plans, whereas non-MCTS only generated distinct plans. This validates that MCTS can explore diverse outcomes by correcting erroneous execution strategies inherent in the direct reasoning approach.
5.3 Validity Evaluation of Collected Data
To validate the effectiveness of the collected dataset, we trained policies on the data and evaluated their performance, focusing on how success rates vary with different dataset sizes. We collected 100 data samples for each task using our method. Each sample includes RGB and depth images captured from six cameras, the joint states, and the action ground truth of the robot.
Policies used for evaluation. The Diffusion Policy (DP) [43] models based robotic control as a conditional denoising process, handling multimodal action distributions and high-dimensional action spaces effectively. The 3D Diffusion Policy (DP3) [44] enhances DP by incorporating point cloud data, improving performance in dexterous manipulation tasks, and demonstrating strong generalization. We evaluated DP3 and DP on 14 tasks using RGB images or point clouds as input, measuring success rates across different episodes.
Experimental Results. As shown Tab. for some relatively easy tasks, DP3 demonstrates few-shot learning capability. For example, in the cup pour easy task, DP3 achieves 67% success rate using only 20 demonstrations generated by our framework. This validates the diversity of our scene generation and the high quality of our collected data. Overall, DP performs worse than DP3; however, its performance improves with increased data. In the open box hard task, the success rate rises from 11.1% with 20 demonstrations to 100% with 100 demonstrations, even surpassing DP3, This confirms that our data scaling approach leads to continuous policy improvement. Notably, some task policies exhibit non-stationary characteristics. Despite achieving high success rate with 20 demonstrations, their actions exhibit instability and risk, as denoted by the **' in the table.
For challenging tasks, such as the long-horizon task blocks stack easy, both DP and DP3 exhibit noticeable performance decline, particularly when the data is limited. The suboptimal performance of DP may be attributed to the inherent limitations of RGB information, whereas DP3, which

---

Num of Demonstrations
20
100
20
50
100
Blocks Stack Easy
Drawer
DP3
0.0
0.0
22. 22.8
DP3
83,3
94.4
92.6483
DP
0.0goo
0.0gn0
0.0g00
DP
95.6g11
100.0
100.0.00
Cup Pour Easy
Dual Bottles Pick Easy
DP3
67.8 8o
75. 75.6
72.2.79
DP3
75.9
96.3 3.69
93.9276
DP
0.0
2.2
0.0
DP
0.0gan
0.0
0.0=00
Dual Bottles Pick Hard
Empty Cup Place
DP3
88 9.136
90.7
94.4=79
DP3
25.0a5.
18.3=47
33.3
DP
0.0g00
0.0a00
0.0ana
DP
0.0g00
0.0ga9
6,72153
Open Box Easy
Open Box Hard
DP3
85.6280
95.6244
95.0=441
DP3
95.6a55
16.1
98.3
DP
93.3=1333
100.0=a0
100.0+00
DP
11.1
93.3a94
100.0=00
Open Drawer
Open Laptop Hard
DP3
58.3+83
76.0=13.1
84,4=113
DP3
100.0=a0
100.0+00
100.0
DP
17.8=22.0
13.3
48.9:3144
DP
15.6#227
111199
35,6:324
Close Box Hard
Close Laptop Fasy
DP3
88 9
96.3226.
96.3=69
DP3
100.0
100.0
100.0
DP
*82.2
451.1:191
31.1
DP
37.8
40.0g
48. 48.9
Handover and Storage
Blocks Stack Hard
DP3
0.0go0
0.0gn0
0.0g00
DP3
0.0mnn
0.0ga
0.0g00
DP
0.0
0.0
0.0sa0
DP
0.0z00
0.0
0.0=00
Table We trained DP and DP3 using 100, 50, and 20 trajectories generated by our method, and evaluated the success rates across 14 tasks with 3 random seeds.
requires modeling actions of high DoFs and managing complex long-horizon tasks. necessitates larger amount of data. Our efficient data collection method enables rapid scaling of data, making it particularly effective for addressing the challenges faced by both DP and DP3 in handling complex long-horizon tasks with limited data. More evaluation details are given in Appendix B.
6
Conclusion
This paper introduces HumanoidGen, framework utilizing automatically generated scenes and synthetic data to facilitate the learning and execution of bimanual dexterous manipulation tasks. Spatial annotations for key points and axes of both assets and hands given, providing sufficient information for scene generation and planning. The LLM planner generates task decomposition and relational action constraints with collision avoidance applied, enabling the following trajectory optimization of tasks. We also employ collision avoidance and MCTS-based reasoning to improve planning efficiency in complex long-horizon tasks. We construct benchmark that contains diverse bimanual dexterous tasks for evaluation. The experiments show that our framework has superior capabilities in demonstration script generation and execution, especially for tasks with long task horizons and complex collision scenarios. MCTS improves LLMs' reasoning ability with insufficient annotations. The training of diffusion policies verifies the quality and scaling capacity of our method.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62427819 and 62306242), the Young Elite Scientists Sponsorship Program by CAST (Grant No. 2024QNRC001), the Yangfan Project of the Shanghai (Grant No. No.23YF111622 1462200), and the Science and Technology Commission of Shanghai Municipality (No. 24511103100).
References
[1] Shan An, Ziyu Meng, Chao Tang, Yuning Zhou, Tengyu Liu, Fangqiang Ding, Shufang Zhang, Yao Mu, Ran Song. Wei Zhang, et al. Dexterous manipulation through imitation learning: A survey, arXiv preprint 23335:5504.03555, 2025.
[2] Johan Bjorck, Fernando Castañida. Nikita Cherniadev. Xingye Da. Runyu Ding. Linxi Fan, Yu Fang. Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GrÜ0t nl: An open foundation model for generalist humanoid robots. arXiv preprint 14734 2025.
10

---

[3] Zhaoyuan Gu, Junheng Li. Wenlan Shen, Wenhao Yu, Zhaoming Xie. Stephen McCrory, Xianyi Cheng, Abdulaziz Shamsah, Robert Griffin, C Karen Liu, et al. Humanoid locomotion and manipulation: Current progress and challenges control, planning, and learning. arXiv preprint urXiv:2501.0211 2025.
[4] Yanjie 7e, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Ji- ajun Wu, Generalizable humanoid manipulation with improved 3d diffusion policies. arXiv preprint arXiv:2410. 10803, 2024.
[5] Xuxin Cheng, Jialong Li, Shiqi Yang. Ge Yang, and Xiaolong Wang- Open-television: Telcoperation with immersive active visual feedback. In 8th Annual Conference on Robot Learning, 2024.
[6] Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, and Xiaolong Wang. Bunny-visionpro Real-time bimanual dexterous teleoperation for imitation learning. arXiv preprint MxXi:2407.031622 2024.
[7] Shiqi Yang, Minghuan Liu, Yuzhe Qin, Runyu Ding, Jialong Li, Xuxin Cheng, Ruihan Yang, Sha Yi, and Xiaolong Wang. Ace: A cross-platform and visual-exoskeletons system for low-cost dexterous teleoperation. In 8th Annual Conference on Robot Learning, 2024.
[8] Qingwei Ben, Feiyu Jia, Jia Zeng, Junting Dong. Dahua Lin, and Jiangmiao Pang. Homie: Humanoid loco-manipulation with isomorphic exoskeleton cockpit. Robotics: Science and Systems, 2025.
[9] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Proceedings of Robotics; Science and Systems, 2023.
[10] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-69000 1992-6903, IEBE, 2024.
[11] Kun Wu, Chengkai Hou, Jiaming I Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, ct al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412. 13877, 2024.
[12] Hao-Shu Fang. Hongjee Fang. Zhenyu Tang. Jirong Liu, Chenxi Wang. Junbo Wang. Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 660-660. IREF, 2024.
[13] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hlansen-Estruch Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning scale. In Conference on Robot Learning, pages 1 1723-1736. PMLR, 2023.
[14] Fanbo Xiang, Yuzhe Qin, Kaichun Mo. Yikuan Xia. Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[15] Yuampei Chen, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAlcer, Hao Dong, Song- Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Syxtems, 35:5150-5163, 2022.
[16] Kevin Zakka, Baruch Tabanpour. Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Frik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujaco playground. arXiv preprint arXiv:2502.08844 2025.
[17] Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint 24Xi::2401.02/ 17, 2024.
[18] Markus Grotz, Mohit Shridhar, Yu-Wei Chao, Tamim Asfour, and Dieter Fox. Peract2: Benchmarking and learning for robotic bimanual manipulation tasks. In CoRL 2024 Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond, 2024.
[19] Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (carly version). In IEEE/CVF Conference Computer Vision and Puttern Recognition, 2025.
[20] Carmelo Sterrazza, Dun-Ming Huang. Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-bodd body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
11

---

[21] Nikita Chernyadev, Nicholas Backshall, Xiao Ma, Yunfan Lu, Younggyo Seo, and Stephen James. Bigym: A demo-driven mobile bi-manual manipulation benchmark. In 8th Annual Conference on Robot Learning. 2024.
[22] Yufei Wang. Zhou Xian, Feng Chen, Hsun-Hsuan Wang. Yian Wang, Katerina Fragkiadaki. Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. In International Conference on Machine Learning, 2024.
[23] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang. Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models, In International Conference on Learning Representations, 2024.
[24] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang. Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In RSS 2024 Workshop: Data Generation for Robotics, 2024.
[25] Huajian Xin, ZZ Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, et al. Deepseck-proeer-v.. 5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search, arXiv preprint arXiv:2408. 08/00, 2024.
[26] ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao. Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang. et al. Deepseek-prover-v2 Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint arXi::250. 2025.
[27] Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, and Lirui Wang. Gensim2: Scaling robot data generation with multi-modal and reasoning LLMs. In 8th Annual Conference on Robot Learning, 2024.
[28] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, brian ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In Workshop on Language and Robotics at CoRL. 2022, 2022.
[29] Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian GE, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, et al. Robocodex: multimodal code generation for robotic behavior synthesis. In Proceedings of the 41st International Conference Machine Learning, pages 36434-36454, 2024.
[30] Wenke Xia, Dong Wang, Xincheng Pang. Zhigang Wang, Bin Zhao, Di Hu, and Xuelong Li. Kinematic- aware prompting for generalizable articulated object manipulation with Ilms. In 2024 IEÉE International Conference on Robotics and Automation (ICRA), pages 2073-2080. IEEE, 2024.
[31] Xi Wang. Tianxing Chen, Qiaojun Yu, Tianling Xu, Zanxin Chen, Yiting Fu, Ziqi He, Cewu Lu, Yao Mu, and Ping Luo. Articulated object manipulation using online axis estimation with sam2-based tracking. arXiv preprint arXiv:2409. 16287, 2024.
[32] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In 8th Annual Conference on Robot Learning, 2024.
[33] Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omnimanip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints. arXiv preprint XXXi:25511008411 2025.
[34] Rudolf Lioutikov, Oliver Kroemer, Guilherme Maeda, and Jan Peters. Learning manipulation by sequencing motor primitives with a two-armed robot. In Intelligent Autonomous Syxtems 13: Proceedings of the 13th International Conference IAS-I3, pages 1601-1611, 1. Springer, 2016.
[35] Simon Stepputtis. Maryam Bandari, Stefan Schaal, and Heni Ben Amor. A system for imitation learning of contact-rich bimanual manipulation policies. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 11810-10 118101 IEEE, 2022.
[36] Fan Xie, Alexander Chowdhury, De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. Advances in neural information processing systems, 33:2327-223 22322,23377,22 2020.
[37] Giovanni Franzese, Leandro de Souza Rosa, Tim Verburg, Luka Peternel, and Jens Kober. Interactive imitation learning of bimanual movement primitives. IEEE/ASME Transactions Mechatronics. 2023.
12

---

[38] Jianfeng Gao, Xiaoshu Jin, Franziska Krebs, Noémie Jaquier, and Tamim Asfour. Bi-kvil: Keypoints-based visual imitation learning of bimanual manipulation tasks. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16850-16857 IEEE, 2024.
[39] Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-objeet understanding. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740- 21751, 2024.
[40] Jianfeng Gao, Xiaoshu Jin, Franziska Krebs, Noémie Jaquier, and Tamim Asfour. Bi-kvil: Keypoints-basee visual imitation learning of bimanual manipulation tasks. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16850-16857 IEEE, 2024.
[41] Giovanni Franzese, Leandro de Souza Rosa. Tim Verburg, Luka Peternel, and Jens Kober. Interactive imitation learning of bimanual movement primitives. JEEE/ASME Transactions on Mechatronics, 2023.
[42] Andrew Choong- Won Lee, Ian Chuang, Ling- Yuan Chen, and Iman Soltani. InterACT: Inter-dependency aware action chunking with hicrarchical attention transformers for bimanual manipulation. In 8th Annual Conference on Robot Learning. 2024.
[43] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 0278336492412236688 2023.
[44] Yanjic Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024.
[45] Anthony Brohan, Noah Brown, Justice Carbajal, et al. Rt-1: Robotics transformer for real-world control scale. In Robotics: Science and Systems, 2023.
[46] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudcep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: large-scale in wn-the-wild robot manipulation dataset. In Robotics: Science and Systems, 2024.
[47] Qingwen Bu. Jisong Cai. Li Chen, Xiuqi Cui, Yan Ding. Siyuan Feng. Shenyuan Gao, Xindong He. Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503. 06669, 2025.
[48] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024.
[49] Seonghycon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Sco. Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations. 2025.
[S0] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning, pages 1820-1864. PMLR, 2023.
[51] Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In 2025 IEEE International Conference on Robutics and Automation (ICRA), 2025.
[52] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint 201iv:2410.00425, 2024.
[53] Xuanlin Li. Kyle Hsu, Jiayuan Gu, Oier Mees. Karl Pertsch. Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, Sergey Levine. Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. In 8th Annual Conference on Robot Learning, 2024.
[54] Haoran Lu, Ruihai Wu. Yitong Li, Sijie Li, Ziyu Zhu, Chuanruo Ning. Yan Shen, Longzan Luo, Yuanpei Chen, and Hao Dong Garmentlab: A unified simulation and benchmark for garment manipulation. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
13

---

[55] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[56] Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unitied 6d pose estimation and tracking novel objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17868-17879 2024.
[57] Zhengrong Xue, Shuying Deng, Zhenyang Chen, Yixuan Wang, Zhecheng Yuan, and Huazhe Xu. mogen: Synthetic demonstration generation for data-efficient visuomotor policy learning. arXiv preprint arXiv:2502. 16932, 2025.
[58] Yiran Qin, Li Kang, Xiufeng Song, Zhenfei Yin, Xiaohong Liu, Xihui Liu, Ruimao Zhang, and Lei Bai Robofactory: Exploring embodied agent collaboration with compositional constraints. arXiv preprint 1614402503.16408, 2025.
[59] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang. Mengdi Zhao, Yao Mu, Pengju An, al. Robobrain: A unified brain model for rohotic manipulation from abstract concrete. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1724-1734, 2025.
[60] Jingshun Huang, Haitao Lin, Tianyu Wang. Yanwei Xiangyang Xue, and Yi Zhu. Cap-net: unified network for 6d pose and size estimation of categorical articulated parts from single rgb-d image. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 11654-11664 2025.
[61] Jianglong Yc, Keyi Wang, Chengjing Yuan, Ruihan Yang, Yiquan Li, Jiyue Zhu, Yuzhe Qin, Xueyan Zou, and Xiaolong Wang. Dex Learning with 1b demonstrations for dexterous manipulation. arXiv preprint arXiv:2506. 17198, 2025.
14

---

Appendix Table of Hem
A Implementation Details of HumanoidGen
15
A.1 Automatic Asset Annotation
15
A.2 Constraint Solver
16
A.3 Scene Scaling
17
A.4 MCTS-Based LLM Reasoning
17
B Details of HGen-Bench
18
Experimental Details and Additional Experiments
20
C.I Real-World Experiments...
20
C.2 Automatic Asset Annotation Evaluation
21
C,3 Additional Challenging Dexterous Manipulation Tasks
22
C.4 Resource and Efficiency Analysis of HumanoidGen
23
C.S Comparison with Existing Generation Frameworks
23
C.6 Comparison Across Different Large Models
24
C.7 Task Descriptions in Data Generation Experiment
24
C.8 Task Descriptions in MCTS Experiment..
24
More Discussion Limitations and Future Work
24
Prompts and Generation Samples
26
E.1 Prompt Template for Scene Generation
26
E.2 Sample Scene Code for Blocks Stack Hard Generated by HumanoidGen
27
E.3 Prompt Template for Demonstration Script Generation
28
E,4 Sample Demonstration Code for Open Drawer Generated by HumanoidGen
31
E.5 Prompt Template for MCTS-integrated Demonstration Script Generation
32
E.6 Sample Demonstration Code for Block Stack Hard Generated by Applying MCTS
39
F
Broader Impacts
41
A
Implementation Details of HumanoidGen
A.1 Automatic Asset Annotation
Function
Grasp
Pinch
Press
Attach Axis
Approach
Parallel
Point
Point
Point
Point
Axis
Figure 7: The illustration of automatic asset annotation.
this section, detail how simplify the annotation process using Stable Diffusion to automatically annotate similar assets and avoid repetitive annotation work. As shown in Fig. 7, assets of the same category have similar annotation information (points and axes), enabling direct annotation migrations across them. Specifically, we utilize the Stable Diffusion encoder for feature point matching. First, from the viewing angle where key
15

---

points are not occluded, we obtain the image I" of the annotated asset and the image _ of the unannotated For the annotated asset, there corresponding key points paon Inon, Following I and /oon, For cach p E p which corresponds to a single pixel in 19, we analyze the similarity of the extracted diffusion features to obtain the corresponding pixel in !non, Since the images are captured without occlusion, can obtain p nnnn in sequence by back projecting the pixcls in n into the 3D asset space. For axes annotation, starting from the obtained key points, the same axis directions as the original are extended to obtain the key axes the unannotated asset. By applying the approach above, the key points and key axes of the new asset are automatically annotated, effectively reducing the necessary effort by manual annotation.
key points p Jp..pp.....ppp P.... and our goal is obtain the method [19]. we extract the diffusion features both
A.2 Constraint Solver
As explained in $2.2, the LLM planner decomposes the long-horizon task into a sequence composed of hand atomic operations and arm movements. The arm movement solving problem is formulated as two constrained optimization problems, with the constraints Cwoel and Crath both inferred the LLM. We consider movement process to consist denoting the initial and the end time step, respectively. At each f, the arm joint angle is Oe, and the pose of the end effector is ex E SE(3).
time steps. Here, E (0. 1...., T) represents each time step, with = 0 and =
The motion solving process involves two steps. The first step is to (i) obtain the target end inffeffecoor pose ey and the target arm angle 0r. This can be formulated as nonlinear optimization problem
arg min ST
Σ C.EC
wars (Mact, Mall)
Wreg ||0
0 nominnll ||
er = frx(OT)
(Kinematic constraints)
(1)
S.L. c(uant: uall) E [Clower: Cupper], J. Vc E C
(Relational action constraints)
fr E Ocollision_free
(Collision avoidance constraints)
where w denotes the weight to balance various optimization terms. C(Uact, Mall) represents relational constraint between Facs and Fan, defined in $2.2. Here, u E {p, v, (p, v)} used a basic component for formulating the constraints, with p. v, and ((p, v) denoting points, vectors, and lines, respectively. Thus, c(Uact, Man) can be further expressed as
d(er Pacts Paltt
(Distance between points)
d(er Pacts Upatt van))
(Distance between point and line)
c( (uat, Usll)
(2)
dil(er Pact, er Vact), Pan) a a(Vaet, Pal)
(Distance between line and point)*
(Angle difference between axes)
where i(, v) is the line formed by the point and the axis v, d(-) is the function to calculate the distance between points or between a point and line, and a(-) is the function calculate the angle difference hetween two axes. Wheg ||0r - @nominatl |I represents the regularization term, which biases the optimization toward the robot's safe configuration. The nominal angles Onominal can generally be predefined using historical data or expert knowledge, such as On and Odefautt The pose er of the end-effector must satisfy the constraints derived from the forward kinematics solution fix(Or), The interval [Clower Cupper] represents the upper and lower bounds of the relational constraints. Finally, @colliston free denotes the set of values that will not result in collisions. We employed the SNOPT solver from the pydrake library to solve this optimization problem.
The sccond step is to (ii) calculate e, and 6,, when < t < T. To address this problem, we utilize the constrained motion planner from the mplib library to minimize the cost function Cost(O:) while guarantecing the satisfaction of the constraints during the movement process. The optimization problem can be formulated as
e. Vt
(Kinematic constraint)
arg min Cost(O.),
s.t.
c(Uact, Vali) (Clower; Cupper], Vc Crath
(Relational action constraints), (Collision avoidance)
0, E Ocollision free, Vt [1,T'-
(3)
where Cost(O:) typically incorporates objectives related to motion smoothness, energy efficiency, and trajectory optimality.
16

---

Original Scene (tabletop-level)
Scaled Scene (room-level)
0
Figure 8: Schematic diagram illustrating the scene scaling of the handover and storage task, extending from a desktop-level scenario on the left to room-level scenario on the right, aimed at enhancing scene diversity in the dataset.
A.3
Scene Scaling
To scale tasks from tabletop-level scenes to room-level scenes, introduced in $2.3, coordinate transformations applicd to align the scene configurations. Specifically, in the original scenc, a point is selected the table edge near the robot the origin o construct Cartesian coordinate system O xyz. The x-axis the direction the robot faces toward the table, the y-axis extends from ó along the table's edge toward the robot's left-hand direction, and the z-axis extends vertically upward from o. Correspondingly, in the new scene, an edge point in open desktop area is selected as the origin o' to construct another Cartesian coordinate system O'-xyz, with x, y, and directions consistent with those in O-xyz. Thus, for point p. vector v, and pose (3) the world coordinate system the original scene, and their counterparts p', and P E (E(33) in world coordinate system of the scaled scene, the following relationship holds:
points
a' To-oor a, where E (,, a' (''
(4)
Based the coordinate transformation defined in Eq. (4), the spatial relationships of objects, robots, kcy points, axes original system O can be directly mapped the target system O'. This allows reconstructed poses (e.g.. object configurations, relational action constraints) to inherit geometric consistency from the source scene, enabling seamless task execution in the scaled environment. By leveraging this lightweight approach, our dataset diversity is enhanced without requiring additional annotations or complex geometric reasoning.
A.4 MCTS-Baie Based LLM Reasoning
Selection. In each iterative loop, the step begins with selection to find the node to expand. In the selection
step, starts from the root node and explores downwards. When node is explored, the algorithm decides
whether to continue exploring one its children or to end the exploration and select the node for expansion. To
define the action to expand from the explored node, we introduce special branch S' = 0 as proposed in [25]. Here, S" is the branch defined in $2.3, which consists of multiple operation steps S with consistent intents. The choice of continuing exploration and expansion is decided by
SelectPolicy(n))
=
arg max S (C))ldr
QULCB(T, S'),
(5)
where we use QDUCD as the value estimation method to balance the values of exploration and exploitation, as referenced in [25]. QDUCB uses a discount y E (0, 1) to smoothly drop those outdated feedback records and give greater weights to the feedback from the latest backpropagation during value estimation. The process can be described as
QDuCD(n,S') Wn,n,S')
ln E (Childee(())))(() (n, (n S¹)
(6)
Nn
N(n.s') Σ
Wy(n,))=
N(n N(n. -
1=1
N(n.8')-1
N,
Σ
(8)
1=0
N(n,S')
[[((n
(9)
17

---

where Tr {Vtrone, n n)),, (1e()) (n.l).... = ne, } denotes the selection trajectory of I-th iteration that ends with as the expanding node. S") = (n, S") E 7} denotes the set of 7 that contains (n, S") I((T) represents the value the trajectory т, which is determined in the backpropagation stage. N(n, )') denotes the number of trajectories I that contain (n, ). W Wn,, 5') denotes the total value of trajectories that contain (n, S"). (n, S") and (n, S") are both results decayed by 7
Expansion. Using the information stored in the exploration node, such as executed code, non-executable code, and the node's runtime scene information, a prompt constructed and fed into the reasoning LLM. Then the LLM infers new executable code. Following the STCR mechanism proposed in $2.3, a subtree is built with the selected expansion node as the root. When expanding, if the next exccution branch Swe under the current expansion node Nnow already exists, i.c., Swe Children (New)) ). new node will not be generated, and the expansion will continue along the existing branch. If the task successfully completed, the MCTS process ends, and the executed code concatenated to the generated code to form the final successfully executed code. If the task remains incomplete, the iterative loop continues.
Backpropagation. Backpropagation is the final step of the iteration. It aims to update the reward R(T), where 7 the selected trajectory in this iteration. Since iteration ends once successful task plan is discovered, the extrinsic reward contains Rextrineic = during the iteration process and cannot be used as a reward. Therefore, we propose an intrinsic exploration value Rusvinsic based on 'valuable moment', which refers to the occurrence of a valuable event, such as "successfully grasp cup', 'pinch cube", 'the object does not fall off when the hand is opened", etc. If the generated code contains at least one 'valuable moment' during execution. then we set Rintrinsic 1 for backpropagation to incentivize the expansion of the LLM.
B Details of HGen-Bench
Benchmark Task Design. Our benchmark builds on ManiSkill3 [52] physics engine. In the simulation, the maximum control frequency can be set to 100 Hz, and the maximum camera capture frequency is 100 Hz. Our tasks span various scenarios. The details are given as follows. (i) Atom operations. To fully leverage the dexterity the hands, we design interaction modes such as pinching and grasping. For example, stacking small blocks requires pinching them with the index finger and thumb, whereas picking up a bottle requires a firm grasp. (ii) Task difficulties. To reflect varying levels of task difficulty, we define difficulty scttings and name the tasks with "casy" and 'hard' to distinguish. For instance, in the dual bottles pick task, the "casy" setting involves picking upright bottles, while the 'hard' setting involves picking fallen ones. (iii) Collaboration modes. We also design bimanual coordination tasks that exploit the capabilities of dual dexterous hands. For example, the block handover task involves picking up a block, handing it over from hand to hand, and placing it. (iv) Long-horizon and geometric reaxoning. We design set of challenging tasks to evaluate these capabilities. For example, the long-horizon task handover and storage involves a sequence of five steps: the right hand grasps block and positions it for transfer, the left hand opens a drawer, then takes over the block, and finally places it into the drawer. We also design tasks requiring geometric reasoning. In blocks stack hard, the robot must stack three blocks vertically. In pyramid stack, two blocks must be placed on the first layer. with a third block stacked on top to form the second layer. To introduce variability, randomize the initial position, pose, and joint angles of articulated objects within certain range. This allows us to collect more diverse data and test the generalization ability of learned policies.
Policy Training Examples. We provide example code for deploying and evaluating different types of policies. In our experiments, we present the training and evaluation results of these policies. The dataset we collected includes first-person RGB images captured by an Intel RealSense D435 camera, aligned depth maps, and point clouds generated using Open3I,, as well as joint angles as part of the observation data. We use first-person RGB images and joint angles as inputs to DP. For DP3, we crop the point cloud to retain only the workspace, i.e., the hand, arm, and the object to be manipulated. The point cloud is then downsampled to 1024 points. This sparse point cloud, together with the joint angles, is used as the input DP3.
We train both DP and DP3 using 100. 50, and 20 trajectories generated by our method. During each training session, we record results from three checkpoints. For DP, we evaluate checkpoints at epochs 150, 200, and 250, while for DP3, we evaluate at epochs 1500, 2000, and 2500. For evaluation, we test each of the three checkpoints using sceds 0, 1, and 2, and report the mean and standard deviation of the success rates.
The full success rate results across 20 tasks are shown in Tab. 3. For relatively simple tasks, DP3 exhibits few-shot learning capabilities when trained with data collected using our method, achieving high success rates with few 20 trajectories. For moderately difficult tasks where 20 trajectories yield insufficient performance, increasing the number of trajectories consistently improves success rates, indicating the effectiveness our data scaling strategy. However, for several extremely challenging tasks, although our data generation method demonstrates high-quality demonstrations, neither DP nor DP3 is able to accurately learn each step. These tasks often involve long- horizon sequences and fine-graines object manipulation. We anticipate that future methods
18

---

Table 3: We present the DP and DP3 results for all 20 tasks using 100, 50, and 20 trajectories generated by our method, and evaluate the success rates across 14 tasks using 3 random seeds.
Num of Demonstrations
20
50
100
20
50
100
Blocks Stack Easy
Close Drawer
DP3
0.0=0s
0.0+00
22.8 22.8+165
DP3
83.3 95.6a33
94,4279
92,6+11
DP
0.0
0.0:uo
0.0:00
DP
100.0scu
100.0
Cup Pour Easy
Dual Bottles Pick Easy
DP3
67. 07.8+138
75.6+9
72 2012
DP3
75.9=118
96,3+66
93 33,9
DP
0.0:mo
2.2
0.0sta
DP
0.0gua
0.0amo
0.0ano
Dual Bottles Pick Hard
Emply Cup Place
DP3
88,9
90.7=114
94,4a79
DP3
25 0..0
18,3+47
33,3e31
0.0.mo
0.0am
0.0gma
0.0gua
0.0gao
6.7
Open Box Easy
Open Box Hard
85.6280
95 05.6444
95.0=41
DP3
95.6+38
16.1
98.3=33
DP
93.3
100.0aca
100.0au
DP
11.1
93.32M4
100.0
Open Drawer
Open Luplop Hard
DP3
58,3+8,3
76.0=13.
84.4+115
DP3
100.0ea0
100,0sa0
100.0ea0
DP
17.8
13.3gm
48.9
DP
15.6
11.1
35.6.124
Close Hard
Laptop Easy
DP3
88. 17.9
96. 96.316
963326
DP3
100.0ga0
100.0.on
100.0.aa
DP
222.2.32
-51.
11.s
DP
37.8
40.0g
48.9.381
Hundover and Storage
Blocks Stack Hard
DP3
0,0
0.0
0.0gna
DP3
0.0gne
0.0ano
0.0an0
0.0,mo
p.0,u
0.0.na
0.0.ua
0.0,mo
0.0100
Block Handover DP3
Close Box Easy
0.0.00
0.0.00
0.0gna
DP3
100.01.0
98. 3133
99.4
DP
0.0_mo
0.0.00
0.0.ma
DP
97.8...
100.0.cm
11.1
Close Laptup Hurd
Handover and Storuge Cooperation
DP3
92.6481
94.4279
963336
DP3
0.0gne
0.0
0.0an0
DP
446.7
33.31%67
DP
0.0.no
0.0
0.0,00
Open Laptop Easy
Pyramid Stack
DP3
71.1557
77.2475
1194
DP3
0.0
0.0
0.0ano
DP
775.6_
"71.1
60.01163
DP
0.0,no
0.0..0
0.0100
that are capable of learning long-horizon behaviors will help address these challenges. As discussed in the main text, during DP training, some tasks show abnormally high success rates even when the policy has not learned valid strategy. We mark such results with **' the table. For example, the task Close Box Hard, with only 20 demonstrations. the arm tends to make rapid and random movements, occasionally closing the box lid by chance, which results in deceptively high success rate. However, this behavior is unstable and unsafe. As the number of demonstrations increases (e.g., 100 demonstrations), the arm no longer moves randomly and instead attempts perform a deliberate closing motion. In this case, the hand needs to approach the lid from behind. Due to DP's lack of spatial understanding and perception, the hand often collides with the lid, failing to maneuver behind it and ultimately leading to lower success rates.
Decision. We adopt diffusion policy framework for both DP and DP3, which predicts a future sequence of H actions conditioned faat steps of past observations. At inference time, the last N.:: H -ohs + predicted actions are executed to form the final control trajectory. In our experiments. we set H and Maees 3 for both DP3 and DP. While both DP and DP3 share the core principle of conditional denoising diffusion, they differ in input modalities and policy architectures.
DP3 processes 3D point cloud inputs using a DP3 encoder, where cach frame consists of 1024 points with 3D coordinates. The encoded feature has a dimension of 128. The architecture incorporates Feature-wis wise Linear Modulation (FiLM) at the down, mid, and up layers of the convolutional UNet to improve conditional representation learning. contrast, DP operates on image-based observations alongside low-dimensional proprioceptive inputs. The visual encoder is implemented using the Multi MultiinmggeDsBEccodee module, which supports multiple RGB observation keys, each associated with its own 8 backbone (without pre- preitrained weights) shared model depending configuration.
Both methods adopt noise schedule with 3mm 0.0001, Ban 0.02, and squared cosine schedule (squaredcos_cam sap__aapp_______ v2) over 100 diffusion training steps. However, the sampling strategies differ: DP3 employs DDIM (prediction (prredcttion_type=ammpl.. type=sample), while DP uses DDPM with e-prediction (prediction_tyyy pereeiitiooottpeeeeeelee.e. type=epsilon) and enables clip_sample to stabilize training.
Training Setting. For both DP and DP3, we adopt exponential moving average (EMA) stabilize the training process. The EMA parameters updated with inv anv_gamma 1.0. power = 0.75, and max_value 0.9999. Since BatchNorm not compatible with EMA, all normalization layers are replaced with GroupNorm to ensure training stability, particularly for DP with image inputs. Both policies use the AdamW optimizer with a learning rate of 1 x 10- B = [0.95, 0.999], and a weight decay of 1 x 10-6 A cosine learning rate scheduler with a linear warm-up of 500 steps applied improve convergence during the initial training phase.
19

---

Table 4: Randomization Settings
Prediam Romdomiiaaion
Angle Randomization
Posiison Ramdbmization
Angle Remdoniization
Blocks Stuck Easy
[-0205, 0,00,, 0,02)
15)
Close Drawer
[0.05, 0.05, 0.18, ..18)
[25,23
Pour Hasy
H-00, 010,, 4014,
L-0,
Unal Hottles Peck
10,005 WMB5, 4105s, (0005)
10, 0)
Doal Bottles Pkc Hard
10J06, 0,025, 1025, 0.033)
[0.0]
Empty Cup Place
10,02, 0.02 0.04, 0.04]
300
Open Box Eay
L00,, 000 4004,, 0.84)
[ 10
Open Box Hant
[400,, 0.00, 4004 0.00)
[00 W]
Upen
HUBS, 0055, 0:18)
25]
Upen Lugtop
[400,
[M0, 30)
Closs Box Hand
(0055, 0.05, 40 18,
[-25, 23]
Clovs Laplup Easy
[-005, 0,05, 0,18)
[-25, 25]
Handover and Storage
[0,01, 0.01,
0]
Blocks Stack Hard
[-0005, 0,005, 00002 (.02)
Handever
[-0,08, 0,005, -0,05, 0.005)
[0. 0]
Box Easy
[-0.55, 0,05, 15, 0.18)
[-25,23]
Hard
E005, 0.5,
Handover and Cospecation
[-001,
Open Laptup Eas
[a.0i,000 6.01, 0.04, ...44)
[:30, 10]
Pyramid Stuck
[-0.22, 0.02, 0.04, 0.04]
1.50, 30)
DP3 trained for 3000 epochs with checkpoints saved every 500 epochs. contrast, DP is trained for 300 epochs with checkpoints saved every 50 epochs. The batch size is set to 256 for DP3 and 64 for DP, reflecting the difference in memory requirements between point cloud and image inputs. Data loading includes shuffling for training and deterministic ordering for validation, with pan__mmory-True to improve performance.
Randomization Setting. In the data collection and policy performance evaluation, We defined the ran- domization ranges for object positions and orientations, as shown in Tab. 4. Specifically, Position Randomiza- tion [21, 22; y1, ye] (in meters) denotes the range of randomized positions, where 21 and 22 represent the bounds for the x-coordinate, and y1 and y/2 represent the bounds for the y-coordinate. Angle Rundomization [0,, 0z] (in degrees) indicates the range for orientation randomization. To accommodate the capabilities of different algorithmss, a small subset tasks (DP3 vs. DP) adopts slightly different randomization ranges.
C Experimental Details and Additional Experiments
C.1 Real-World Experiments
To further explore the application of HumanoidGen in real-world settings, conducted experiments focusing on both data generation sim2real transfer.
Figure 9: Visualization of real scene and simulation scene.
Experiment Setup. As shown in Fig. 9, we set up a real-world environment that closely mirrors the simulation environment. This includes a white table and humanoid robot performing bimanual dexterous manipulation tasks. The robot used is the Unitree H1-2, equipped with D435i depth camera mounted on its head, with resolution of 640x480. The robot's dexterous hands the same Inspire hands used in the simulation environment, For the tasks, we selected the dual bottles pick easy and close laptop hard from the (GGen-Bench for real-world transfer, using physical assets that match their simulated counterparts. The object poses in each trial were randomized to test robustness. Specifically, for the dual buttley pick eaxy. the placement was randomized within range of x em and +15" in orientation, while for the close laptop hard, the randomization range was 6 em 6 cm and +10 to +20 orientation.
Experimental Procedure. The experiment consists of three main steps: (i) Real2Sim Pose Estimation. Using the existing digital assets, we first estimate the current poses of objects in the camera coordinate system with FoundationPose [56]. The estimated 6D poses are then transformed into the world coordinate system, allowing the corresponding digital assets to placed the same poses in the simulation environment. (ii) Trajectory Collection. We leverage LLM reasoning and the trajectory generation module of our framework to automatically generate execution trajectories in simulation, which are then executed on the real robot to collect real-world data. For each task, collected 25 real-world trajectories. (iii) Policy Training and Sem2Real Transfer. To evaluate the effectiveness of the collected real-world data, train diffusion policy models using 5 and real-world trajectories, respectively. In addition, investigate the contribution of simulation data to real-world policy performance, we augment the real- world dataset with additional simulated trajectories. Specifically, for each task, we train and evaluate policies using 5 real-world trajectories combined with 100 simulated trajectories, aiming to analyze how simulation data improves real world performance.
20

---

Table 5: Experiment Results. DP represents the diffusion policy success rate under different training
settings: 5 real trajectories (5R). 25 real trajectories (25R), and 100 simulated + real trajectories
5100S+5R)
Tisk
Collection Success Rate (%)
Collection Time (s)
(5R)
DP (25R)
DP (100S+5R)
Dual Bottles Pick Easy Laptop Hard
85
16
0S (U/20)
60% (12/20)
65% (13/20)
(4/20)
(14/20)
(14/20)
Experimental Results. We evaluated two tasks, dual bottles pick easy and close laptop hard, in real- world experiments. The metrics include the data collection the performance
rate, the average collection time per trajectory. and [43] models trained under different data settings. As shown Tab. the average time collect a single real-world trajectory is only about seconds, and the data collection success rate exceeds 85% for both tasks. The results of DP model training demonstrate that policies trained with 25 real-world trajectories significantly outperform those trained with only 5. This indicates that the success rate increases with the amount of real-world data. Furthermore, pretraining with 100 simulated trajectories followed fine-tuning on woaldworld trajectories achieves comparable even better performance than using 25 real-world trajectories alone, particularly in the dual bottles pick easy task.
C.2 Automatic Asset Annotation Evaluation
To assess the effectiveness of our automatic annotation method based on annotation migration, we conducted quantitative analyses.
Cup (rim)
Cup (handle)
Bowl
Drawer
Laptop
Figure 10: Visualization of annotation migration.
Experimental Setup. illustrated in Fig. 10, we selected parts from four commonly object types for automatic annotation. The cup contains two parts: the rim and the handle. For each object category, 20 assets were randomly selected from [14, 24. 19J. The rendering and reprojection processes for automatic annotation were implemented using ManiSkill3 [52], the same simulation environment
data generation framework.
21

---

Table Evaluation of annotation time and success rate.
Evaluntion Metric
Cup (rim)
Cup (handle)
Bowl
Drawer
Laptop
Manual Annotation Time (s)
4,00
4,00
4,00
6.00
6.00
Automated Migration
Migration Success Rate (%)
100.0% (20/20)
90.0% (18/20)
100.0% (20/00)
0% (18/20)
100.0% (20/20)
Experimental Results. We compared the annotation time between manual and automatic annotation and evaluated the success rate of automatic annotation, as summarized in Tab. 6. For manual annotation, simple rigid objects such cups and bowls required about 4 seconds per part, while articulated objects like laptops and drawers took around 6 seconds. In contrast, our automatic annotation required only ahout 0.7 seconds per asset on average, achieving substantial acceleration. The success rate of automatic annotation was also high. reaching 100% for the cup rim, bowl, and laptop, and over 90% for other parts. The few failures, such as those for the cup handle and drawer, were primarily caused by significant geometric variations, such as differences in handle shapes between circular and rectangular designs.
C.3 Additional Challenging Dexterous Manipulation Tasks
To further evaluate the scalability and robustness of our framework beyond the 20 main tabletop manipulation tasks, design five additional challenging tasks focusing bimanual coordination, dexterous contact-rich operations, and complex tabletop environments, as illustrated in Fig. 11.
Blocks Stack Hard With Barrier
Rotate Safe Knob
Press Toaster
Open Safe Door
Dual Lift Pot
Figure 11: Illustration of the five additional challenging bimanual dexterous manipulation tasks used for supplementary evaluation: rotate safe knob, press toaster, open safe door, dual lift pot, and blocks stack hard With barrier. These tasks involve coordinated bimanual motions, contact-rich dexterous operations, and complex tabletop environments, highlighting the scalability of our framework.
additional tasks include rotate safe knob, press toaster, open safe door. dual lift pot, and blocks stack hard with barrier. In rotate safe knob, the robot precisely pinches and rotates a knob, requiring accurate fingertip control to prevent slippage. In press toaster, the robot presses switch using single finger, demonstrating finc dexterous motion beyond grasping pinching. The open safe door task involves pulling open door with vertically oriented hinge, testing manipulation of articulated mechanisms, In dual lift pot, the robot must grasp both handles of a pot simultancously and lift testing coordinated bimanual manipulation. Finally, blocks stack hard with barrier requires stacking blocks in an environment where more than 60% of the workspace is occupied by barriers, posing severe spatial constraints.
Experimental Setup.
Table 7: Evaluation of additional challenging dexterous manipulation tasks. Each task is tested under randomized initial positions and orientations.
Metric
Rotate Safe Knob
Press Toaster
Open Safe
Dual
Blocks Stack Hard With Barrier
Randomization cem cm,, +15 5em 5cm, +15 5em ccm, +15
6cm, 93.00
6 &cm, +0 73.00
Success Rate(
91.09
87.72
95.24
Experimental Results. As shown in Tab. 7, framework achieves high success
all five tasks, with three exceeding 90%. These results further validate the scalability of our method handling bimanual coordination and contact-rich rich dexterous manipulation. Even the most challenging case, blocks stack hard with barrier. where over 60% of the workspace is blocked by obstacles, the framework still achieves 73% success rate, outperforming existing baselines under comparable conditions.
22

---

C.4 Resource and Efficiency Analysis of HumanoidGen
We analyze average computational and token consumption across all 20 tasks in HGen-Bench, including both time cost and LLM resource usage. The detailed statistics are summarized in Tab. 8.
Table 8: Average time and resource consumption for each stage in the HumanoidGen data generation pipeline, computed over all 20 tasks in HGen-Bench. Manual annotation requires limited human effort, while all other stages are fully automated.
Metric
Manual Annotation Automatic Annotation
Generation
Script Generation
Data Collection Execution
Time (s)
4-6
1.17 Automated Process
12.64
18.31
14.40
Resource Cost
Human Operator
LLM (2,958 tokens avg.) LLM (3,745 tokens avg.) Motion Planner (0.53
is important to note that generating Depending the scenario, certain steps can be skipped or reused, significantly improving efficiency: (i) For pose randomization of manipulated ohjects, tabletop obstacles, or room-level scene extensions, the system can directly execute the previously generated running scripts to collect new trajectories. The randomization operations are executed at the millisecond level and thus have negligible time cost. (ïi) For tasks sharing identical tabletop setups but differing in manipulation procedures, such handover and storage and handover and storage cooperution. For these tasks, scene initialization can be reused, requiring only the generation of new exccution scripts. (iii) When different objects are used but all exist in the asset library, no manual intervention required; automated scene generation and code synthesis are sufficient to produce new executable scripts for data collection. Only when introducing
trajectory does not always require executing all stages of the pipeline.
object categories absent from the asset library manual annotation needed, which takes merely 4-6 seconds per asset class.
These results demonstrate that HumanoidGen achieves high efficiency in both automated data generation and resource utilization, enabling scalable and reproducible large-scale evaluations HGen-Bench.
C.5 Comparison with Existing Generation Frameworks
To demonstrate the advantages of our proposed framework, conduct comparison with several state-of the-art robot manipulation data generation frameworks, including both augmentation-based and zero-shot generation approaches.
Augmentation- Based Frameworks. These methods [51, 57] require existing expert trajectory as the basis for data expansion, which typically involves teleoperation- based data collection. Moreover, manual annotation is needed to segment long-horizon demonstrations into multiple sub-stages before augmentation can be applied. When the task execution mode changes, such as switching from lefi-to-right to right-to-left handover, modifying the stacking order in block stacking task, expert data collection and segmentation needed. contrast, our framework achieves object-level reusability and full automation without human intervention. enabling flexible and scalable task generation under varied manipulation settings.
Table 9: Comparison of different robot manipulation data generation frameworks. Only our proposed HumanoidGen provides unified and fully automated pipeline that supports bimanual dexterous hands, scene generation, room-level synthesis, and dynamic collision management.
Dexterous
Tabletop Scene
Room-level Scene Synthesis
Dynamic Collision
Framework
Bimanual
Hand
Generation
Management
RoboGen [22]
Yes
No
Yes
No
Gensim [23]
No
No
Yes
No
No
Gensim2 [27]
No
No
Yes
Yes
No
RoboTTin [19]
Yes
No
No
No
No
RoboPactory [58]
Yes
No
No
No
No
HumanoidGen((uur))
Yes
Yes
Yes
Yes
Yes
Zero-Shot Generation Frameworks. For frameworks that support zero-shot data generation without
relying on expert demonstrations, a comparison with HumanoidGen is summarized in Tab. 9. It can be clearly observed that HumanoidGen is the first framework to provide a systematic solution for bimanual dexterous hand manipulation problems. Other frameworks do incorporate dexterous hands end-effectors. RoboTwin [19] and RoboFactory [58] lack table-top top scene generation and room-level scene scaling, both of which are included HumanoidGen. demonstrating the comprehensiveness of our work. Additionally, only our framework includes dynamic collision management by LLMs, enhancing the flexibility and effectiveness of our framework in handling collisions during long morizon tasks.
23

---

C.6 Comparison Across Different Large Models
To further investigate the impact of different large models serving as the planner within framework, conduct a comparative study involving a reasoning model, chat model, a multimodal model, both with and without visual input. Specifically, we evaluate DeepSeek-R/ (reasoning model), DeepSeek-Chate Chat-v3 (chat model), and GPT-4o in both its language-only and multimodal configurations. Four representative tasks, each from distinct category in Fig. 5, are selected for this experiment.
Deepseek
Despseck
angguge only)
(Multi modall
Generation Success Rote (%)
100-
75
25
Draver, zingle
Laptop single comptex colision scenarics
Storage, bimarual, long-horizon,
aimanusi, motion spacee
mosio space
complex collision scenorics
Figure 12: Comparison of different large models used the planning module in our framework: reasoning model (DeepSeek-R) RI), a chat model Crepseek-Chat-Ve and multimodal model with and without visual input (GPT-40 language-only and GPT-4o multimodal).
As shown in Fig. 12, all four models achieve high success rates in generating valid constraints and exccutable planning code. Specifically, GPT-44o (with and without image inputs) shows consistently good performance across all four tasks, whereas the Deepseek reasoning model and the Deepseek chat model have slight perfor- mance decline when generating plans for bimanual and long-horizon tasks. The consistent high performance of these different models across various tasks demonstrates the effectiveness of our framework and compatibility with diverse models.
C.7 Task Descriptions in Data Generation Experiment
We describe our 20 tabletop-level tasks in detail in Tab. 10. The initial positions of target objects in all tasks randomized. The tasks include 12 single-arm tasks and 8 bimanual tasks. Dexterous hands are used as end-effectors in all tasks. For bimanual tasks, the appropriate arms to manipulate target objects selected according to distance between the arms and objects. Some tasks involve object handoff' between two hands, like block handover and handover and storage. Furthermore, handover and storage cooperation requires the coordination between hoth arms to complete the task.
C.8 Task Descriptions in MCTS Experiment
As mentioned in $5.2. to evaluate the effectiveness of MCTS in enhancing the demonstration generation process of HumanoidGen, we simplified task descriptions to provide minimal guidance in the experiments. As shown in Tab. 11, did not notice the operation order the operation executor, both of which inferred by the reasoning LLM.
D More Discussion on Limitations and Future Work
Despite the promising results achieved by our framework, several limitations remain. First, human intervention has not been completely eliminated. For asset categories and atomic manipulation types that have not been previously annotated, automatic labeling through annotation transfer instead. Consequently, generating a task that involves novel asset category still demands prior human annotation its manipulation primitives. However, we note that several recent works have proposed learning-base -baed methods for predicting contact points, axes, or grasp poses [59, 60, 61]. Although these approaches are not yet fully mature and still face generalization challenges, they demonstrate certain zero-shot capabilities that could be integrated into annotation pipeline. Combining these advances with generative framework holds promise for fully automated data generation, which we plan explore future work.
possible, requiring manual annotation
Second, our current framework cannot yet generate all types dexterous manipulation tasks, On one hand, due to the limitations of the ManiSkill3 [52] physics engine, tasks involving deformable or fluid objects cannot be simulated. On the other hand, our arm control currently relies low-level motion planner, making it difficult handle manipulation tasks with ambiguous dynamic objectives, such push-T cloth flattening. These

---

Table 10: The descriptions of our 20 tasks.
Task
Description
rectangular cuhe is placed on the right side of table. The right hand pinches cuhe and moves hand over the left hand. The left hand then pinches the cuhe and moves the cube above the target cube to release it.
Block Handover
Tho culees, cuhefl and cubel, are placed on the table. Inirially, both the left and right hands simultameeasly pinch cahef) and cubel, respectively. The left hand then put cubell specified position. Following this, the right hand moves cubel to pot it above cubel.
Blocks Stuck Easy
Three cubes, cubell, cubel, and cuhe2, are placed oe the table. Inirially, the left hand pinches cubel) and right hand pinches cubel simultangussly. The left hand puts cubell at the target place. Then the right hand puts cubel above cubell.
Blocks Stack Hard
Finally, the right hand picks up cuhc2, pats ahove cubel.
box placed the table. Initially, the right hand approaches the box lid from above. Then, the right hand moves Blip the box lid down.
Close Box Easy
placed on the table. Initially, right hand grasps the hox lid, While mointaining its grasp, right hand adjusts the box lid's Is's opeaness to 0.3 and releases it.
Close Box Hard
A drower is placod on the rahle. Initially, the left hand moves to graep the drwer handle. With the left hand attached to the drawer hundle, the left hand pushes the drawer back openness of Finally, the left hand releases the drawer handle.
Close Drawer
laptop placed on the table. First, the right approaches laptop screen from above. Then, right hand moves to flip the laptop screen down.
Close Laptop Fasy
A laptop is placed on the table. First, the right hand grasps the laptop screen from above. While maintaining its grasp. the right hand adjusts the laptop screen's openness 0.3. Then, right hand releases the luptop screen and above it. Finally, the right hand moves to flip the laptop screen fally down.
Close Laptop Hurd
Cup Pour Easy
bowl. Finally, the right hand tilts the
A cup and a bowl are placed on the table, First, the right hand grasps the cup and moves it above and in front of the pour its contents into the bowl.
Two bottles, bottle0 and bottle are placed on the table. The left and right hands simultancoussly grasp the two bottles lift them to target position.
Dual Bottles Pick Easy
Two bottles, bottle0 and bottle are placed on the table. The left and right hands simultaneously grasp the two bottles and lift them to the target position.
Duul Bottles Pick Hurd
Empty Cup Place
A cup and plate are placed on the table. The task to use one hand to grasp the cup and place directly over the plate.
drawer a rectangular cubeo are placed on the table. First, the left hand grasps drawer handle and pulls to
an openness of 0.9. Thon the left hand releases the drawer handis, Next, the right hand pinches the cube and moves it to
Handover and Storage
hand over the hand. The left hand
moves to pitch
cuhe, taking from the right hand. left hand
moves the cubc above the open drawer and releases it to put into the drawer. Afterwards, the left hand moves to grasp the drawer handic again, pushes it back
openness of and finally releases the drawer handle.
A drawer and a rectangular cubei) are placed on the table. First, the left hand graps the drawer handle, and the right hand pinclees cubci simullaancussyy Then, the right hand moves the
an intermediatc position while the left hand polls the drawer out to an openness of 0.9 at the same time. The left hand then releases the drawer handle. Next, the left hand moves to pinch the cube, taking from the right hand. The left hand moves the cuhe above the drawer and releases to put into the drawer. Afterwards, the left hand moves to grasp the drawer handic again, pushes it back to an openness of 0, and finally releases the drawer handle.
Handover and Storage Conperation
A box is placed on the table. First, the right hand moves be under the box lid. Then the right hand moves to flip the box lid up.
Open Box Easy
box placed on the table. First, the right hand adjusts the box lid's openness to 0.8 and releases After this, the right hand releases the box lid and moves to be under it. Finally, the right hand moves to flip the box lid
grasp the box lid. While maintaining grasp, the right hand
Open Box Hard
A drawer is placed on the table. First, the left hand moves to grasp the drawer handle. With the left hand grasping the hand releases drawer handle.
Open Drawer
drawer handle, pulls drawer out an openness I. Finally,
laptop placed the table. Finst, the right hand moves to be under the laptop screen. Then, the right hand moves flip the laptop screen up.
Open Laptop Easy
A laptop is placod on the table. First, the right hand grasps the laptop and flips it up to an opennces of 0.55. Then, the right hand releuses the initial grasp and moves to he under the liptop screen. Finally, it moves flip the laprop screen to be fully up.
Open Laptop Hurd
Three cubes, cubel), cubel, and cubc2, are placed on the table. First, pinch cubeO and cubel. I. respectively. The left hand then moves cubeO to it at the turget position. After this, the right hand moves cubel to it the right side of cubel). Then, the right hand pinches cube2 and moves it to put the middle and above cubell and cubel.
hand and the right hand simultancously
Pyramid Stack
tasks involve continuous state adjustments rather than planning toward single target pose, which cannot be realized solely through motion planning. In the future, we plan to extend our framework to support more diverse physics engines, such IsaacSim [15] and MuJoCo [16], enabling richer simulation scenarios. Moreover, leveraging the extensibility of our framework, we will explore combining goal-conditioned motion planning with goal-agnostic pre-traine trained atomic operation models, forming a unified library of atomic operations to support a wider range of dexterous manipulation tasks.
Finally, we plan to expand Beeen-Benchhae with additional assets, scenes, tasks, and evaluation policies to enable more comprehensive benchmarking and broader applicability.
25

---

Table 11: The descriptions of tasks in the MCTS experiment.
Description
Single
top
Blocks Stack Easy
Stack cubes table single pile.
Blocks Stack Hard
Stack three cubes the table
Pyramid Stack
Stack the ihree cubes pyramid shape the with cuhes bottom and cuhe stacked
E Prompts and Generation Samples
We show the prompt templates and the sample code generated by LLMs in generation and demonstration script generation (with and without integrating MCTS). The meanings of placeholders in prompt templates are explained in Table 12.
Table 12: The placeholders in the prompt templates and their meanings.
Placeholder
Meaning
ASSET INFO
The information of the assets in the asset library.
ASSETS ATTRIBUTES
The default attributes of the assets.
asset states when the scene is initialized, before any actions are exccuted.
INITIAL ASSET STATE
CURRENT_ASSET STAATE
The current states ussets on tabletop.
Code for actions exccuted by the robot transform the assct state on the tabletop from the initial state to the current state.
EXECUTED CODE
PROHIBITED ACTION
The actions prohibited for the next step.
ASSETS STATUS
The current states of the assets in the task
ROBOT END _ED_FFFCTOO
The current state of the robot end-effector.
E.1
Prompt Template for Scene Generation
You are a professionsl AI simulation environment code generation assistant capable and
of generating logical reasoning monologues for the initial scene of the task based on the user's given task, i.e., scene is designed this code construct the acens.
code. You need generate
to generate why initial Afterward, provide an anaver that includes the
Scene information (all coordinates are in the world coordinate systen): Dual-ars rabot, pose.p-[-4 06,0,0,, pase. pose.q-(1,0,0,0] Table surface, "x from -0.42 to -0.19, from -1.1 to 1.16, 2-0"
Available agsets:
ASSETS_IXF?
10 Code Example: Task to place two cubes side by side. ""python
12 from bunanoidgen. enva, example. task eny import # Inport necessary libraries
tregister_ent ("place_call cubes_ cacbe__sdde_bae side_by_side" aax.oppidde_tttpp=2000 environnent, name can be aet based the taak Muet inherit from TsbleSetting options: Dict):
Register
14 class PutkkoCuehdjeeetEaacc (TaeeiEa((T)leeSttting))) saaace_cube__eideeeee def _load__ceee auper 10___eod__ccae (oppttacaa)()))) self._add_ol object yellow (type_oxme-"cube"" type_id-1) name=""ube", obj_id-O,
16
Load objects
18
19
self gelf._add_objer "cub_" type_id-0) name=" od-_ green cube
def _initialize episade (self, env_idx: torch. Tenser, aptions: Dict): : Set
object positiona
26

---

22
() wtw.iiinnaalieeee....oo apiaada Lenx_idx, optiona) self _e__ pose "typee_aaee""ulle obj_id-O, pose-saplen. Pose
23
p- p0 0.28. 0.02]. 9- O, 0, 0]
25
27
gelf. set _et_j_eeeeeeettttece___eeeeee pose "sspe_aane""cube", obj_id-1, pose-saplen. Poze p-[-0.32, -0.32, 0.02].
eet
29
q-11, o, 0, 0]
30
)
32
def check chack_ auccesscccssssseee (self):
34
po - self cube [0] pose numuy () [0]
p1 self [i]- w.w..p..uupp............pppp...p [0]
36
0.02,0.02]
37
target_ purges_poritionn111000
-0.02.0.02]
ops [0.03, 0.03]
auccess_0 (p0b target.poattion.((51) <= apa for in range
(2))
40
success_1 (abb(((p1 (2))
Satet_poocition_(11 <- eps [i] for in range
41
print(*cub cucube [0] position:", po)
42
print (*cube [0] target position:", target.porition.0)
print (*cube [1] position:", p1)
44
print (*cube [t] target position:", target.position..1
print ("success_
success.0)
46
print suuc success_1)
return success_0 success_1
48
Notes:
'type_id" represents different models of the same type of object, while 'obj_id' objects
indicates the order 52 The left hand's operable range is [-0.42, -0.19],
type added.
[-0.07, 0.36]; the [-0.42, -0.19], y [-0.36, 0.07]. Objacts must placed within these intervals (note: meeting the sufficient condition
right hand's range
is necessary but not orientation, collisions, auccessful grasping may still fail due to abject gripper constrainta).
53 The robot's coordinate eysten aligns with the world frane: +I (frent of robot), robot), 54 The object's bounding bax (BBX) represents its dimensions in its oun coordinate orientation quaternion q= 0100011111
(up robot).
syaten (axis-aligned shen example, can with BBX [0.054, 0.1, 0.054] has lengthe of 0.054 unite along x/z-axes and 0.1 units along the y-axis.
56 Now please generate the initial scene of task "TASKKAME".
E.2 Sample Scene Code for Blocks Stack Hard Generated by HumanoidGen
fron lunanoidgen enva example................ task_ent import
dregister_enr btabockssstackhheee nax max_ max_opisooe__ttop==000
class Blocksstaccklardacctttttt (TableSetting): env_name- "blocke_stack_hezz" hard"
def Llaad_sceeeaserf options: Dict):
super so coneiooos)))))) self _dd__bjject (type_name"" type_id-0)
10
self adlff_add_object (type_nane=* cube", typo_id-1) add____dd__bjjcttcccc (typa_name=* cuba", type_id=2)
12
san.tttalllee.epsssdddelllll env
Tensor, options: Dict): (env_idx, optiona)
14
anperr()..iiittilllaae....
15
default_pose
16
Pogien.PoseCCe (p- (-0.22 02. 0.32. 0.05], 9-(1.0, 0, 0])
17
sapien Pose (p- 02. -0.32, 0.06]. q-(1, 0, 0, o)) ol),
sapien 450.45, -0.32, 0.05], q=[1,
19
relf.randon_scone. default, pofeult..ose--ell wew.lll.geeeraandddeeeee ret_rrnnonn.. pose pow.oonntttaatttttttttooooo (defaultt.ose-deetttl if not self randen_ once or (self. raado_raaddooocc and not hasattr (self, randon_pose
self malf.ranie rander__dn_ppee pose
self self._eet.cbjeecppoee
27

---

25
type_ type_nane _are__aa=""am came="cube",
26
obj -i-0,,
pose=self randon_pose
28
30
gelf ces.ceeee...j.cc.pooo.
type_neme= "eee_aanee"cce
32
obj-id-1,
[1]
37
type_nene- "umee-
obj_id-2 pose-self. www.on..rddd..pp.. pose
39
def Sback_auccesa (self):
print (*.....***... check,
44
positions -
self. [0] pose. pose.p.numpp cumpy ()
46
cuba. [1] p..e.p. numpy () [0]
47
self. cube [2) pose. numpy () [0]
49
poaitiona... aort (xay-laaaaad
[2])
epa_xy
51
eps_z 0.005
height_diff
53
xy_aligned =
abs (positions [c] [0] positions (0]) aps_xy and abs (positionn
[0] positions [i][t]] <= aps.xy and
55
abs (positions [1] [o] positions (o]) <- epa_xy and abs (positions [1] positione (2) (1]) (1)) epe_xy
56
58
abs (poaitions [1] [2] positions [0] [2] height_diff) <= eps_a and
59
abs (positioo [2] [2] positions [i] [2] height_diff) eps_z
61
return ay_ali and 2_aligned
E.3
Prompt Template for Demonstration Script Generation
professional assistant for generating perforn tabletop tasks. Please generate logically structured code execute the sper-ffedified tasks based on the provided context.
Scene information:
robot. pose.p-[-O.Bl 00,0,00, pose. q=1,0,,,,,000000 "robot_ barbot_base_lie connection betueen the "torso_ link" and the "pelvis". Table surface, x [-0.42, -0.19), [-1.1, 1.16], a=0
Assets ateributes (The default state of the assets. not the current state): ATTRIBTET
status
assets):
11 ASSETS_STATUS
12
13 Robot end-effector (vriet) current status:
14 ROBOT_END_EPPECTE
16 Available functions:
17 hand_pre_graap (hand_ nane):
18
Function: Adjusta the thunb novement of the specified hand to position it
opposite the index finger, preparing for subsequent graeping operations.. This should typically be called before executing 'nove_to_pose__ scce to reach the pre-gras grasp pose.
Return:
20
Arge:
hand_name (atr, optional):
(both), "right", "laft",
"all".
Example:
planner hend_ preeppprrrr""""efft" the pre- grasp pose
Control the left hand fingers to assume
24
planner. www.prre.rrapprrppppppppp Control the right hand fingera to assune
the pre-graap pose
28

---

25
defining planner pra..pre.grap(("al11"
def band_grasp (hand_nane grasp_object, obj_id);
28
Function: Control the hand to close.
Return:
30
Args:
(etr, Handatory): "right" "left'.
32
grasp_object (str, Mandatory): The type of the graaped object.
33
obj_id (int, Mandetory): The object id the graeped object.
Example:
35
planner . hand haadaagaaap""" ("lapt" cra
Centrol the
left hand planner. weg.grppp"righe -grsep.object""car objecct""c"n"" "beet="can,,obj_id_" obj_id-1) Control the graep the bottle
greap bottle
36
right hand
37
38
def cecllppropppeeee (hand_ name):
Usage: Similar the of the hand.pre.grasp function.
40
pinch Ceanch(Cun (hand_ pinch,object, obj.id): Usage: Similar to the usage of the head_grasp function.
12
44
def opan_hand (self, hand_name):
45
Function: Control the hand to open the fingers. Return: None
47
Args:
(etr, Mandatory): "right" "left".
49
50
cnneraae_constraiie
abj name obj_idd action hand_nano):
51
Function: Ganerate the constraints for the end end-effactor when perforning specific action.
Return:
53
Args:
obbj_anne
Mandatory): object
operated.
55
-ebj_id (int, Kandatory): The object id of the operated object.
action Can be "pinch", "graap", "target", "move", "targeti", "target2"
(str, Mandatory):
correspondang
constraints.
57
"pinch": Nove to the pese for pinching the object
"grasp":
pose grasping "target"; Move the "target" poze
object.
59
"targeti" pose "target2": Move the 'target2' pase "nove": Nove to spacific pose hand_name (atr, Mandatory): "right" or "left". openness (float, Optional): [necessary for openness constraint ganaration] opennesa the manipulated object after being nanipulated. should be between 0 or 1. This argnent is only used when the action is grasp" "pinch".
"targeti";
61
objact.
63
64
65
constraint relative_obj _ane (str, Optional): [necessary for relative generation] the nane of the object to used as reference for defining the relative relative_obj (int, Optional): Inecessary for relative pass constraint object
66
generation] id the relative pose.
relative_p array, Optional): [necessary relative pase generation] the relative position of the end end-atfector to the referance object.
68
Exanple:
Constreint_1-planae cenorate_constreeeeeeeeeeeeeeeee (dbj_aaeettt abj_id-0, action ""pinch", hand mam==="tt=""" band enf-ctfet to move to the pre-pinch poze for pinching cand constraint_l-planner
the conetraints for left
(aaarraattttaah__anneeoao can" obj_id-0, action ="grasp", "and__aae=")eft") # Generate the constrainta for the left hand end-effector to move to the pre-grasp pose for graeping can0 obj_id=0, action Genorate the constraints for the grasping target pose obj_id=1, action end-efiector (ot.eortttattttt....aaaaoeeeee". ott.rraat_conttssittooooooo
hea.ppaantt.ggnortte.oommm constraint_I-ploneee www.reeeeeonneeeeeeeeeeeeeeeeeee coraatt_oosstaittsoooottt [obj._aaeob ""target", band_came="left")
72
Constraint_r-plannee www.rgte.eonnaaeeeeaeeeeeee ="nove*, hand_naee""right".r relative_p-r (10,) weelatrre....._aaaee... Generate the end-effector pose for the right hand efter grasping cani to place it 6en above can0 slong the world coorinnaa systen's z-axis constroint_1-plannes copprrantt.--plamee www.aannnr.ggnratt...mm c cenerattcootrraattaaaa..... some imme_sstieelatod.oojoc.
relaattree.bb.__dd
73
obj_id-0, action="target1"" left") # Generate the constraints for the left hand end and-cttecoor nove to the targeti pose with the left hand pushing the object auttearrtiulatedeoom
hand_name-"
to along its articulation axis puutrriitt_rppannae iamorate_constraiti (obj_ sono_artieulattdddoojoo obj_id-0, action="target?"
24
hand_name="
29

---

right") * Cenerate the constrainta for the right hand end-effactin to nove the target2 pase with the right hand pushing the object artieulated__bbeee constraint Ipllanar
aleng its articulation axis (abj_
25
obj_id=0, action="grasp', nand_nam } Generate the constraints for the left hand end-effector to move pre-graep pose graeping object art.iculatedddhheectt
26
constreint_1-ppeener generate
wew.rate.cousszzaass.... constraints (ob_
abj_id-0, action= grasp'' nend_name""'eft"" opennens=0.77 Canerate the constraints for the left end effector pose graaping articulated.obJ.
constraint
to an plamning notion to turn the "toae_articulateddo000 openness
77
28 def wtwpsste_ed__eeeeteteeeeeeeeeeee (eessaat.. hand_
(constraints Function: Calculate the end effector pase that satiefies the constraints
Return: .. targett
81
Arge:
constraints (list, Mandatory)
83
hand_name (atr, Mandatory): "left" or "right'
84
Exanple:
torget_effector peoeee planner. ganeatte.ndd.fffectrr...o
constraint_ hand_ "ane=""a===""t"
87 def tove_to_posee wow...........icccrrroooooooooe
with sapien. Pose, hand_name, attach_obj-Fsles False.. abject_id=o): Function: Control the robotic arm move the and-affactor of hand_name" to the 'pose'.
88
Return: Args:
90
pose (sapien.
Mandatory): target pose
hand_name'
hand_nane (str, Mandatory):
"right".
93
attach_obj (bool or liat, Mandatory): Whether there is attached object in the hand during the movenent.
object_nama (str list): [Required "attach_obj' True] The type the attached object. object_id
Mandatory); (Required oattach_obj' True] The object id of the attached object.
Exampla:
97
If 'aand_name' is 'all', where index [0] and [i] carrespand to the paranaters for the left and right bands, respectively.
that other paraneters
1ist type,
98
you need to indicate the objects that are currently in hand (grasped or pinched)
arge attach_obj. object_nano. and object_id. both bands are holding objects, set these arga with list. planner wowe.oveeeopeeee www.ooee.o..ose.ttth.eeee.. (trpeettttttt....rrrrrrr.... Salse,Faless planner . noveoveett.. so_peae____peeeeeeeepeeeeeept_p_ with_ screv screw (planner right_ hand_ init. pose, right hand
*all" attt"cctthhh
100
right")
the initial pose.
101
103
Code exanple:
1. The right hand graape the can. Initial conditions: Not yet performed any action
step 0: Set the right hand fingers to a pre- pre-grasp pose
1 ppython
108 # Set the right hand fingers to a gresgrasp pase
planner.. gra.dappegraap
110
step 1: Hove the hand grasp pose ppython
constroint_F-planeee coneesstt_castrriitttte.....e grasp", band_name='rigie
obj_id-O, action="
Ganerate target pose end-otfactor move right
target
116
saaget_effector_pot planner. generate.cnd_ (constraint. Vand_nane-'righe cotaerrr.oo.too.pss..tttt screw
117
"right", attahh", obbachh
119
step 2: Clase the right band grasp the can python
Close the right hand to grasp the can
planner band_grasp ("rigg""rghht grapp.objectt obbj_iiiid)))
30

---

124
Notes:
coordinata information,
othervise apacified, (irent rebot),
world
coordinate syaten.
world +Y (left of robot), +2 (up of robot).
robot's coordinate systen aligns
"type_id" represents different modele of the same type of object, while 'obj_id' type added.
indicates the order which objects of the
131 The object's bounding bax (bbox) representa its dimansions in its oon coordinate orientation quaternion q=[1,0,0,01) For
systen (axis-aligned exauple, can vith bbozi("nin" 1-0.08,-0.08 -8,00008.0002 "naz"; 0,,08,0008,0000 0.04
along Please place each step in separate code block, enclosed within " "'pythen tags. The number of steps should correepond exactly to the number of python cie blocks your respense, 113 The paranatera in the cade block corresponding to each step are not abared. carry When selecting
has lengths
along
2-axis.
missing blacks.
Please
paraneters
previous step
step.
operate on object, please closer to the objact.
Each line of code must have at least one level of indentation.
The left hand's operable range right hand's range ia x [-0.42, -0.19], [-0.36, 0.07]. objecta muat be necessary
[-0.42, -0.19] y [-0.07, 0.36]; the
placed vithin sufficient condition successful grasping may still fail due to object orientation, collisions, gripper constraints).
neeting range
137 secees
Please generate the code to execute "TSKK
E.4 Sample Demonstration Code for Open Drawer Generated by HumanoidGen
from humanoidgen. netion_planning .i.thg....12. h1_2. h1.22. ionaeooooiiongggeeaaaed.dd solver_env import def step (planner Hunano1dorrionPlaaeee
# Step 1: Left hand graapa the draver planner paand_pre_grapp"11. constraint_r planner.gonerate.eeo
obj__ame="dreee"" _oneratte_constaatttt obj_id-0,
"giiep""rrspp, band_nana="left"
10
target_effeecor.ppe
Saaauer.gnerate..oom gede (constraint.r,
left")
12
planner nove_to Love__to_pose target_effector.poee "left", attach.ob-=False
14
16
planner hand_graspp"12 ("left", grasp_object= "rasp_objec=="áreee" obj_id-O) # Step 2: Left hand opens the drazer to an openness of constraint_open. planner generate_constreintee obj_nane="arsser"" abj_id=0,
18
19
21
actien-'grasp", hand_name- "and==mmem"'"" openness-1
23
25
26
target carget_oppea_poe
www.ee...geeeeeetee............. cenerate_ond_.ffeeee pos (constraint_
laft")
37
target.open.pose,
29
"left"
attach.obj-True,
31
objact_nane= aojact__nam=""raaver"
32
object_id-0
34
# Step 3: Left hand releaass the draver planner.. oped_uand (*left")
31

---

E.5 Prompt Template for MCTS-infegrated Demonstration Script Generation
You are a professional assistant for generating code that enables dual -rrn robots perforn tabletop execute the user upeciffed tacks based on the provided context.
generate logically structured
Static Scene Elements:
The static scane clements include a robot and a table. The robot's 's base_lime link ( pase paaa. [-0.85, 0, 0], pose.q - [1, 0, 0, 0]. The table surface spans [-0.42, -0.19],
poaition
palvis)
(-1.1, 1.16],
Systen Information:
Since the robot's base coordinate frane has pase.q [1, O, 0]. its xyz directioas align with the sorld coordinate systen: x points forvard, points robot's left, points upuard.
10 ASSETS_ATTRIBE
Note: The 'orientation' 'status' represents rotation in coordinate systen under corresponding 'description',
12
Initial
14 AASTIAL.ASSET_STT
asset (object) spose actions are executed. Pase consista of Cartesian coordinates and quaternion in the world coordinate systen: [x, y, 2, W, X, y, 2The following 7-dimensionel poses all follew this fornat).
scene just initialized,
any
16
18 CURRENT_ SASEENT.SSET_SSATEE
states (poses) asbets(s)jects) tabletop 20 s=ssss
Executed Action Code:
EXECUTED_CODE
23 Note:
by
tabletop from the initial state to the current state.
26 Prohibited Next Actions:
PROHIBITED_ACTO ACTION
Note: The step
the probibited actions listed above. 'Same neans completely identical. For 'nove', any addition, renoval, or considered different action. Applies only to
modification of the next step; subsequent steps can still choose those actions.
30 Robot Attributes:
hend_key_point
"rrrbaand.((baas_llettt ["bass_loftt_hand "'graapppoitttbaee_eeee " "left_band":
pinch_point bebb-pooitbbbeeee ledt__aaddd__ hand"],
www.ittbbbrrrrigt..bb.... "right_hand" ": "bass_rrght, rindt_band"
pooint_base_rrght
34
35
hend_kes axis:i
36
"left_hand"" "llff_ppacch_ee axis", "left_pinch_.riet..22 "ttpppiccc_..itt...p......s"""".
left_ring 2_indx__ "'ooat.graapp.aiii wtw.aiiii..otttggaa...r.r.. wraptratttpp....ppppp... 2ra.prrrrr.....l.a.ax.s.")
37
"right_ hand" ['right__pte pactttppich____eeee sinch_azie" axis', "rrihtt brighttpiich_uttee pinchhurist.33 wrist.._aattt..ll...... 2_p_al_pp______a_ee__ae_aee____a axis",
right_ring index"
axis "wwghtt.gassp.rrise 2_priitt__pal______emm
38 }
39 default_pose:f
40
seeft_band":: [left_ hand init.poeee_
41
"right_hand (right_hand_inn hand hand_init_poso
43
Available
45
Constraint (env type, and_effector, franne hand_ xey.dxxy_ponttt abject_key.ppo bxid_axia, object_axie]
46
Function:
47
"Coastraint" defines hard constraints that must be strictly satisfied.
Eatablishes spatial equivalence
19
(point2poiatt Specified end- point target point. targat
(parallel) Specifiad end-etfector direction.
51 Arge:
(Environnent Handatory): The plannor's bound operating environnent. Must reference the planner's environnent instance through 'planner any' property.
32

---

53
type (str, Mandatory): "point2point" 'parallel", addeeffector_tran (str, Nendatory): "L_hend,base, or "r_hand, base, link
55
hand key_pennn (mp adarray:: [Required when 'type' is 'point2point') Pre- notion efffccot anchor point sorld coordinates.
56
Mojjec..ky_pplin (ap adarray): [Required vhen 'type' is "point2point"] Target's corresponsing point world coordinstes.
57
aaid_axis (np. :darray)) [Bequired vhen 'type" is "parallel<] Pre-notion end- effector aligament exis in vorld coordinates. object_axis (ap. ndarray); [Required when 'type' 'parallel]] Target's reference axis in world coordinates.
Exanple:
60
# Right paln facing down to sraep(Top- grasping of the object) Constraint(
eny-planner.cn
type-"parallel",
cesffector.faae="r__e. base_link" hand_axia=gor
(planner anv aria_name="right sante"right.graapp... axia") paln points toxard the frent the paln. Pinch action is similar.
object_ (10,0
World frane,
67
68
Right paln facing up grasp (Down Top graaping of the objact). It there is a table or other objecta below the target, it often causes collisions Constraint(
it
20
eny=planmer.com
72
typa="parallel",
23
and ond_ctro "ft_offfcoor_frae=""____ frane="r_ base_____see_ink bank""ink",
band_aria=gen axria=get_axti aarta_t_env((plane
axi._nane="right. wtw.ggh.t........... "x)_" The back of the paln points toward the front of the paln. Pinch
25
object_azisenp (10,y(10,0,1))
World frame,
77
78
The diraction from the right wrist to the palm center is facing forvard. Constraint(
80
env-planmer.eav type-"parsllel",
82
end www.ffocoor_traee==_ee bane__bs_e__ikk."
axia_name="
right.graep_krlee wri.t_ axis*), The direction fron the wrist to the palm canter axis-nt array ([1,0,0)), World
85
87
# The diraction from the right pinky finger to the index finger is parallel the x-axis of objectO.
88
Constreint(
89
any=planner.env,
type-"parsllel",
91
band_arierget..riii.ii..oom ba______bbsee_nkk.
env maxii__mme"rrghtt_riii (iniii__rrrn____deex)") # The direction fron the right pinky ringer to the index ringar axiettxxie-eet_aa ixie__ (planner. eav, "x", obj typee="=ebbe""""" obj_id=0), The coordinates of objecto's x-axis in the world coordinate systen.
95
object0 hand, align x-axis parallel to world coordinate systen's 2-axia, making its x-axis point upward.
97
Constreint(
98
onv=placner.eav
type=""point2posnt"
odd_effector_frace "rrran="=_hand_baae base_aaadd_______eee_ae__aaaeeed link",
101
hand potyyppoootgor.ari___oo (planner. eav, "x', cbj_tyye="objeet"" obj_1d=0), www.yyooittoobeeettcoooo
102
array (Eo, (01,,, array (Io, 0.1, 0.1)) in the objecto's coordinate systen.
right- grasp point coincides [0, 0.1] objecto coordinate systen.
Constraint(
env-planner.eav "ope="point2poonat" ofd has_h__baae__ikk""
33

---

110
hand_ wow..ootttget..pooooo www...naannnapperr....... point_nane=" grasp_ pointtbbse__tteeeeeeeeeeeeeeeeee hand"),
trpe_nane nane- nameeaeejecc''""
obj_id-0, rolated_point aarp..rrayy(C ([0, 0.1, 0.11)), ap.array((O), 0.1, 0.1])
the abjecto's coordinata syaten.
112
114
After object0 1a in hand, position 0.1n above objecti. Constreint(
eny"planner.co type="point2point"
and_effactor.frane=t1 hand
(planner.. env typ. obbjeet
120
object jojjet.key.point-gee..p env (planner wtpptttpeaeeeeeeeeeeeee nanee-eaaeeeeeeeeepepeeeee_eee__ nane-"object" obj ibj_ array(10, 0, 0.1)), ar.array ((0, o, 0.1]) the vorld(robot) coordinate
123
baadd_ key_ppiat'' and "object_key, key- represent txo points in the world correspending
coordinate eysten. 'band keydd_ke-peint effactor, and type='pointspoint indicates the intention for the two points to coincide.
'aandaaiis' "object_aris"
similar meanings 'point'. or right__innch.xts parallel 10,0,-1] neans executing the paln facing downvard.
125
126
"right_ saaet.grapp_aiie or 'right_pinch_ parallel [1,0,0] means executing paln facing forsard.
127
"right.grasp_axia
"iight.pinch_.xie parallel [0,1,0] means executing with the paln facing left. graft_graap axia' 'laft_pinch_ axia' with the paln facing right.
[0,-1,0] exacuting
130
type and andd_effcttrr_eee frame hand_key_poins w object. koy_post haid, objact_ axia):
The usage is similar to the Constraint class, but it is not strict optinization objective.
133
functions:
135
gat_point__ite anv (anv, point_nans sype_name
related, ,otntt.pennessssss)
Function: Get specified point position in world frane.
138
Return: point.position (dp.ndarray)
Arge:
140
eny (Eavironment, Mandatory): The planaer's bound operating environment.
planner's environnent
through 'planner
property.
point_nane (atr, optional): The point nane. Example:
btttpppoottbbseelll...eeeee type_nane (atr, optional): The reference object relative to the target point
142
143
obj_id (int, optional): The reference object id. rolated_point (np.rdarray, optional): relative to the objact canter in the abjact coordinate systen. Dafault: arrarray (10,0,0)
coordinates
target point
openness (int, optionsl); Represents the openness degree of articulated object. type_nane is an articulated object, this value can be set to coordinates corresponding point_nane degree.
epecific opennese
Example:
447
Get the current coordinates of the right-band pinch point the world coordinate systen.
got_point__n_ooe (planner... www.pppnaneee" strppaaannssrrp............ap harppppoiit.bbree.iich....aae
get_axis_ in_env (eav, axis_nane, obj_type. obj_id): Function: Get specified direction vector in world frane.
151
Return: dbrection_vactt. vector (dp.naarray) Arge:
153
env (Eavironment, Mandatory): The planner's bound operating envirenment, Must reference the planner's environnent instance through 'planner.. enr' proparty.
name Erample: 'right_pincl axis", obj_type (str, optianal): The reference abject relative to the target axis. obj_id (int, optional): The reference object ID.
axis_name (str, Mandatory): :
156
158
def opea_hand (self, hand nane): Function: Control the hand open.
34

---

161
the efd-effector 'band_neme" to Return: None
Args:
hand_came (str, Nandstery): *all", "right", "left",
daf pand_pre.graap (band_name):
166
Function: Adjusta the thumb novement of the specified hand to position it finger, preparing for subsequent grasping operations This should typically be called before executing "ove_.to.pose.uith.ce to reach the pre-graep pose.
opposite the
Return: None
Args:
(atr, Mandatory):
"right", "left', Default: *all",
170
graspp (hand_ grasp_object. obj_id):
172
Function: Control the hand close.
Retura: None
Args:
175
hand_nane (atr, Mandatory): "right", or "left".
grasp_object (str, Mandatory): The type graaped object.
Example:
obj_id (int, Mandatory): object
grasped object. Example: cbjap=="ject=cann and obj_id=0, it indicates the first can objact addad to the environmens.
179 def hand seld_pre_pincte (hand_name):
Function: Adjusts
specified poeition opposite the index finger, preparing for subsequent pinching operations. typically called before executing tore__to.pese.utth to reach the pre-pinch pose.
Return:
Hone
Arga:
183
hand_name (atr, optional): "all", "right', or "left", Default: "all",
185
def hand pinch (hand name, pincb_object, obj_id):
clesing operation Typically used for small and hard-to-graap objects.
Only
finger.
187
Return: None
Arga:
189
hand__ame (str, Mandatory): "right", or "left",
pinch_object (str, Mandatory): type the pinched object. Example:
obj-id (int, Mandatory): object objec_object-carr and obj_id-O, indicates the first object added to the anviranment.
pinched objact. Exampla:
193
def end. www_eefeettrr_poc.ccooooooo ceeeecooristttt band_aame): Function: Calculate the target pose the this step based on the constraint and cest. Return: effargtt.
nove
196
197
constrainta (list, Mandatory): 11st of Constraint objecta, with the nunber Constreints deternined by specific requiremente.
198
hand_pane (etr, Mandatory): "left" or "right"
tpn def nove_to_post poss__itt_aacrrrpppa (pose: sapien. Pose, hand_name, staah_objjFalls
mojject_name-oone object_id-O)
201
Function: Contrel the robotic the 'pose'.
203
Args:
pose (sapien. Mandatory): target pase movement
hand_nane".
hand_ (etr, Mandatory): The specified robotic arm for the movement. attach_obj (bool, Mandatory): Vhether there attached object in the hand during the novement.
object_nane (str): [Required attached object
'attach_obj" True] type
obj_id (int, Mandatory): [Raquired when 'attach_obj' True] objact of the attached object.
210
211 Incorrect example code:
right hand grasps
table:
113 step 1: Grasp cand
<re etpyhhon
015
# Set the right-hand fingere pre-grasp
planner hand_pre__reeeeeeeeeeeeeeeeeeeee wrwppprrreppprieeee ("rasppprightt right")
35

---

118
conatraints=
seeetrinnt..apeene
Constraint(
onv=planner.eny
123
type-"parallel",
odd_effector_fret
fraectorrrrrreeeeeee seauned"__aadd__ee _inkk,
125
inn.i..innnn(peenee axld_axl in__iv__eee (planner .nn, axi5,
object_axie-mp. arrey 0, 1])
)
Generate target pose
end
right hand
target poge
terget_effecctor_pe planner p oe (constraints. Mand_uene-"righe p wtt..oooo....ooo.itloooooo (tareetttttteeeeetorrroo ,right "right", attach.
132
Close the right hand to graap the can planner bend. dreep_.
ii-d))
235
error: abject_axis-ae array (10, 1]) represents positive axis in the world coordinate system. It it is parallel to the grasp axis, it indicates a botton-up up grasping direction. For an object on table, the hand reach belav object. To perforn top-down graup along direction, use axia-nt. ([e, 0, -1))
238
example
040
1. The right hand graaps the can from right to left, then lifta it by 0. In, and then releases it.
141
step 1: Grasp
242
"python
Set right-band fingere pre-grsep pose
244
planner hend.pre.grrep ("rigtt"""
146
conatraints-[]
247
Add point to to-point constraint to align the grasp point the right hand point
the
248
cons tettttrintt..... append
Constraint(
150
ony-planner.cnv.
sypa="point2ponnt",
152
hew._bbaaadddeee...___aaaa _iakk",
potttppotttgttpoinne anv (planner anv paint _aae=="
graap-ppont__bee bwweiittbaeerrgttt.... hand"), Grasp point
right band
354
object_key we..t.eeepott-ett.pooo cett_point_ii_eee
point of (planner env type_nane="""" can"" obbj_i obj_ii
255
357
Add Constraint to slign the pinky index axie of the right hand with the world z-axia
appennt.epppeed Constraint(
259
eny=planner.cnv
t61
typa="parallel",
end seffecttor.frme-"yyheee
1ink",
063
aand_axis=get_azie. wwwisssi__eee___ee_eee (planneer The direction from the pinky finger to the index finger object_axie axietnp_xieepp...ee array ([0,0 [0),, the object, such as att_axis_in_ent (planner. env, "2"" obj_type-* obj_id=0)
wxis__ane="righttt.. 2_ing____iee
epecific
165
Generate the pose for the end effector and the hand the target pase
sarget_effector_poss planner goneratee conerstt_add (constrainte. hand_name= rrghht")
planner. toamerr.nve..ooppse..o pl (tereesrrrttrrrgteeffeetto
"right" attachhabj Falaee)
270
271
Close the right hand to grasp the can planner band_ ('right" graep_object="catt obbj_id0)))))
step
by
keeping pose unchanged
125
< "python
conatraints=01
# Add point-to- toint-t--point constraint lift the 1m constraints append Constraint
228
36

---

180
onv=planner env,
und_effector mnd_effector.franee wrrroooorroerrrrreee (planner env type_name= "ppp__aa""ccn".cco___100 oob__d=00) prr--potttgggppoiatt
point
284
object Keet._ey_pp poyy.poon-get.polln pointppin..ia. in, (planner commerreny. env "ape_"am"""ca" _bb_iid .id +0p array ((0,0.0.1)), 0m
center point after being lifted by
2166
Add parallel constraint to keep the x-axis of the can unchanged constraints. append(
289
Constraint(
eny=planner.com
191
type-"paraleel,,
292
end oad_offector_frame- base_ link""
hand_oris-get__xit wtt..ri.iin.en(plaaeeee.. obj_id=0), a The x-axia of the can
axis_name="x"" ttpp:/""p".o.ppppprpp"._
object_axie-get, intesett_iiiiieeeeeeeeeeee. env (planner. env, exie_ obj. type""caa""" obj_id=0), # Keep the x-azis unchanged
296
197
Add parallel constraint to keep the y-axis of the can unchanged wwwtrrintss.ppeee...
Constraint(
emv-planner.eny
201
sype="paraalel",
ceffecttorffrao=""_haatt
link", env anr.ars.r.....ssnnne....... typp=""pp="ccn
103
obj_id-0), The y-axis of the can
object_axia=get aaris_in_avv (planner. axis_name= abj_ type=" obj_id-0), Keep the y-axis unchanged
106
right to planner and (constraints, "and_pame="right")
target pose
target
target srgett_offecto__ocr
309
"right" stlach_obj -True planner to_ve__t__ooe__ii_ withh_crrew (target, efa.etttteeleettrrppeo "right" object_nane objject__dd]
310
Open right
312
planner ond ("righ""
113
step3:
right "python
pose
115
Move right
initial pose
117
right_hand, hatt_onit_pose, "right", planner nove set_pose_utth_eeee
attach_obj=False)
318
119
step Exauple 1, place
grasping
121
step
ensure palm facing downvard.
123
pyyhonn
constraints-[]
a Add point-to-poinn constraint lift the 0. constraints sesstrainss..ppedd.
Constraint(
128
ony=planner.eny
type-"poantipoint""
130
and_effector and_effactor_frame== baa____aae_
hand_key poiit--et..pont_de (planner. env Syge__eme-"car "aaauaee""nnno____"""" idbj_
point
per.kkke--oonn-tt.... iininnnn (prannnnerr env stpa_name="boxl" bobj= ,ojj =0) array ((0,0,0.1),,
_ij_id
point boul
133
135
# Pala facing down
336
constraints append(
Constraint(
eny=planner.env type="parsllel',
140
axn__aria-gtt.axtaa (nv . env axxta_nae=="rght "rapigttt_.aaaaaaaaa The back of the pala points toward the front of the paln. object_axie=pe arrey (0,0,-1]) World frane.
37

---

144
Generate the pose for the end-effector and the right hand the target pase
target_effector.p
planner cenerato_end_effere (sonatrainta, hand name- riighet
"right" attach_ objeerue, # Open right hand release the planner. opan_ ("rrnnghh"" ("right")
planner move to_ve_ poo_poe__ithh_ceeeeeeeeee object_nane="en cam""""m""""""" "ta.eenne""a""ooseeeeiea
151
...
153
step3: Nove the right hand back to its initial pose python
155
# Move the right hand back its initial pose planner - Love_to-_p to_pooo_-oooo wit_t (planner sttach_obj=Palss)
"right",
157
159
t61
'type_id' represents different madels of the same type of object, while fobj_id' indicates the order in which objects of the same type are added.
The left hand's operable range
[-0.07, 0.38]; the right hand's range ia x [-0.42, -0.19], y [-0.36, 0.07]. It muat be range.
[-0.42, -0.19],
ensured that the pinching, placing object. it must be ensured that they are all within operational range the corresponding hand.
within
grasping,
163 The object hounding box (bbox) representa its dimensiona in its oun coordinate systen (axis-aligned when the orientation quaternion is -[1,0,0,01) For with bhox: {"min": [-0.08, -0.08, -0.02], "max":
example,
10.088,0002], has lengths of 0.16 units along X/Y-axes and 0.04 units slong Z-azie.
364
Please place each step in separate code block, enclosed within "'yythen tags.
correspond exactly to python blocks in your reaponse, with no extra or missing blocks.
steps
"c.
365
Variables in differant code blocks are not shared, meaning variables from blocka When placing an object, release it elightly above the target position. splitting
previous
aubsequent
166
multiple moves. example, when placing can into boxl, first nore the can to height above just above ensure the height neither too high too lov avold inverse kinematics issues or collisions. Typically, an additional height 0.03 an added.
collisione,
boul,
releasing Hosever
368 When planning, please take collision issues into account. For ezauple, if the to
height
z=0. 300.03<0.08, right hand to grasp from right to left, as it will cause collision betveen the hand and the table. Sinilarly, when placing the object soneuhere, it ia position.
object
acceptable to release it elightly above the 369 If 'Executed Action Code! is empty. it means no code has been executed, If not subeequent
eupty, generate only If 'Frhiibited Fext Actions' is empty, there are no restrictions on the next action. not empty, avoid executing any in the first aubsequent step.
370
to
371 When performing operations such as noving an object specific location for
factors should be fully considered. For example, when placing an object on a table, it is acceptable to suspend it slightly above table.
placement,
322 Before executing grasp or pinch actions, greeg and pre- -inch actions must be executing another grasp pinch action,
perforned, respactively. corresponding pre-action muat be perforned again. 373 When calling 'Conetrsint', paraneter names must be explicitly specified using the 'name=value" format.
174 Each line of code must have at least one level of indentation. con-collieion poses, which can avoid collisions with other hands or objects.
Please generate the code to continue executing the task 'TASK, D
TASK_NOTE
38

---

E.6 Sample Demonstration Code for Block Stack Hard Generated by Applying MCTS
Tron bunanoidgen metion_ pwa.nnnppannnngg...mom iaaiing.... h1.2 $1.2..000 aolution generated solver inport step(planner: hataeaaaaiiiiiiiiaaaaaaaaeee
# Step 1: Right hand picks cubel and places it at the stacking base position planner paad__pre.piee ("rghtt)
constraints
Constraint(
env-planner eav.
sype="pointtpointt,
10
hand
int.-ooottnnn.a(ppppl... (planneee anv, point_name="
12
keject_key.posnt-gat.. on in.e.ttannneeeaaerrr env, type_name"'cube,, obj_id=1)
13
Constraint(
15
ony-plammer.cny
typa="parallel",
18
aand_aria-get.arta.llt aww_ariiiaaeevaaeeeee_aeeeeeeeee env (planner any, arih.name-rrghhh.iicc. axject_ arrey ((O, 0, -1])
20
22
wwanner.gneerat....mm generate, and wttp:aatteddddeettrrrrraoo wowaccor_pposeeeeeoooooeeoooeeee (coonsttaatt "right") planner tww.. (target..ooe "right", attach.obj
target_ planner
24
25
planner hand ("right", pinch_objact==aubee obj-id=1)
27
constraints
29
emv=planner.eng
30
type-"point2point"
med_effectos_frame-
sarpaaeee-__a_d_bbbar ha____aadd_eee_eeeee_a_____aaa_e base_link",
32
wtt.yppotttgge.oittt.... env (planner env, type_ iij.id hand_key.
33
object.key.point wrwpy ((-0.3, 0.0, 0.05]
35
Constraint(
36
env-plasner env,
type="parallel',
38
fraaee-rrrrraerrrrr_rr "ink"_
hand_ axisgetis-eet_arte axis_in_ent in_eevee (planner (pplannee object.ariesnp.arre ([0 0, -1])
nane- sitie_snee"rriht_piiee pright.ppacccaaii
40
42
target_pose planner, generate_ end_ pose (constrainte, "right")
planner tooe__t__pse__tt sc.esss_iithscrre (target_pose, "right", attach_obj objeerue, jjject_nane="cutt object_id-1 id=1))
46
planner open hand hand opan_hand
("rightt)
48
planner www.atoeee.oooeeoeee wove_tt__esseeeeeeee.ttt___sssee to_pose__tteeeeeeppeeeeeeeeet_e_ with screw (planner. right_ baadd__aa_____ hand_init_poo. init_pose, "right", sttach_obj=Fales
30
Step 2: Left hand picks cubed and stacks it on cube1
planner hand pand_ppr__incheee ("lefft"")"")
52
constraints
Constreint
55
eny=plasner.c env
type="pointtpoontt
57
end_effector. "1 base__bae__lnde hand
key hand_key
(plaannerr......... env, point_name="
pinch-point__ase. bartt
object_key_per jejet_key_pddat-get.pp.... in_enviinee (planneer. env, type_ obj_id=0)
60
) Constraint(
62
env-planner.env typa="parallel"
env.
64
wofffector.frane-- h_nndd__ base bass_link",
www_seeetxxiiiinneeee env env eny sxie_neme-'eeft_pii axis")
object_axie-sy array ([0, 0, -1])
39

---

69
target_poae planner generate, and_eefector_ppe (constraints, "left")
70
tor....o.......t.eeeee (srgeee (target........... "left", attach.
72
planner pinh ("left", abnec_abjeet="ceet obj_id=0)
74
constraints
Constraint(
76
eny-planner.
sype="pointtpoint"
hand_ seey-pontt-ga-ppliie an (planner. env, type_name""cabe", obj_id =0)
80
objeci_key.point-e array ([-0.3, 0.0, 0.09])
82
Constreint(
eny-plasner. eav.
type="paraalel',
85
and_effactor_ sefffactor_frae="/ baa__annaae___a_a_______ base_link"
(planner env, axis.neme-'left adde.nam--1ett.ppnno.... dend_aris-get.axt (ppieenneevpplenner..........
87
object_axie-np array ([0, 0, -1])
89
90
target_poos planner genorate www.edddodooroooeeeeeeeeeoo.eeee coeeee.fecoor.peeeeeeee (constraints "left")
planner www.nooee_oeseeeeeee__ee tt___tt___e_____eee wttppseewtth____.eeeeppppph setee_iith__reeeeeeee (target_pose, "left', attach_obj-Trut
object_idss id=
93
planner open haa__hann(""eft""
95
planner wtt.:ttooooaeethhh.eoo..e (planner.. haft_hand_init.ppaa "left', sttach_abj=Palse
97
Step 3: Right hand picks cube2 and completes the stack planner pww.duppeeepipprreeeeeeee
99
constraints
101
Constraint
eny-planner.com .nv.
type="point2peint",
104
(planneer env point_nane- siiht_hand""
pinch.point_
Dajact_key_pot gaty (wwannnnr........m. any, type_name="cuhe" obj_id-2)
) Constraint(
109
env-planner env.
type="parsllel",
111
detfactor__faae="> harrrr_addbbbe_leaeee
axt.arrraaett.ata.lll anv_inneevveeee (planner (n..ii...nv((paneer abje-pp arrey ([0, 0, -1])
env, axia_name= rrii.naaeeriggt..phhch.....
113
114
116
target_pose planaer.gemorate gww.rete and_ efdeee.ooo (constrainte, "right")
planner nova_to__oseeeeeo.oeee___sssssee www.peowwtt..ccrrrttttaeee..
"right", attach_obj-Falsd
118
planner ("rght", oineh obj.id-2)
constraints
125
eny=planner.cm env.
sype-"poont2pointt
125
and_effecten
www.eeettr.rraeeeeea. frane="r baad__baa__inx"
hend poo..poitt-ee..piat... eov (planner. env, type_aame-"cube". obj_id
-2) pejoct_key_point-aee wrw.y ([-0.3, 0.0, 0.13))
129
Constraint(
eny=planner.eng
131
type-"parallel",
base lill_lini link", ond_effector_ ( axxis_in_env (planner axieet_xxiienpp.rre (IO, 0, -1])
oxis_name"'righe "its_aaee"righht.picc...t
114
136
target planner generate__et ancaatte_edd__eeee wwddeeffetorrppee (constrainta, "right") planner nove_te poove_to_pose.....oo seee__iih_screre (target_pose, "right", attach_ obb-Tr object_nane- nane- "cube" object_id-2) open_hand ("right")
138
40

---

Broader Impacts
This work advances the field of robotic manipulation by enabling automated task creation and demonstration collection for bimanual dexterous systems, which can significantly reduce the and complexity of data acquisition humamoid robotics. Traditional methods often rely manual demonstration design, expert supervision, or extensive environment resets- processes that are time-consuming, labor-intensive, and difficult to scale. By contrast, our framework automates these processes through an LLM-driven planning framework that generates semantically meaningful and physically feasible tasks without human intervention. This not only simplifies the pipeline from task specification execution but also collected demonstrations.
resets-procceses
consistency and diversity in the
The proposed framework has potential to accelerate research on autonomous humanoid agents, particularly domains such as assistive robotics, disaster response, and intelligent automation, where bimanual coordination and fine-gratd grained hand manipulation are critical. For instance, in assistive robotics, precise bimanual operations, such opening pill bottle pouring liquid from container another, require both high-level task reasoning and low-level motion control. Similarly, disaster response scenarios, rohots may need to manipulate tools, open doors, or handle irregularly shaped ohjects constrained environments. These tasks demand robust and adantive manipulation capabilities that go heyond simple pick-and-place actions. Our framework enables the generation of such complex interaction sequences, allowing researchers to train and evaluate policies on long-horie long-horizon, multi-step tasks at
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are specifically claimed the Abstract and Intro- duction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the Appendix supplemental material.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does contain thcoretical proofs.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
42

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully describe our experiment setup in Scction 5, and the information provided is enough to the reproducibility of our main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clcarly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer:
[Yes]
Justification: We provide sufficient material supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (httpa://nips.. httpp//////pp..ccppppippae cc/public/ muides/CodeSiiee
asionPolicyyy for more details.
While we encourage release of code and data, we understand that this might not possible, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :// //nips/p..///////// cc/public/ muttdes//ooeeSubbneessooooooo for more details.
43

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specify our training and test details in Section 5. We think that the information we provide is sufficient to understand the experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide statistical results in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please see our experimental details in Section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
44

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:///eerrrpppppee cttppps.ccpppiieee
csdudelinee??
Answer: [Yes]
Justification: read the NeurlPS Code of Ethies, and make sure that our research conducted in paper conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Appendix in supplemental material.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system Icarns from feodback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
45

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite works of related assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
46

---

Answer:
[NA]
Justification:
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [Yes]
Justification: Please see Section 2 and Section 3 for the details of how use LLM. Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy http://pp///////ppp///////////// should or should be described.
for what
47


Reviews:
[{"review_id": "bKLyAtBZkY", "reviewer": "Reviewer_NeLy", "summary": "This paper introduces HumanoidGen, a framework to automatically generate tasks and demonstrations for bimanual dexterous manipulation, addressing a key data scarcity problem for humanoid robots. It uses an LLM planner, enhanced with Monte Carlo Tree Search (MCTS), to produce a sequence of spatial constraints that guide the robot's actions. The primary contribution is a scalable method for generating high-quality training data, validated by showing that diffusion policies trained on this data exhibit positive performance scaling.", "questions": "- Will any real-world experiments be added? Or what's the barrier that prevents deploying the data generation system to real robots?\n- What's the difference and connection of the proposed method to the branch of augmentation-style data generation methods? What are the time and effort expenses of the two styles?", "limitations": "No discussion on limitations in the main text. I read the limitations in the appendix, but I think there are quite a few unmentioned limitations, e.g., cluttered scenes, deformable objects such as cloth folding, and contact-rich tasks such as push-T.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- Presenting a very detailed data generation pipeline for bimanual dexterous manipulation.\n- Also presenting a benchmark for humanoid manipulation.\n- Achieving a higher data generation success rate than RoboTwin, and a credible data scaling-up trend in the DP & DP3 experiments.\n\n## Weaknesses\n- Lack of real-world experiments. I'm willing to see at least some small-scale verification, either by running the data generation pipeline in the real world or by sim-to-real transfer.\n- While this paper focuses on zero-shot data generation, there is a line of recent work doing augmentation-style data generation, exemplified by MimicGen (CoRL'23), DexMimicGen (ICRA'25), and DemoGen (RSS'25). These methods require a small set of human-collected demos as a start, which is not required by HumanoidGen. However, the pipeline of HumanoidGen is quite long. If not all the components are well-automated for all the tasks, it still requires considerable human intervention. So, I am curious about the total time and effort cost of the zero-shot generation pipeline, and how it compares against the augmentation-style methods.\n- Lack of discussions on limitations in the main text. See the comments in the limitation section.\n- Lastly, a very small comment: I think the Primary Area of this paper should fall into Applications rather than Deep learning.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point. \n\n**Weaknesses**\n\n**W1:**\n\nIn response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.\n\nWe conduct experiments following 5 steps:\n1）**Real-World Setup and Task Design**. We built a real scene corresponding to the simulation environment. Two tasks are designed for experiments: \"Dual Bottles Pick\" and \"Close Laptop\".\n2) **Real2Sim Assets**. For task-related objects, we prepared both real objects and their corresponding simulation digital assets. \nSpecifically, we obtained the digital assets of standard objects such as Coke cans from online asset repositories. For objects whose digital assets can hardly be found, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance. \n3) **Real2Sim Pose Estimation**. Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose (CVPR 24). The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate. \n4) **Trajectory Collection**. We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task for 300 iterations.\n5) **Sim2Real Policy Transfer**. To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 5 real-world trajectories plus 100 simulated trajectories. The results are as follows: \n\n|Evaluation Metric              | Dual Bottles Pick | Close Laptop| Notes|\n|-|-|-|-|\n| **Scene Randomization Range** |  6 cm x 6 cm, ±15°  | 6 cm x 6 cm, +10° to +20°\t   |Position (cm) and Yaw (°)|\n| **Data Collection Success Rate**  |  85% | 90%  |\n| **Data Collection Time**  |  16 |  13 |Seconds per episode\n| **DP Success Rate (5 Real)** |  0% (0/20)  | 20% (4/20)  | N=20 trials per condition\n| **DP Success Rate (25 Real)** | 60% (12/20)  | **70% (14/20)** |N=20 trials per condition\n| **DP Success Rate (100 Sim + 5 Real)** | **65% (13/20)**  | **70% (14/20)**|N=20 trials per condition\n\nAs shown above, our approach consistently achieves over 85% success rate in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.\n\n**W2:**\n\nWe note two differences between our approach and the augmentation-style methods:\n1) The difference in **annotation types**. Our method requires annotating **operation points and axes on the assets**. These annotations correspond to different robot atomic operations, such as grasping and pinching. In contrast, MimicGen and DexMimicGen collect trajectories by teleoperation and annotate **different subtask segments in the trajectories** for further replay. Compared to their annotation type, which relies on collected task-related trajectories and is hardly generalizable to new tasks, our annotation type is task-agnostic and supports further task extension. By combining manual annotation and annotation transferring, our method allows building a large-scale library with annotated assets, supporting automatic script generation and data collection of diverse tasks.\n\n2) The difference in **resource consumption**. The table below shows the time cost of our method at different stages:\n\n| Metric                        | Manual Annotation | Automatic Annotation | Scene Generation | Script Generation | Data Collection Execution |\n| :---------------------------- | :---------------: | :------------------: | :--------------: | :---------------: | :-----------------------: |\n| **Time (s)**            | 4-6               | 1.17                 | 12.64            | 18.31             | 14.40                     |\n| **Resource Cost**             | Human Operator    | Automated Process    | LLM (2,958 tokens avg.) | LLM (3,745 tokens avg.) | Motion Planner (0.53 s)   |\n\n\nWe evaluated the time cost of our method versus augmentation-style methods under several scenarios:\nFor new tasks with existing assets:\n   Our method can generate trajectories fully automatically, with no manual effort required. The fixed time cost per task before generating trajectory data is **30.95s**, which accounts for Scene Generation and Script Generation. The time to generate a single trajectory equals the execution time of one run, averaging **14.4s**.\n\n* DemoGen requires teleoperation to collect an initial trajectory and additional manual effort to crop and segment point clouds. It's fixed time cost averages **86s**. Generating an additional trajectory only takes **0.01s**, but this approach cannot provide data augmentation for vision-based policies and cannot guarantee that the generated data is always feasible.\n* DexMimicGen has an even longer preparation time, averaging **253s**. Besides data collection, manual segmentation into subtasks is required before trajectory generation. After generation, it still needs to execute trajectories to collect data, with each trajectory averaging **16s**.\n\nFor new assets and new tasks:\n   Our method requires an additional **4–6s** of manual calibration. Nevertheless, it remains the most time-efficient overall. This efficiency stems from the fact that our approach does not rely on a pre-collected trajectory as the basis for generation; instead, it leverages annotated assets.\n\n\n**W3:**\n\nWe appreciate the reviewer's suggestion on limitations. We will move the limitation section from the appendix to the main body of the paper and further analyze the limitations of our work. We provide a detailed discussion below in response to the limitations mentioned by the reviewer.\n\n- **Cluttered scenes**: Our framework can handle tasks with cluttered environments and distractor objects. This capability stems from the Active Collision Avoidance and Dynamic Collision Management mechanisms in our framework design. Specifically, the Active Collision Avoidance enables the LLM to plan steps that first move occluding objects aside before manipulating the target object. The Dynamic Collision Management, on the other hand, enables the LLM to determine which collision bodies to consider during path planning. These mechanisms together can effectively manage complex collision scenarios, such as those found in cluttered task environments. In addition, our MCTS reasoning enhancement enables the system to iteratively correct failed attempts and continue searching for a valid trajectory in case the LLM fails to generate a successful plan in one shot. To the best of our knowledge, such capabilities are not present in prior works such as robotwin(CVPR2024), gensim2, and DexMimicGen.\n\n- **Deformable objects**: Our framework currently doesn't support deformable object manipulation for two reasons: First, the Maniskill simulator lacks deformable object support. Second, our framework targets tasks with clear goals, making operations like flattening a crumpled shirt challenging due to the need for real-time feedback on fabric wrinkles and excessive LLM reasoning steps. However, extending our framework with more diverse atomic skills, such as training specific DP policies for subtasks like flattening, could enable support for these \"goal-ambiguous\" tasks, highlighting the system's extensibility.\n\n- **Contact-rich tasks**: Contact-rich tasks require rapid reactions to runtime feedback. For tasks like push-T and shirt-flattening, the need for fine-grained step-by-step corrections arises from their goal-ambiguous nature, where each action can lead to multiple outcomes without a fixed solution. This makes precise trajectory replication impossible in our current framework. However, our system can instead generate alternative strategies, such as directly grasping and placing the T-shaped object. To better handle such tasks in the future, a promising direction is to enrich the framework with additional model-based atomic operations.\n\n**W4:**\n\nField of affiliation. We thank the reviewer for the suggestion regarding the appropriate field of affiliation. While our work is indeed related to applications, we believe it also appropriately falls within the scope of deep learning. Our work uses LLMs to generate expert demonstrations for bimanual dexterous manipulation, enhances reasoning with MCTS, and evaluates models like DP and DP3, all closely tied to deep learning methodologies. We appreciate the reviewer’s thoughtful suggestion.\n\n**Questions**\n\n**Q1:**\n\nWe added real-robot experiments, which are discussed in the \"Weaknesses\" section. Regarding the difficulties in deploying the framework to real systems mentioned by the reviewer, we believe the main challenge lies in the sim-to-real gap of physics simulation and visual rendering. The former can be addressed by applying system identification to optimize the parameters of the actuators in the simulation. The latter can be addressed by modifying lighting and brightness settings in the simulator to match real-world conditions.\n\n**Q2:**\n\nDue to the character limit, we refer the reviewer to the discussion in the 'Weakness' section. In addition, our object annotations are reusable across multiple tasks, whereas augmentation-style data generation methods are typically limited to augmenting data only in terms of spatial position."}, {"review_id": "ta691Jzk1h", "reviewer": "Reviewer_mhYR", "summary": "This paper presents HumanoidGen, a framework for automatic data generation in bimanual dexterous manipulation, leveraging LLM-based planning, spatial annotation, and MCTS-enhanced reasoning. The system annotates both assets and robot hands with spatial keypoints and axes, and uses LLMs to generate code-like constraint chains representing manipulation plans. These are then converted into executable demonstrations through trajectory optimization. To handle insufficient annotations or long-horizon tasks, the authors introduce an MCTS-based introspective mechanism that improves LLM reasoning robustness. Based on the proposed framework, a benchmark, HGen-Bench, is constructed with 20 tasks, and the generated demonstrations are used to train diffusion policies.", "questions": "1. Could the authors more clearly and precisely define the intended scope and applicability of the proposed framework?\n2. Can the authors discuss whether their framework is applicable to more complex, contact-rich manipulation tasks beyond the current benchmark?\n3. Can the authors elaborate on the scalability potential of their method, especially in terms of annotation cost and generalization to diverse scenes and objects?", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "The main strengths of this paper lies in the design of a complete and automated framework for generating bimanual manipulation demonstrations in simulation. The framework integrates spatial annotation, LLM-based constraint planning, and MCTS-enhanced reasoning to handle long-horizon and complex tasks. Its effectiveness is supported by empirical comparisons with RoboTwin, where HumanoidGen achieves significantly higher success rates in demonstration generation, highlighting the practical value of the proposed method.\nHowever, there are several weaknesses in the paper that raise some concerns for me.\n* The proposed data generation framework depends heavily on manually designed spatial annotations for both assets and dexterous hands. While the annotations are carefully constructed and detailed, this level of manual specification may not scale well to more diverse or real-world object sets, especially those with complex geometry or articulation. Moreover, the paper does not provide sufficient empirical evidence to support the scalability of this framework. This raises concerns about whether the approach can scale up effectively beyond the curated simulation environments used in the current benchmark.\n* While the paper is framed in the context of humanoid robotics, its focus is exclusively on bimanual dexterous hand manipulation. However, \"humanoid\" typically encompasses broader challenges such as locomotion, mobile manipulation, and whole-body control. Given that the proposed framework and benchmark do not address these aspects, the use of “HumanoidGen” and “HGen-Bench” as naming choices feels overstated. I recommend the authors adjust their claims and more cautiously position their work to avoid overgeneralization.\n* Even within the claimed setting of bimanual dexterous manipulation, the actual use of dual-arm coordination and dexterity is limited. From the videos provided by the authors on the project website, it is evident that although both arms are present in most tasks, only one arm is typically active at a time. Except for a few tasks like handover, most demonstrations could plausibly be completed by a single arm, and the generated solutions often resemble single-arm strategies. Moreover, while multi-fingered hands are used on the robot, their function is largely restricted to simple pick actions, which are not fundamentally different from standard two-finger grippers. Therefore, even within the scope of bimanual dexterous manipulation, the paper does not address the truly challenging aspects such as coordinated motion, contact-rich dual-arm manipulation, or in-hand dexterity, which casts doubt on the depth and significance of the contribution.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will explain your concerns point by point.\n \n**Weaknesses**\n\n**W1:**\n\nTo address the concern about the scalability of our framework to a wider range of manipulable assets, we conduct further automatic annotation transferring experiments on 5 types of objects with the method mentioned in the paper. \n\n| Evaluation Metric             | Cup (Handle) | Cup (Edge) | Bowl  | Laptop | Drawer |\n| :--------------------------- | :----------: | :--------: | :---: | :----: | :----: |\n| **Manual Annotation Time (s)** | 4.00         | 4.00       | 4.00  | 6.00   | 6.00   |\n| **Automated Migration Time (s)** | 0.72  | 0.72 | 0.79  | 0.81   | 0.80   |\n| **Migration Success Rate (%)** | 90.0 (18/20) | 100.0 (20/20) | 100.0 (20/20) | 100.0 (20/20) | 90.0 (18/20) |\n\nBesides, we discuss integrating other works into our framework to effectively handle objects with complex geometry and articulation, as mentioned by the reviewer.\n\nOn the one hand, for articulated objects, we note works that focus on predicting affordance points and articulation axes to guide manipulation. Some recent works, such as AffordanceLLM(CVPR 2024) and CAP-NET(CVPR 2025), demonstrate strong generalization capabilities across various object categories, including both numerous single rigid objects and articulated objects. We also note that the input for their prediction and the annotation transformation process, specifically object point cloud sampling and 2D-3D back-projection, can be performed in simulation without noise or error. We believe these methods can provide point and axis annotations on articulated objects, effectively addressing the reviewer's concern regarding articulated objects.\n\nOn the other hand, for objects with complex geometry, which pose significant challenges in both annotation and manipulation, we note that this represents a distinct research area where current methods are still immature. Some works, such as DexGraspNet (CoRL 2024) and DexVLG (arXiv:2507.02747), explore generating dexterous hand grasping poses from 2D and/or 3D perception. Although these methods have certain generalization limitations, they show promise in supporting the manipulation of objects with complex geometry. Despite the immature state of research in this area, we note that our framework is compatible with such approaches, offering scalability to handle objects with complex geometry.\n\n**W2:**\n\nWe apologize for any confusion the name \"HumanoidGen\" may have caused to the reviewer. We would like to clarify that we have explicitly stated that our work focuses on generating data for bimanual dexterous manipulation tasks in the \"Data Generation for Bimanual Dexterous Manipulation\" part of our title. Both the abstract and introduction sections center around bimanual dexterous manipulation tasks, and we haven't found any content in the main body of the paper that could easily lead readers to misunderstand our work as claiming support for locomotion or whole-body control.\n\nIf the concern lies primarily in the naming, we are open to revising it by removing the \"HumanoidGen:\" part of our title to emphasize the scope of our work as reflected in the main title. Furthermore, we believe our framework also holds potential for future extension to coordinated manipulation involving the lower body.\n\n**W3:**\n\nWe appreciate the reviewer's comments and concerns regarding the limitations of our work. To further demonstrate our framework's capability in handling tasks involving bimanual coordination and dexterous hand manipulation, we conducted experiments on five additional tasks. As shown in the table below, our framework achieves a high success rate across the five new tasks, demonstrating its scalability to new tasks.\n\n| Evaluation Metric          |Dual Lift Pot          | Rotate Safe Knob | Open Safe Door | Press Toaster | Blocks Stack Hard With Barrier |\n| :------------------------ | :---------------: | :--------------: | :------------: | :-----------: | :-----------------------------: |\n| **Scene Randomization**(Position, Yaw) | 12 cm × 6 cm, ±15° | 5 cm × 5 cm, ±15° | 5 cm × 5 cm, ±15° | 5 cm × 5 cm, ±15° | 6 cm × 8 cm, ±10° |\n| **Success Rate (%)**     | 93.00             | 91.09            | 95.24          | 87.72         | 73.00                           |\n\n\nWe further address the reviewer's concern in detail below:\n\n1) Coordinated Motion: \nOur framework's adaptability to bimanual coordination tasks is demonstrated by the experiments in our manuscript and the additional \"lift pot\" experiment. We note that our contribution to bimanual coordination lies in leveraging LLMs to reason about the execution sequence of both arms and to dynamically avoid collisions arising from inter-arm interference at the planning level. Furthermore, we apply task decomposition with MCTS to enable diverse strategies in bimanual manipulation. For instance, in the “dual_bottles_pick_hard” task, the LLM must not only infer the constraints for grasping and placing blocks, but also determine how the two arms should coordinate, including the order of execution, the stacking locations, and the choice of base block or position. These choices, such as stacking on an existing block or selecting an empty region, require the LLM to reason over multiple coordination strategies. Therefore, we believe our framework can effectively handle bimanual coordination tasks using the proposed planning methods.\n\n2) Dexterous Hand Operation: \nCompared to parallel 2-finger grippers, the dexterous hand's superior structure provides greater flexibility in end-effector poses, but also introduces increased complexity in determining appropriate hand configurations during manipulation tasks. On the one hand, our work aims to strike a balance between flexibility and complexity by defining atomic operations, providing a simplified yet effective approach to enable automatic data collection for dexterous manipulation tasks. We also demonstrated our framework's capability to incorporate additional atomic dexterous hand operations through the additional tasks like   'Rotate Safe Knob' and 'Press Toaster' above. On the other hand, by adding more atomic operations and annotations, integrating additional annotation methods, and incorporating other in-hand manipulation policies as framework components, our framework can extend the current utilization methods of dexterous hands to new tasks and further expand and leverage the dexterity and flexibility of the dexterous hands.\n\nTo summarize, our framework supports bimanual coordination through MCTS-enhanced LLM reasoning with collision management at the planning stage. Moreover, we provide a method to balance the flexibility advantage and the complexity inherent in dexterous hand manipulation, and leverage the framework's scalability to enable further utilization of dexterous hands.\n\n**Questions**\n\n**Q1:**\n\nWe discussed this in the \"Weaknesses\" section. Overall, we define our framework to be a data collection framework for bimanual dexterous manipulation tasks, with the following key aspects to be noted: \n\n1) We leverage LLMs to generate task scenes and plan scripts, obtaining manipulation trajectories through an optimization-based motion solver. \n2) We leverage MCTS to enhance LLM reasoning for stable and diverse plan script generation. \n3) We provide diverse atomic manipulation operations in our framework, including grasping, pinching, pressing, pulling, and rotating, as well as basic operations on articulated objects, supporting a variety of bimanual dexterous manipulation tasks. \n4) Our framework's effectiveness is primarily demonstrated by our experiments with DP and DP3. \n5) The scalability of our framework is demonstrated by the annotation transfer experiment and further enables integration with works on affordance prediction and dexterous grasping pose generation, allowing extension to a broader range of object types.\n\nTo summarize, our framework focuses on bimanual dexterous manipulation tasks and demonstrates strong advantages in enabling automatic data collection for applications in this domain.\n\n**Q2:**\n\nOur framework is highly scalable, supporting extension to complex, contact-rich manipulation tasks. It leverages LLMs for script generation, enhanced by MCTS for higher planning success rate and diversity. Currently, it supports atomic operations such as grasping, pinching, pressing, and rotating, and can be expanded with new annotations, operations, and dexterous policies. In short, our framework efficiently scales to more complex tasks and richer atomic operations.\n\n\n**Q3:**\n\nWe answer this question from the following perspectives:\n\n1) Annotation scalability and generalization to objects: Our framework supports annotation transfer, enabling extension to diverse objects, as demonstrated in the additional experiments. Its modular design allows integration of new annotation methods (e.g., affordance prediction), atomic operations, and dexterous manipulation policies, facilitating generalization to more complex objects.\n\n2) Scene scalability and generalization to new tasks: As stated in our manuscript, the generated task scripts can be executed directly across all 120 RoboCasa scenes without additional annotation effort. We also elaborate on our framework's adaptability in the response to Question 2, highlighting its capabilities at both the LLM reasoning level and the atomic operation level. \n\nTo summarize, our framework demonstrates strong scalability and generalization, enabling effective adaptation to diverse objects, complex manipulations, and a wide range of unseen scenes and tasks through its flexible integration of high-level planning and low-level operation components."}, {"review_id": "aE3ETQK5gi", "reviewer": "Reviewer_A6rG", "summary": "This paper presents HumanoidGen, an automated framework for generating diverse bimanual dexterous manipulation tasks and collecting demonstrations for humanoid robots. The framework leverages atomic operations, extensive spatial annotations, and LLM-based task planners to generate chains of actionable spatial constraints, which are then solved by off-the-shelf optimizers. To handle long-horizon tasks and incomplete asset annotation, the authors introduce a MCTS variant to enhance reasoning. The proposed HGen-Bench benchmark covers 20 tasks with randomized scene configurations, and validates the generated data with 2D and 3D policies.", "questions": "- What hardware was used (GPU/CPU)? Approximately how much time does it take for asset annotation, demonstration generation (including LLM inference and trajectory planning), and overall data collection? Are there any observed computational bottlenecks?\n- How do you determine if two assets are similar enough for annotation migration? Do you have accuracy or failure statistics for the migration process, and what are typical failure cases?\n- Have you observed any LLM hallucinations in collision ignore list generation or planning with excessively large MCTS trees? How do you manage such complexity in the search process?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- Automated and Scalable Pipeline: The system is highly automated, from asset annotation to scene/task/code generation and demonstration collection, which greatly reduces the labelling and handcrafted programming workload by human.\n- Novel Method in Asset Annontation: The paper innovatively leverages Stable Diffusion encoders for automatic cross-asset annotation, minimizing the manual effort required for large-scale asset libraries.\n- Strong LLM Integration: Planning is formulated at a constraint-chain code level and solved by an LLM-augmented pipeline, including an MCTS variant to enhance reasoning and self-correction for long-horizon tasks.\n- Comprehensive Benchmark: The authors design a broad and diverse set of tasks (HGen-Bench) with clear evaluation metrics.\n\n**Weaknesses:**\n\n- Simulation-based Setup: Most experiments and policy learning are performed in simulation. The robustness and scalability to the real robot has not been validated yet.\n- Limited Discussion of Auto-annotation: Although the Stable Diffusion approach is well motivated, the manuscript does not quantify annotation accuracy or failure rates for point-axes migration compared to manual annotation, also it lacks discussion for potential matching failure due to dissimilar assets.\n- LLM Latency and Computational Load: The approach relies on multiple LLM calls and optimizer for path planning; the actual computational resource and bottleneck are not discussed.\n- Potential Risk of Hallucination: The LLM is responsible for dynamically generating the collision ignore list, but the paper does not discuss potential errors caused by LLM hallucination, which could lead to physically invalid collisions and noisy data.", "was_revised": true, "final_justification": "Overall, the paper is of high quality, and I am maintaining my score of 5.", "rebuttal": "Thank you for your insightful comments. We will explain your concerns point by point.\n\n**Weaknesses**\n\n**W1:**\n\nReal-World Experiments: We thank the reviewer for pointing out the lack of real-world experiments in our manuscript. In response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.\n\nWe conduct experiments following 5 steps:\n\n1. **Real-World Setup and Task Design.** We built a real scene corresponding to the simulation environment, which includes a Unitree H1_2 robot equipped with the Instant-Lab dexterous hand, a white table, and the task-related objects. Three tasks are designed for this data collection experiment: \"bimanual bottle grasping\", \"opening a laptop\", and \"placing a block\".\n\n2. **Real2Sim Assets.** For task-related objects, we prepared both real objects and their corresponding simulation digital assets. \nSpecifically, we obtained the digital assets of standard objects such as Coke cans and Sprite bottles from online asset repositories. For objects whose digital assets can hardly be found, such as the specific plate we used, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance. \n\n3. **Real2Sim Pose Estimation.** Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose. The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate. \n\n4. **Trajectory Collection.** We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task mentioned in 1) to train DP, trained for 300 iterations.\n\n5. **Sim2Real Policy Transfer.** To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 10 real-world trajectories plus N simulated trajectories. The results are as follows: \n\n|Evaluation Metric              | Dual Bottles Pick | Close Laptop| Notes|\n|-|-|-|-|\n| **Scene Randomization Range** |  6 cm x 6 cm, ±15°  | 6 cm x 6 cm, +10° to +20°\t   |Position (cm) and Yaw (°)|\n| **Data Collection Success Rate**  |  85% | 90%  |\n| **Data Collection Time**  |  16 |  13 |Seconds per episode\n| **DP Success Rate(5Real)** |  0%(0/20)  | 20%(4/20)  | N=20 trials per condition\n| **DP Success Rate(25Real)** | 60%(12/20)  | **70%(14/20)** |N=20 trials per condition\n| **DP Success Rate(100Sim+5Real)** | **65%(13/20)**  | **70%(14/20)**|N=20 trials per condition\n\nAs shown above, our approach consistently achieves over 85% success in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.\n\n\n**W2:**\n\nWe conducted a quantitative analysis of asset transferability for three types of objects: \"cup\", \"laptop\", and \"drawer\": \n\n| Evaluation Metric             | Cup (Handle) | Cup (Edge) | Bowl  | Laptop | Drawer |\n| :--------------------------- | :----------: | :--------: | :---: | :----: | :----: |\n| **Manual Annotation Time (s)** | 4.00         | 4.00       | 4.00  | 6.00   | 6.00   |\n| **Automated Migration Time (s)** | 0.72  | 0.72 | 0.79  | 0.81   | 0.80   |\n| **Migration Success Rate (%)** | 90.0 (18/20) | 100.0 (20/20) | 100.0 (20/20) | 100.0 (20/20) | 90.0 (18/20) |\n\nAs shown above, the time for automated annotation migration is significantly shorter than manual annotation, demonstrating its human-intervention-free advantage. This is especially evident when annotating complex articulated objects, whose operation points and axes often do not align with the object's geometric center or axes, making manual annotation non-intuitive and difficult. In contrast, automated annotation transfer does not face this issue.\n\nHowever, transfer is not always guaranteed to succeed. On the one hand, there are inherent limitations in the transfer model itself, such as potential point deviations—particularly when the asset is very small or far from the camera. On the other hand, even within the same asset category, the same action may apply to different locations. For example, cups without handles are typically grasped around the center, whereas cups with handles are usually grasped at the handle. Therefore, it is necessary to perform the transfer separately for these two types of cups.\n\n\n\n**W3:**\n\nWe conducted a quantitative analysis of time and resource consumption across different stages for 20 tasks. The statistics are as follows:\n\n|                               | Manual annotation | Automatic annotation | Scene Generation | Script Generate | Script Execution With Data Collection\n|-------------------------------|------|------|------|------|------|\n| Time | 4-6s  |  1.17s | 12.64s | 18.31s | 14.4s\n|Resource Cost | Manual|Auto|Auto LLM(avarage 2958Tokens)|Auto LLM(avarage 3745Tokens)|Auto Motion Planner(0.53s)\n\n\nThe above table summarizes data on a per-task basis. However, tasks can vary significantly depending on the number of execution steps involved. On average, a single motion planning step takes approximately 0.53s, and the input to the code generation model typically contains around 3745 tokens per call.\n\n\n**W4:**\n\nWe agree with the reviewer that LLMs can produce potential errors due to hallucination. We observed this problem in the first experiment of our paper, which led to the failure cases there. We believe this problem can be addressed by our MCTS-enhanced LLM reasoning approach. Specifically, in this approach, we handle such errors by storing relevant failure information at the corresponding node in the search tree where the error occurred. When MCTS revisits that node during future exploration, this information is provided to the LLM as part of the prompt. As shown in our second experiment in the paper, this approach allows the LLM to proactively avoid repeating similar mistakes, which can help it avoid such hallucinations. Besides, our framework executes LLM-generated plans in simulation to collect trajectories once the automatically generated task conditions are satisfied. This helps avoid potential issues related to data quality and experimental safety that may arise from LLM hallucinations.\n\n\n**Questions**\n\n**Q1:**\n\nThe GPU and CPU used in our experiments are Nvidia RTX 4090 and Intel Core i9-14900K, respectively. The details of time consumption in our framework are mentioned in the \"weakness\" section. As for computational bottlenecks, we observed that such bottlenecks exist in GPU and RAM resources when parallelizing multiple simulation environments for training data collection.\n\n\n**Q2:**\n\nAs mentioned in the \"Weaknesses\" section, transferable assets must belong to the same category and share identical geometric features; otherwise, the transfer may fail or introduce significant errors.\nTypical examples include cups with handles versus those without, or drawers with round handles versus horizontal handles, as discussed in the \"weakness\" section. \n\n**Q3:**\n\nReferring to the \"Weaknesses\" section, we do encounter hallucination issues when using LLMs. To manage this complexity, we record the specific cause of any execution error induced by hallucination, such as syntax errors, runtime failures, or task failures, at the corresponding tree node where truncation occurs. This mechanism helps prevent the recurrence of the same hallucination during subsequent planning exploration."}, {"review_id": "4kVLbEYIY9", "reviewer": "Reviewer_uN4c", "summary": "This paper introduces HumanoidGen, an automated framework for generating high-quality demonstration data for humanoid robots performing bimanual dexterous manipulation tasks. Unlike existing robotics datasets that primarily focus on single-arm manipulation, HumanoidGen specifically addresses the complexities of coordinating dual arms with dexterous hands. The framework employs three key components: (1) detailed spatial annotations for both assets and dexterous hands based on atomic operations, (2) an LLM-based planning system that generates chains of actionable spatial constraints for arm movements, and (3) a Monte Carlo Tree Search (MCTS) enhancement to improve LLM reasoning for complex long-horizon tasks. The authors develop a comprehensive benchmark called HGen-Bench containing 20 diverse manipulation tasks performed by a Unitree H1-2 humanoid robot with Inspire hands in simulation. Experiments demonstrate that HumanoidGen achieves superior success rates compared to existing methods, especially for long-horizon tasks with complex collision scenarios. The authors also show that the collected demonstration data effectively scales with 2D and 3D diffusion policies, confirming the quality and utility of the generated datasets.", "questions": "To what extent is your LLM-based approach truly automated? Does the pipeline require human filtering, validation, or intervention at critical stages, such as spatial annotation creation, constraint generation, or demonstration selection? If human oversight is needed, could you elaborate on these touchpoints and discuss how they might impact the scalability of your method for larger object libraries or more diverse task domains? Additionally, have you explored methods to further automate these potentially manual processes?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Effective system for data generation: The paper addresses the significant lack of high-quality bimanual dexterous manipulation data, which is crucial for advancing humanoid robotics.\n- Comprehensive spatial annotation system: The detailed framework for annotating key points and axes for both assets and hands provides a solid foundation for LLM-based planning.\n- Innovative constraint-based formulation: By framing manipulation as a sequence of spatial constraints rather than pre-defined functions, the approach enables more flexible and generalizable task execution.\n\nWeaknesses:\n- Reliance on extensive annotations: The approach requires detailed spatial annotations for both assets and hand operations, which could limit scalability to new objects and environments.\n- Limited real-world validation: All experiments are conducted in simulation, leaving questions about how well the approach would transfer to real-world humanoid robots with physical constraints.\n- Task complexity limitations: Despite improvements, the system still struggles with highly complex bimanual long-horizon tasks (e.g., HandOver and Storage), achieving lower success rates compared to simpler tasks.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed several of my concerns; therefore, I will maintain my original score.", "rebuttal": "We thank the reviewer for the encouraging feedback and valuable comments.\n\n**Weaknesses**\n\n**W1:**\n\n**The scalability to new objects**: To evaluate the scalability of our framework across a diverse set of manipulable assets, we perform supplementary annotation experiments on five object categories: two rigid types of \"cup\" (with and without handle), \"bowl\", and two articulated objects, namely \"laptop\" and \"drawer\". For transferring annotations across similar objects, we employ the Stable Diffusion-based method introduced in the paper.\n\n| Evaluation Metric                                  | Cup (Handle) |  Cup (Edge)   |     Bowl      |    Laptop     |    Drawer    |\n| :------------------------------------------------- | :----------: | :-----------: | :-----------: | :-----------: | :----------: |\n| **Manual Annotation Time (s)**                     |     4.00     |     4.00      |     4.00      |     6.00      |     6.00     |\n| **Automatic Annotation Transfer Time (s)**         |     0.72     |     0.72      |     0.79      |     0.81      |     0.80     |\n| **Automatic Annotation Transfer Success Rate (%)** | 90.0 (18/20) | 100.0 (20/20) | 100.0 (20/20) | 100.0 (20/20) | 90.0 (18/20) |\n\nFor a new object, manually annotating an autonomous action point along with its axis takes only 4 seconds for rigid objects and 6 seconds for articulated objects, thanks to the SAPIEN visualization interface and our annotation-assistive scripts. For automatic annotation transfer, the objects listed in the table are randomly selected from the SAPIEN asset library. The success rates are all above 90%, and the transfer time is under one second, which ensures the scalability of our approach to large-scale asset migration.\n\nBesides, the Stable Diffusion-based method is just one of the approaches we use to simplify the annotation process. Many existing works have explored affordance extraction and structured object axis annotation, such as RoboBrain(CVPR 2025), and CAPNET(CVPR 2025). These methods can be seamlessly integrated into our framework to support zero-shot asset annotation and enable large-scale asset expansion.\n\n**Scalability to New Scenes**: Our annotation process is object-centric. Once atomic actions are annotated for an object, they can be applied across different scenes involving that object. At the room level, our framework leverages the rich asset information in RoboCasa to directly generalize tasks to 120 scenes, significantly increasing scene diversity. At the table level, our framework integrates constraint-optimization-based target pose solving, dynamic active collision handling, and MCTS-enhanced reasoning. These techniques enable robust adaptation to task scenarios involving object randomization, cluttered environments, and diverse execution strategies.\n\n**W2:**\n\nReal-World Experiments: We thank the reviewer for pointing out the lack of real-world experiments in our manuscript. In response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.\n\nWe conduct experiments following 5 steps:\n1）**Real-World Setup and Task Design.** We built a real scene corresponding to the simulation environment, which includes a Unitree H1_2 robot equipped with the Instant-Lab dexterous hand, a white table, and the task-related objects. Three tasks are designed for this data collection experiment: \"bimanual bottle grasping\", \"opening a laptop\", and \"placing a block\".\n2) **Real2Sim Assets.** For task-related objects, we prepared both real objects and their corresponding simulation digital assets. \nSpecifically, we obtained the digital assets of standard objects such as Coke cans and Sprite bottles from online asset repositories. For objects whose digital assets can hardly be found, such as the specific plate we used, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance. \n3) **Real2Sim Pose Estimation.** Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose(CVPR 2024). The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate. \n4) **Trajectory Collection.** We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task mentioned in 1) to train DP, trained for 300 iterations.\n5) **Sim2Real Policy Transfer.** To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 10 real-world trajectories plus N simulated trajectories. The results are as follows: \n\n| Evaluation Metric                 | Dual Bottles Pick | Close Laptop              | Notes                     |\n| --------------------------------- | ----------------- | ------------------------- | ------------------------- |\n| **Scene Randomization Range**     | 6 cm x 6 cm, ±15° | 6 cm x 6 cm, +10° to +20° | Position (cm) and Yaw (°) |\n| **Data Collection Success Rate**  | 85%               | 90%                       |                           |\n| **Data Collection Time**          | 16                | 13                        | Seconds per episode       |\n| **DP Success Rate(5Real)**        | 0%(0/20)          | 20%(4/20)                 | N=20 trials per condition |\n| **DP Success Rate(25Real)**       | 60%(12/20)        | **70%(14/20)**            | N=20 trials per condition |\n| **DP Success Rate(100Sim+5Real)** | **65%(13/20)**    | **70%(14/20)**            | N=20 trials per condition |\n\nAs shown above, our approach consistently achieves over 85% success rate in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.\n\n**W3:**\n\nFor complex tasks, which typically involve more execution steps and more intricate bimanual coordination, reasoning becomes more challenging for LLMs, leading to a significant drop in success rate compared to simpler tasks. Despite this challenge, our framework still maintains a success rate above 50%, showing its capability in handling bimanual complex and long-horizon tasks. Besides, this planning process can be further enhanced by MCTS to improve both the reasoning success rate and the diversity of execution plans. Furthermore, our collision avoidance mechanisms enable the framework to excel in executing tasks with complex collision scenarios. This is demonstrated by our significantly higher success rate in tasks involving complex collision scenarios compared to RoboTwin, as detailed in our manuscript.\n\n\n**Questions**\n\n**Q1:**\n\nWe answer the questions as follows:\n1) Our approach is automated with object annotations and the necessary atomic operations provided. \n2) We list manual parts of our work as follows: On one hand, the atomic operations in our framework are currently manually predefined to ensure stable performance across diverse dexterous manipulation processes. On the other hand, for annotation transfer, our framework currently requires a separate manual annotation for one object per category to serve as a reference. Besides, we note that once these initial conditions are met, our framework enables fully automatic data collection, eliminating the need for manual effort in constraint generation or demonstration selection.\n3) In terms of the manual labor involved in annotation transfer, it currently represents the minimal necessary effort required to scale our framework to a larger object library. We believe this level of manual input is highly efficient compared to fully manual annotation and significantly enhances the scalability of our framework. Moreover, we note that our framework is compatible with methods for automatic object annotation, such as affordance prediction and dexterous grasping pose generation, which could further reduce human intervention in the annotation process.\n\n**Q2:**\n\nWe note some possible approaches for further automating the whole process:\n1) Regarding the predefinition of atomic operations, we notice the recent advancements in research on dexterous hand manipulation pose generation. For example, DexGraspAnything(CVPR 2024) and DexVLG generate grasping and manipulation poses for dexterous hands based on 3D and/or 2D perception. Since 2D and 3D information can be provided accurately and reliably in simulation, we believe integrating these methods into our framework could further enhance the data collection process by automatically providing diverse and feasible end-effector pose candidates, reducing the need for manual definition and expanding the range of supported manipulation operations.\n2) For object annotation, our framework supports the integration of additional methods to further automate the annotation process. These methods include affordance point and axis prediction approaches, such as CAPNET(CVPR 2025) and RoboBrain(CVPR 2025), which can provide the necessary annotations to enable the successful execution of our framework.\nIn summary, our framework has high scalability, enabling the integration of additional methods to improve the level of automation in the processes mentioned above."}]

Author rebuttal:

Meta-review: The paper proposes HumanoidGen, a framework for automatically creating tasks and collecting demonstrations through atomic dexterous operations combined with LLM-based reasoning. A new benchmark is introduced and evaluated using both 2D and 3D diffusion policies.

The main concerns include:
1. Lack of real-world experiments (NeLy, A6rG, uN4c)
2. Requiring considerable human intervention and labels  (NeLy, mhYR, uN4c)
3. Missing comparison with augmentation-style methods (NeLy)
4. Overgeneralization of the scope and the actual use of dual-arm coordination and dexterity is limited (mhYR)
5. Lack computation cost description (A6rG)
6. Low performance on complex bimanual long-horizon tasks (uN4c)

The rebuttal adequately addressed most concerns. Three reviewers remained positive (1 Accept, 2 Borderline Accept), while the single negative reviewer (Borderline Reject) did not engage further. Given the overall consensus, the AC recommends acceptance, with the expectation that the authors incorporate the raised feedback into the final version.
Decision: Accept (poster)