RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
Authors: Chunru Lin
Haotian Yuan
Yian Wang
Xiaowen Qiu
Tsun-Hsuan Wang
Minghao Guo
Bohan Wang
Yashraj Narang
Dieter Fox
Chuang Gan
Venue: NeurIPS 2025 poster ()
Keywords: Robotic Tool Design
Robotic Tool Use
Soft Material Manipulation
Generative Models
Primary area: reinforcement_learning

Abstract: Endowing robots with tool design abilities is critical for enabling them to solve complex manipulation tasks that would otherwise be intractable. While recent generative frameworks can automatically synthesize task settings—such as 3D scenes and reward functions—they have not yet addressed the challenge of tool-use scenarios. Simply retrieving human-designed tools might not be ideal since many tools (e.g., a rolling pin) are difficult for robotic manipulators to handle. Furthermore, existing tool design approaches either rely on predefined templates with limited parameter tuning or apply generic 3D generation methods that are not optimized for tool creation.
To address these limitations, we propose **RobotSmith**, an automated pipeline that leverages the implicit physical knowledge embedded in vision-language models (VLMs) alongside the more accurate physics provided by physics simulations to design and use tools for robotic manipulation. Our system (1) iteratively proposes tool designs using collaborative VLM agents, (2) generates low-level robot trajectories for tool use, and (3) jointly optimizes tool geometry and usage for task performance.
We evaluate our approach across a wide range of manipulation tasks involving rigid, deformable, and fluid objects. Experiments show that our method consistently outperforms strong baselines in both task success rate and overall performance. Notably, our approach achieves a 50.0\% average success rate, significantly surpassing other baselines such as 3D generation (21.4\%) and tool retrieval (11.1\%). Finally, we deploy our system in real-world settings, demonstrating that the generated tools and their usage plans transfer effectively to physical execution, validating the practicality and generalization capabilities of our approach.

RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
Chunru Lin""; Haotian Yuan',, Yian Wang", Xiaowen Qiu', Tsun-Hsuan Wang", Minghao Guo", Bohan Wang', Yashraj Narang'. Dieter Fox*, Chuang Gan Unniversity of Massachusetts Amherst
"Massachusetts Institute of Technology *NVIDIA SMIT-IBM Watson Al Lab
National University of Singapore
Abstract
Endowing robots with tool design abilities is critical for enabling them to solve complex manipulation tasks that would otherwise be intractable. While recent generative frameworks scenes and reward functions, they have not yet addressed the challenge of tool- scenarios. Simply retrieving suman-designed tools might not be ideal since many tools (e.g., rolling pin) are difficult for robotic manipulators to handle. Furthermore, existing tool design approaches either rely on predefined templates with limited parameter tuning or apply generic 3D generation methods that are not optimized for tool creation. To address these limitations, we propose RobotSmith, an automated pipeline that leverages the implicit physical knowledge embedded in vision-language models (VLMs) alongside the more accurate physics provided by physics simulations to design and use tools for robotic manipulation. Our system (1) iteratively proposes tool designs using collaborative VLM agents, (2) generates low-level robot trajectories for tool use, and (3) jointly optimizes tool geometry and usage for task performance. We evaluate our approach across a wide range of manipulation tasks involving rigid, deformable, and fluid objects. Experiments show that our method consistently outperforms strong baselines terms of both task success rate and overall performance. Notably, our approach achieves a 50.0% average success rate, significantly surpassing other baselines such 3D generation (21.4%) and tool retrieval (11.1%). Finally, we deploy our system in real-world settings, demonstrating that the generated tools and their usage plans transfer effectively to physical execution, validating the practicality and generalization capabilities of our approach
automatically synthesize task settings, such 3D
Introduction
Tool use is fundamental capability of intelligent agents [6 49
enabling them to extend their physical affordances to accomplish tasks that would otherwise be unattainable. In both biological and artificial systems, the ability to use tools is critical for performing complex interactions with the environment, particularly under morphological or task-specific constraints [60 35 13 [8]. For robots operating in unstructured settings, developing robust tool-use behaviors is essential for achieving coneral-purpose manipulation capabilities.
Recently, a number of generative frameworks for robotic manipulation have emerged [54 51 53 12 20 23 32), demonstrating strong potential to scale up data collection and task generalization. However, these frameworks have overlooked tool-use scenarios, they typically rely on retrieving
'denotes equal contribution
Project page:
github. io/RobotSmithh
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Real
Sim
Initial
Flatten Dough
Scoop Sauce
Spread Sauce
Sprinkle Sesame Before/After Baking
Figure 1: Our pipeline enables fully autonomous long-horizon manipulation by generating and using tools for each subtask without human invol The robot completes the entire pancake-making process from shaping dough to spreading sauce and adding sesame. The bottom row shows simulated execution with generated tools, while the top row depicts the corresponding real-world stages of pancake preparation.
objects from datasets curated for human use rather than robotic manipulation. For example, rolling pin might be selected for dough-flattening task, but such a tool is difficult for robotic arm to manipulate, A more suitable alternative for robots might be a custom-destt -estgned presser composed of flat board and handle shown in Figure ] Hence, shifting from using a predefined, often limited toolset to designing tools tailored to specific contexts, including different scenarios and robots, may unlock new possibilities for purppsse manipulation.
Existing tool-design approaches fall short in addressing this gap. One line of work 35
B
18
26
assumes predefined tool templates tailored for specifie tasks and only optimizes the parameters within those templates. While efficient, such methods struggle to generalize beyond narrow set of tool types, such rigid-link structures. Another line of work explores tool generation by optimizing geometry in the latent space of 3D generative models [9) (2). However, these methods depend heavily on the quality and task alignment of the learned latent space, which often not specifically designed for tool representation -ineittttiooniiminningg the scope and effectiveness of the generated tools.
To bridge this gap, there is clear need for automatic tool-design framework capable of generating effective, task-specific tools from scratch. Large pretrained models, such as vision-language models (VLMs), have demonstrated strong capabilities in spatial reasoning (63 10,, affordance perception
60 27.. and commonsense physical inference [11 55 -all of which are essential forms of prior knowledge for tool design. For example, spatial reasoning enables the model determine how a tool should be shaped (c.g., connectivity and size) to interact effectively with objects in the environment. Affordance perception allows it infer what geometric features are needed to enable specific functions such as scooping, pressing, or lifting. Commonsense physical inference further helps anticipate how the tool's geometry and structure will affect its physical interactions, ensuring it can successfully fulfill the task.
To this end, propose RobotSmith, automated framework that integrates the rich prior knowledge in large foundation models with a joint optimization process in physics simulation to enable both tool design and tool use for robotic manipulation. Given specified manipulation task, RobotSmith generates customized tool geometry, determines its placement within the scene, and synthesizes corresponding manipulation trajectory for the robot. The framework is built around three core components:
1. Critic Tool Designer: to generate candidate tool geometries grounded in visual context and described in json format.
Two collaborating VLM agents engage in iterative proposal and refinement
2. Tool Use Planner: Based on the designed tool and scene configuration, the system generates a manipulation trajectory using a minimal set of APIs abstract away low-level control.
3. Join Optimizer: The tool geometry and trajectory parameters are jointly fine-tuned through feedback from physed simulation to maximize task performance.

---

We evaluate the proposed framework across suite of simulated manipulation tasks requiring nontrivial tool use, such as reaching an object outside the workspace, pouring water into a narrow mouth bottle, cutting dough, and so on. Our method demonstrates superior performance relative to several baselines, including those using fixed tool libraries or generating tools directly. To assess physical feasibility and transferability, we additionally validate the approach in real-world settings: selected tools are fabricated using 3D printing, and their corresponding trajectories are executed on a physical robotic platform. The real-world experiments confirm that the generated designs and plans can be successfully realized in practice.
In summary, our key contributions are as follows:
We propose unified framework that integrates large foundation models with physics- based simulation to enable fully automated tool design and tool-use planning for complex robotic manipulation tasks. Our full pipeline, including code and APIs, will be made publicly available to support further research.
We propose parameterized, modular tool representation with a set of APIs for tool geometry synthesis. This enables a dual-agent, VL.M-based designer to iteratively generate and refine tool designs using visual feedback, supporting precise, editable tools that ground high-level concepts in physically realizable geometry.
We validate our approach in both simulated and real-world settings, demonstrating that it outperforms baseline methods and enables tasks that are otherwise difficult to solve without tools in simulation, improving the success rate from 2.8% to 50%, and confirming its physical plausibility through real-world executions across diverse manipulation tasks.
2
Related Works
2.1 Generative Pipelines in Robotics
Many recent works have explored building generative pipelines to automatically collect robotic manipulation data, offering the potential to scale up synthetic data generation for training and These pipelines typically involve several key components,
evaluation [54] 51 9 53
12
20
23
38
including task generation 54
51
23
33
50],
scene synthesis [53 54 32
41 (66, and reward
function design [6] 5 59
46
48
36
34
62 However, most existing efforts focus simple,
rigid-body tasks such as pick pick-and-py place or door opening/closing. with limited attention to more complex scenarios like manipulating deformable materials. Such tasks often require specialized tools (e.g., spoons, dough pressers) that are adapted for robotic use, which current pipelines cannot automatically generate. Our pipeline aims fill in this gap by automatically generating suitable tools for each given tasks.
2.2 3D Tool Design
Designing task-specific tools is labor-intensive and typically requires significant domain expertise. Prior work in tool design has generally targeted specific tasks using predefined tool templates [1.
35 13 18] repurposed cxisting objects in the scene [28 40 60), or employed generative 22
models synthesize tools from scratch 19]. These approaches optimize tool design using various
methods, including differentiable physics 22 stochastic optimization 14,, reinforcement learning
35 13 28]. and supervised learning for generative frameworks [57 19 However, none of these methods are fully scalable or automated across diverse task settings. They are typically constrained by predefined templates or rely heavily domain-specific data, limiting their ability to generalize to broader tool-use scenarios. In contrast, our method is designed to handle more diverse and complex situations by leveraging the prior knowledge embedded in foundation models and the robustness of physics-based simulation 30)
2.3 Robotic Tool Use
Learning to use tools a crucial capability for robots, significantly expanding the range of tasks they can perform. Prior research [25]]
has explored teaching robots to use tools to accomplish complex tasks, such as manipulating chopsticks (55. More recently, RoboCook [47
39 17
29
24
45
enabled a robot to make dumplings using tools, simplifying the challenging problem of deformable-
3

---

(a) Critic Tool Designer
Tool Use Planner
Joint Optimizer
Design
freaching
tool for
Proposer
"parts":lt
"geon": "Cylinder",
"parars" [[0.01,0.2] iis handle";
User
Cylinder: 0.4]
Box:[e.05, [8,05,8.15, 0.1,0 Box:[0. [0.10,0.10, 0.02]
outside workspace).
}, ...,, "assembly"s
(Too)(
(oo1)
20. 0,2,0, 0,00)
20ve(0.2,0.2,0.222
The scene configurat
Nove(0.1,..0,0.11.5..0
...
Nov0(0.1,000,0.1,55.8
Modify: I. Longer mrck
(...).
Cyitee
CMA-ES Joint Optimization
"parts":[( I[ "geon"; "geon"="cyLinder" "Cylinder", [[0,01,
(1..).
Proposer
Cylindor: 8.4]
"is handle": false b ...], "assenbly":..
Box: lox:[00.55 (0.05,,0 0.,,8,,,,11..... 03.,5
Box:
Gras)(Tool) Move 22,
Done!
Move 02,,
Critie
Figure 2: Pipeline Overview. The pipeline consists of three modules: (a) Critic Tool Designer, where two VLM agents iteratively propose and evaluate tool designs. The proposer takes in the user prompt, outputs a design, and renders an image, which is passed to the critic for feedback. This process continues until the critic is satisfied with the design; (b) Tool Use Planner that generates usage plans with minimal set of high-level robot APIs; and (c) Joint Optimizer that fine-tunes tool geometry and trajectory based on task performance.
object manipulation and making it feasible for robotic arm. These approaches typically rely on existing or husan-designed tools and learn their use through affordance modeling 15
44
dynamies prediction [7 3 58 31. or guidance from large language models [60]. However,
approaches are often task-specific and assume the availability of suitable tools. Prior works [35] [3 [8] take a step further by jointly learning tool parameters and the policy for using them. Nonetheless, this method still requires predefined tool template for each task. In contrast, our approach addresses tool design as part of the problem. Given a task that requires tool, we jointly optimize both tool's 's design and the corresponding robot trajectory, enabling the agent reason about both aspects simultaneously.
Method
Problem Setting. Each manipulation task is defined (o, where T is natural language task description (c.g., lift a bowl), So is the initial 3D scene configuration that specifies the objects and their spatial arrangement (c.g., bow1=Mesh ( 'bowl. 'bowl.bj',
pos=(0,0.5 0.3,0), euler= (0, 0)))), and M a task-specific reward function (e.g.. bowl height and no gripper contact). The system aims to produce (9, (p. e), where G is 3D mesh of the tool (detailed in Sec. 3.1. (p.c) E R R the initial placement of the tool, and a = {a), a2...., a, is robotic manipulation trajectory composed of high-level actions (described in Sec. 3.3 Task success is then evaluated using the metric M(S U (p, e), a.), computed via physics- physics-based simulation or physical execution.
We develop automated framework that jointly addresses tool design and tool use for robotic manipulation tasks. The framework. which we refer to as RobotSmith, consists three components: (1) Critic Tool Designer 3.2 (2) Tool Use Planner and (3) Joint Optimizer 3.4] An overview is shown in Fig. 2 and Fig. g To support flexible and interpretable tool generation, we also introduce a modular and parameterized representation for tools in section 3.1
3.1 Tool Representation
We design a modular and parameterized representation for tools that supports both procedural construction and parameterized refinement. Each tool consists of set of parts, either basic geometric primitives (e.g., box, sphere, cylinder) or complex shapes generated by a text-to-3D model, along with size parameters.

---

To specify how individual parts are positioned and connected. we introduce an assembly function, programmatic representation inspired by Constructive Solid Geometry (2]. Tools are constructed as a tree of operations applied to parts, where each node corresponds to geometric transformation, either unary (e-g., translation, rotation, scaling) or binary (e.g., union, subtraction, alignment). Although implicit, this CSG-like structure promotes valid tool construction by avoiding self-intersections and ensuring physical connectivity between components. By modularizing geometry and assembly logic, the representation supports flexible refinement: for example, directly shrinking part may break connections with adjacent components, but the assembly function enforces alignment constraints, allowing the system to automatically reposition parts and maintain structural coherence. An example of our tool representation is in Appendix A
Another key advantage of our representation is that all tool parts are parameterized. During the joint optimization stage, we can fine-tune parameters like size, position, or orientation to improve task performance without redesigning the tool from scratch. Moreover, by combining procedural primitives with generative 3D components, our framework can represent a broader range of tools than prior methods (Sec. 4.3) capturing both simple structures and semantically meaningful geometries.
3.2
Critic Tool Designer
The module consists of two collaborating vision-language model (VLM) agents: Proposer agent and a Critic agent based on 03-mini 42). These agents engage in an iterative design loop. as illustrated in Fig. 2 (a).
Specifically, the Proposer receives the task description and initial scene setup as input and proposes an initial tool design, which is then rendered into multi-view images passed to the Critic along with the design rationale. The Critic provides detailed feedback based on both task-specific requirements and user-defined structural principles; for example, ensuring that all parts connected, that the tool is graspable, and that the tool is long enough to avoid undesired contact (e.g.. with water). This feedback returned the Proposer, which refines the tool accordingly. This process repeats iteratively until the Critic returns a termination signal ("Done"), indicating that the design satisfies all key constraints. This approach offers an effective way to explicitly embed important design constraints and principles into the tool generation process.
3.3 Tool Use Planner
Once the Critic approves the tool design, it prompts Tool User agent to generate initial robot trajectory for using the tool, as shown in Fig. (b). This trajectory is represented as a sequence of calls to three minimal APIs:
grasp euler):
grasps the object with the specified orientation, implemented via sampling-based search over feasible grasp poses.
release (): opens the gripper to let go of the object. move move(pos, (pos, euler): moves the gripper
target position and orientation.
Our grasp euler) API is designed to robustly grasp objects given desired gripper ori- entation, specified in Euler angles. It takes as input the target object and the intended orientation of the gripper during grasping. We then randomly sample pairs of farthest points on the object's surface to identify potential contact positions for grasping. These candidates are filtered based on their alignment with the input gripper orientation, discarding those that deviate significantly. For each valid candidate, attempt to execute a grasp in simulation by lifting the object and checking whether it remains securely held. Upon the first successful attempt, the corresponding grasp pose is recorded. Subsequent calls to grasp grbs, euler) reuse this cached pose, ensuring consistent and efficient execution. move (obj, euler) is implemented by first solving inverse kinematics (IK) to get target joint pose, followed by motion planning to avoid collisions.
3.4 Optimizer
While LLMs and VLMs offer strong priors and can propose reasonable initial designs and action plans, they often lack the precision required for physically- (b), the initial trajectory and design proposed by VLM agent fails to accurately execute the task. Therefore, after the initial selection of candidate tool shapes and trajectories, perform a joint
tasks. As demonstrated in Fig. 2
5

---

Reaec
Holda Phane
1:ft Bowl
1: iift Pieaty
Dough Calabash
Cui Dough
Flaiten Dough
Bottie
Transpor Water
Figure 3: Gallery of the curated robotic manipulation tasks, that spans diverse physical and functional settings, including rigid, deformable, and fluid objects. This diversity allows comprehensive testing of tool design and usage capabilities across realistic and challenging scenarios.
optimization process to refine both components in tandem. This step is crucial, since suboptimal pairing-ee even with a plausible tool or trajectory alone- can fail to complete the task.
In our pipeline, the agent in previous steps determines both which parameters to optimize and their initial values. Let q Rd, denote the trajectory parameters and Rd, denote the tool shape parameters. As illustrated Fig. 2 (c). corresponds to the input parameters (position and Fuler angles) of the Move API, while defines the dimensions of the cylindrical and box components that make up the tool.
We support geometry optimization by allowing each shape parameter 8; E vary within range of [0.5 : 2.0s; ( where o is the initial value. The trajectory parameters q include translational and rotational waypoints in Cartesian space. Translational components are constrained to lie within +0.2 m of their initial values, while rotational components may vary within =m radians.
Since the initial proposal (s(0), ())) generated by the language model often fails to complete the task, we apply the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [21] to optimize the joint parameter (s, q). CMA-ES searches for improved solutions by iteratively sampling population of candidates {(s. q:) where À 20 is the population size. Each optimization run for 50 iterations.
To evaluate each candidate solution, we execute the trajectory with the corresponding tool in the Genesis simulator and compute task-specific scalar objective: MMss, q:). which reflects the success quality of task completion. This objective is then used to guide CMA-ES toward better configurations. Please refer to Appendix [for the specific evaluation metrics used for each task.
4 Experiments
4.1
Setup
Task Design. As in Fig. we curated robotic manipulation tasks inspired by everyday human activities. These tasks vary widely in both physical properties and functional requirements, involving objects made of rigid, liquid, and soft bodies. This diversity ensures that our tool design module accommodates range of physical interactions, while also allowing our pipeline to be rigorously evaluated across realistic and varied challenges encountered in daily life. The task details can be found in Appendix
Metrics. We scale each task's metric M standard score P E [0, 1]. We run each experiment times to report the best score Phest and the overall success rate SR [[P>0. Please refer to Appendix for more details.
We test our pipeline using the Genesis simulator and conduct experiments on both NVIDIA GeForce RTX 4090 GPUs and 2080 Ti GPUs.

---

Table I: Main results in simulation. We report Pres / SR, where Pbes is the best performance out of 8 trials (normalized to 0-1), and SK is the success rate over those trials. A trial is considered successful if the P score exceeds 0.8.
Method
Reach
Hold Phone
Lift Bowl
Lift Piggy
Dough Calabash
No Tool
0.00/
0.0%
0.09/ / 0.0%
0.33 / 0.0%
0.00 0.0%
0.56 / 0.0%
Retrieval
0.09/
0.0%
0.84 / 25.0%
0.30 / 0.0%
1.00 / 25.0%
0.21 / 0.0%
Meshy 37
0.69
0.0%
0.99 .5%
0.13/ 0.0%
1.00 / 25.0%
0.93 25.0%
Ours
0.93 / 50.0% %
0.95 75.0% %
1.00 / 50.0% %
1.00 / 87.5 %
0.84 / 25.0% %
Method
Flatten Dough
Cut Dough
Fill Bottle
Transport Water
Overall
No Tool
0.24/ 0.0%
0.82 25.0%
0.08 / 0.0%
0.00 / 0.0%
0.24/ / 2.8%
Retrieval
0.56 / 0.0%
1.00 / 37.5 %
0.12/ / 0.0%
0.67 / 12.5%
0.53 / 11.1%
Meshy 37
1.00 12.5%
0.86 37.5%
0.85 / 25.0%
0.00 0.0%
0.72 21.4%
Ours
1.00 / 62.5 %
0.95 / 37.5%
0.87 / 37.5%
0.89 / 25.0% %
0.94 / 50.0% %
4.2 Baselines
We compare our method against several baseline approaches:
No Tool: The robot attempts to complete the task using only its default end-effector. An LLM agent is prompted to generate an initial manipulation trajectory, which is then optimized using CMA-ES.
Retrieval: Suitable tools are selected from a curated 3D asset database based on semantic similarity to the task description, where we use BlenderKit [6]ss embodied search engine to retrieve the top- matching tools by "a tool for T". The LLM agent provides an initial trajectory, which is subsequently optimized.
3D Generation (Meshy): We use Meshy 377, a pretrained text-to-3D generation model, directly synthesize tool meshes from prompts such as "a tool 7". As in other baselines, the trajectory is initialized by an LLM and refined using CMA-ES.
3D Editing (Slk):Talk): We adopt ShapeTalk M. a framework to facilitate fine- ane--grinne 3D shape editing through natural language instructions, a baseline for collaborative 3D editing. It is integrated with the Critic agent to iteratively generate and refine tool designs via natural language interaction. Still, the trajectory is initialized using an LLM prompt and optimized with CMA-ES.
Results As shown in Tab. [ our method consistently outperforms all baseline approaches in both task suc- cess rate and overall performance across most tasks.
Transport Water
Bottle
Flatten Dough
Reach
Retrieval
Baseline No Tool struggles to com- plete any of the tasks, highlighting the essential role of purposeful tool design in enabling complex manipu- lations. In rare cases, such as cutting dough, success is possible, but often in unintended ways ways-for for instance, the robot nearly crushes the dough, break- ing it into just two uneven parts rather than executing a clean cut.
Meshy
ShapeTalk
(adlind)
Figure 4: Failure cases of baseline methods.
Baseline Tool Retrieval offers moderate improvement but often selects tools originally designed for human use, such as standard rolling pin for dough flattening, which are poorly suited for robotic grasping and control. Additionally, retrieval systems rely heavily on keyword-basee and frequently misalign functional intent with suitable geometry. For example, as shown in Fig. 4 querying flatten dough may yield hammer due to linguistic associations, despite the tool being physically inappropriate. Similarly, retrieval might suggest a fork for reaching, or return tools with mismatched scale and proportions.
matching
7

---

For the Meshy baseline, due to their flexibility in shape generation and strong priors, they perform reasonably well on tasks that require specific tool geometries, such as dough calabash, cut dough, fill bottle, and hold a phone. These tasks benefit from the model's ability generate complex and diverse shapes. However, this method tends to over-emphasize visual realism, such as smoothness, symmetry, and texture, rather than functional utility. As illustrated in Fig. f in the bottle-filling task, the model generates funnel with three pipes, disrupting controlled water flow. These examples highlight common limitation: despite visually plausible outputs, the lack of functional grounding often leads to failures in physical execution.
ShapeTalk, representative 3D editing method, falls short in all our tasks. While it allows iterative editing, its reliance on natural language and implicit representations makes difficult perform meaningful modifications. In practice, we observe that edits often lead to incoherent or unusable tools. We deem that limitation stems from its lack of geometric and functional grounding. In contrast, our method leverages a parameterized and part-based representation with explicit assembly logic, making inherently more editable, tunable, and interpretable. This design enables precise, ceauuticalll-meaninne modifications and better adaptation to task-specific constraints.
4.3
Tool Diversity
As shown in Fig. 5 our framework can generate diverse tool designs for the same task, each offering different functionality, such as pushing, scooping, or enclosing. This variety arises from our generative process, which explores multiple valid solutions guided by task goals and physical constraints.
Reach
Flatten Dough
Figure 5: Diverse effective tools generated by Figure 6: Real-world execution of tools gen- system for the reach task (top) and the flat- erated by our system for the dough calabush len dough task (bottom).
task (left) and hold phone task (right).
4.4 weal-world Experiments
Hardware setup. To validate the real-world applicability of our method, we 3D printed the tools designed for the hold a phone and dough calabush tasks and tested them using a physical XArm7 robot equipped with parallel gripper. As shown in Fig. (6, the generated tools are physically feasible, structurally stable, and effectively fulfill their intended functions. The robot was able grasp, manipulate, and interact with the tools in real-world settings, demonstrating that our parameterized design and optimization pipeline produces artifacts suitable for real deployment.
Long-horizon Task. In addition, we conducted long-horizon task-makinn a sesame pancake to evaluate the system's performance across multiple, sequential manipulation steps. As shown in Fig (top row), the system autonomously designed and used distinct tools for each subtask, including flat- tening the dough, scooping sauce, spreading it evenly, and sprinkling sesame seeds. This demonstrates the effectiveness of our method in decomposing complex tasks and its ability to generate diverse, task-specific tools and coordinated usage plans that work in combination to achieve multi-step goals.
4.5 Ablation Study
We conduct ablation studies to assess the contribution of each major component of our pipeline.
Specifically, we remove:
the text-to-3D generator, replacing generated components with primitives only;
the trajectory optimizer, using the initial action plan from the Designer without refinement; the tool optimizer, keeping the initial tool geometry fixed during execution.

---

Table 2: Results of ablation study: Dropping any component degrades performance, with optimiza- tion emerging as the most critical contributor.
Method
Best Performance 1
Success Rate ?
Ours w/o Text-to-3D
0.69
33%
Ours w/ Optimizer
0.28
5%
Ours w/o Tool Opt
0.67
32%
Ours
0.94
50%
These ablations help isolate the impact of generative modeling, action refinement, and geometry tuning, and confirm that each component plays a critical role in overall task success and physical feasibility.
As shown in Tab. 2 removing the text-to-3D generator and relying solely on primitive shapes leads to noticeable drop in both best performance and success rate, underscoring the importance of generative modeling in producing task-approe components. This is particularly evident in tasks like fill bottle and hold phone, where simple primitives fail to capture the necessary geometry, while text-to-3D offers the flexibility to generate complex and functional designs. Disabling the trajectory optimizer ppeimizer-therbby skipping action refinement results in the most significant performance drop (from 0.94 and 50% success rate to 0.28 and 5%). demonstrating that relying solely on the VLM agent to generate trajectories is insufficient for accurate task execution. This highlights the critical role of the trajectory optimization phase. Additionally, removing the joint optimization of tool geometry leads to reduced performance. Although the tools may appear visually correct and can pass the Critic agent's s evaluation, certain tasks-such as Fill a Bottle and Lift a high geometric precision for success. In such cases, jointly optimizing tool geometry is essential for achieving reliable performance. Together, these results demonstrate that each component plays a vital role in the robustness and effectiveness of the overall pipeline.
4.6 Failure Analysis
While our method achieves strong performance overall, we observe several failure cases arising from different stages of the pipeline. Design failures often stem from misalignment between the Designer agent and the Meshy text-to-3D generator. The Designer may specify only single functional part, relying on Meshy to generate geometry, but Meshy's ' outputs are not always controllable in terms of size, orientation, or detail. As result, the generated mesh may not match the designer's intent, leading to tools that are difficult to grasp or use effectively. Tool use failures typically occur when the tool's 's orientation is poorly defined. Designers often provide ambiguous axes overly generic rotations (e.g., 90º or 180 flips), which are not sufficient for nuanced interactions like scooping or precise alignment, complicating execution and downstream optimization. Grasping failures arise from the limitations of our grasp API, especially when the tool is too heavy or when the required motion is abrupt, causing the grasp to fail during simulation. Finally, optimization failures occur in tasks involving tools with many shape parameters or long complex trajectories.
5 Conclusion
We present RobotSmith, a unified framework that enables fully autonomous tool design and use for robotic manipulation by combining the semantic reasoning capabilities of large foundation models with physics-based optimization. Extensive experiments across variety of challenging manipulation tasks, both in simulation and the real world, demonstrate the effectiveness, robustness, and practicality of our approach. We believe this work represents a step toward more generalizable, embodied intelligence capable of solving long-horizon tasks through physical tool creation and interaction, and paves the way for future intelligent manufacturing and adaptive robotic assistants.
Limitation. While our framework introduces modular and expressive tool representation that supports both procedural and generative components, the current optimization process only involves scaling, rotation, and translation. This may restrict exploring more complex structural changes that could better adapt tools to specific tasks. Enabling richer forms of tool modification, such as topology editing or part reconfiguration, could further improve their diversity, functionality, and effectiveness.
Acknowledgement This project was supported by MURI N0001 424 NSF IS-2444386 and NVIDIA.

---

References
[1] P. Achlioptas, I. Huang, M. Sung. S. Tulyakov, and L. Guibas. ShapeTTalk: A language dataset and framework for 3d shape edits and deformations. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[2] K. R. Allen, T. Lopez-Guevara, K. Stachenfeld, A. Sanchez-Gonzalez, P. Battaglia, J. Ham- rick, and Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.00723. 2022.
[3] K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 1 117447):93022293311 2020.
[4] G. Authors. Genesis: A universal and generative physics engine for robotics and beyond, 2024.
[5] K. Baumli, S. Baveja, F. Behbahani, H. Chan, G. Comanici, S. Flennerhag, M. Gazeau, K. Hol- sheimer, D. Horgan, M. Laskin, et al. Vision-language models as source of rewards. arXiv preprint 13Xi::2312.09187,, 2023.
[6] BlenderKit Team. BlenderKit: Online Asset Library for Blender. hhtpps//www. htop:://www..eeddekiii blenderkit com/ 2025. Accessed: 2025-05-12.
[7] Brawer, M. Qin, and B. Scassellati. causal approach to tool affordance learning. In 2020 IEEE/RSJ international conference intelligent robots and systems (IROS), pages 4394-8399 IEEE, 2020.
[8] B. Chen, 7. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-144655 2024.
[9] Chen, Y. Mu, Q. Yu, Wei, S. Wu, Z. Yuan, Z. Liang, C. Yang, K. Zhang, W. Shao, et al. Roboscript: Code generation for free-form manipulation tasks across real and simulation. arXiv preprint arXiv:2402. 14623, 2024.
[10] A.-C. Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.0/884 2024.
[11] A. Cherian, R. Corcodel, S. Jain, and D. Romeres. Limphy: Complex physical reasoning using large language models and world models. arXiv preprint arXiv:2411.08021 2024.
[12] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv: 16309, 2023.
[13] Y. Dong, S. Han, Cheng. W. Friedl, R. C. Muchacho, M. A. Roa, J. Tumova, and F. Pokorny. Co-designing tools and control policies for robust manipulation. arXiv preprint arXiv:2409. / 111,, 2024.
[14] I. Exarchos, K. Wang, B. H. Do, F. Stroppa, M. M. Coad, A. M. Okamura, and C. K. Liu. Task-specific design optimization and fabrication for inflated-heam soft robots with growable discrete joints. In 2022 International Conference on Robotics and Automation (ICRA), pages 7145-7151. IEEE, 2022.
[15] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 3902-3):202-216, 2020.
[16] J. Goodall. Tool-using and aimed throwing in community of free-living chimpanzees. Nature, 20144926):26441126, 1964.
[17] E. K. Gordon and R. S. Zarrin. Online augmentation of learned grasp sequence policies for more adaptable and data-efficient in-hand manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5970-5976. IEEE, 2023.
10

---

[18] M. Guo, Z. Liu, S. Tian, Z. Xie, J. Wu, and C. K. Liu. Learning to design 3d printable adaptations on everyday objects for robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 024-830, IEEE, 2024.
[19] H. Ha, S. Agrawal, and Song. Fit2form: 3d generative model for robot gripper form design. In Conference on Robot Learning, pages 176-187. PMLR, 2021.
[20] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-guu guided robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR, 2023.
[21] N. Hansen and Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159-1955 2001.
[22] Hirose, T. Fujisawa, H. Yoshida, H. Kawasaki. D. Hirayama, and Sasaki. Concept of esg and its material properties. In RCC Dams-Roller Compacted Concrete Dams, pages 465-473. Routledge, 2018.
[23] P. Katara, Z. Xian, and K. Fragkiadaki. Gen2sim: Scaling up robot learning in simulation with generative models. In 2024 IEEE International Conference on Roboties and Automation (ICRA). pages 6672-6679. IEEE, 2024.
[24] K. Kawaharazuka, K. Okada, and M. Inaba. Adaptive whole-body robotic tool-use learn- ing on low-rigidity plastic-matt made humanoids using vision and tactile sensors. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 583-589. IEEE, 2024.
[25] L. Ke. J. Wang, T. Bhattacharjee, B. Boots, and S. Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 202/ IEEE international conference robotics and automation (ICRA), pages 6185-6191. IEEE, 2021.
[26] M. Li, R. Antonova, D. Sadigh, and J. Bohg. Learning tool morphology for contact-rich manipulation tasks with differentiable simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1859-18655. IEEE, 2023.
[27] X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18061-18077 June 2024.
[28] Y. Li, T. Kong, L. Li, Y. Li, and Y. Wu. Learning to design and construct bridge without blueprint. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2398-2405. IEBE, 2021.
[29] Liang and Boularias. Learning category-level manipulation tasks from point clouds with dynamic graph cnns. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1807-1813, IEEE, 2023.
[30] C. Lin, J. Fan, Y. Wang, Z. Yang, Z. Chen, L. Fang, T. Wang, Z. Xian, and C. Gan. Ubsoft: A simulation platform for robotic skill learning in unbounded soft environments. In Proceedings of the Conference on Robot Learning (CoRL), PMLR 270, pages 4899-4894. PMLR, November 2024. Also available as arXiv preprint arXiv:2411.12711.
[31] X. Lin, Z. Huang. Y. Li, J. B. Tenenbaum, D. Held, and C. Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint arXiv:2203. 17275, 2022.
[32] L. Ling, C.-H. Lin, T.-Y. Lin, Y. Ding, Zeng, Y. Sheng, Y. Ge, M.-Y. Liu, A. Bera, and Z. Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXiv:2505.02885 028366 2025.
[33] B. Liu, Y. Zhu, C. Gao, Y. Feng. Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing 36:44776 2023.

---

[34] R. Liu, Y. Du, F. Bai, J. Lyu, and X. Li. PEARL: Zero-shot cross-task preference alignment and robust reward learning for robotic manipulation. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st Inter- national Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 09946330664. PMLR, 21-27 Jul 2024.
[35] Z. Liu, S. Tian, M. Guo. C. K. Liu, and J. Wu. Learning design and use tools for robotic manipulation. arXiv preprint (.X.v:23111.0755. 2023.
[36] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representa- tions and rewards for robotic control. In International Conference on Machine Learning, pages 23301-23320. PMLR, 2023.
[37] Meshy AI. Meshy: AI 3D Model Generator. 2025-05-11.
https://www..mett
ai/ 2025. Accessed:
[38] Y. Mu, J. Chen, Q. Zhang, S. Chen, Q. Yu, C. GE, R. Chen, Z. Liang, M. Hu, C. Tao, et al. Robocodex: multimodal code generation for robotic behavior synthesis. In Proceedings of the 41st International Conference on Machine Learning, pages 36434-36454, 2024.
[39] F. Nagata, K. Watanabe, and K. Izumi. Furniture polishing robot using a trajectory generator based cutter location data. In Proceedings 2001 ICRA. IEEE international conference on robotics and automation (cat. No. 01CH37164), volume 1, pages 319-324. IEEE, 2001.
[40] L. Nair. N. Shrivatsav, and S. Chernova, Tool macgyvering: A novel framework for combining tool substitution and construction, arXiv preprint arXiv:2008. 10638, 2020.
[41] Ni, Hao, Wu, L. Kou, Liu, Y. Zheng, B. Wang, and Y. Zhuang. Generate subgoal images before act: Unlocking the chain-of-thought reasoning in diffusion model for robot manipulation with multimodal prompts. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13991-14000 2024.
[42] OpenAl. Gpt-3 o3-mini, 2025. [Large language model].
[43] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7278-7288 IEEE, 2020.
[44] X. Qiu, Y. Wang, Cai, Z. Chen, C. Lin, T.-H. Wang, and C. Gan. Lucibot: Automated robot policy learning from generated videos. arXiv preprint arXiv:2503.09871, 2025.
[45] F. Richter, J. Lu, R. K. Orosco, and M. C. Yip. Robotic tool tracking under partially visible kinematic chain: A unified approach. IEEE Transactions on Robotics, 38(3):1553-1670, 2021.
models [46] J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXii:2310.122911, 2023.
[47] H. Shi, Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. arXiv preprint arXiv:2306.144477 2023.
[48] S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Biyik, D. Sadigh, Finn, and L. Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36:55681
2023.
[49] C. P. Van Schaik and G. R. Pradhan. A model for tool-use traditions in primates: implications for the coevolution of culture and cognition. Journal of Human Evolution, 44(6):645-66) 6666664 2003.
[50] Y. Wan, J. Mao, and J. Tenenbaum. Handmethat: Human-robot communication in physical and social environments. Advances in Neural Information Processing Systems, 35:12014-12026 2022.
[51] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang. H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models, arXiv preprint 11.00:2310.01361, 2023.
12

---

[52] T.-H. J. Wang, J. Zheng, P. Ma, Y. Du, B. Kim, A. Spielberg, J. Tenenbaum, C. Gan, and D. Rus. Diffusebot: Breeding soft robots with physics-augmented generative diffusion models. Advances in Neural Information Processing Systems, 36:44398 44423, 2023.
[53] Y. Wang, X. Qiu, J. Liu, Z. Chen, J. Cai, Y. Wang. T.-H. J. Wang, Z. Xian, and C. Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting. Advances in Neural Information Processing Systems, 37:67575-67603 2024.
[54] Y. Wang, Z. Xian, F. Chen, T.-H. Wang. Y. Wang. K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455 2023.
[55] Y. R. Wang, Duan, D. Fox, and S. Srinivasa. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.07018, 2023.
[56] J. Wen, Y. Zhu, M. Zhu, J. Li, 7. Xu, 7. Che, C. Shen, Y. Peng. D. Liu, F. Feng, et al. Object-centric instruction augmentation for robotic manipulation. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 4318-4325. IEEE, 2024.
[57] Y. Wu, S. Kasewa, O. Groth, S. Salter, L. Sun, O. P. Jones, and I. Posner. Imagine that! leveraging emergent affordances for 3d tool synthesis. arXiv preprint arXiv:1909.1356. 2019.
[58] A. Xie, F. Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects tools with visual foresight. arXiv preprint arXiv: arxiiv//000 1904.0558,.. 2019.
[59] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2reward: Reward shaping with language models for reinforcement learning, In The Twelfth International Conference on Learning Representations.
[60] M. Xu, W. Yu, Huang, Liu, X. Zhang, Y. Niu, T. Zhang, Xia, Tan, and D. Zhao. Creative robot tool use with large language models. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
[61] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, Erez, L. Hasenclever, J. Humplik, al. Language rewards for robotic skill synthesis, arXiv preprint arXiv:2306.0mm 086477 2023.
[62] Y. Zeng. Y. Mu, and L. Shao. Learning reward for robot skills using large language models via self-alignment. In International Conference on Machine Learning, pages 58366-58386. PMLR, 2024.
[63] 7. Zhang, F. Hu, J. Lee, F. Shi, P. Kordjamshidi, J. Chai, and 7. Ma. Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities. arXiv preprint arXiv:2410. 17385, 2024.
13

---

A Tool Design
A.1 An example of tool representation
name": "Funnel Tool with Handle",
parts": [
"geom"; "mesh",
"prompt": "metal funnel for liquid pouring", "parameters": [0.3, 0.3, 0. 15] "is_graspable", false },
"geon": "cylinder", "prompt": "" "parameters"; [0.02, 0.15] "is_graspable"; true
1, "aaseebly": "def assembly (parts: list[dict]] *placement"; "def placement
Place the tool*
Each part in the tool representation specifies a geometric component using the "geom" field, which supports six options: "mesh", "cylinder", "cube", "ball", "ring", and "tube". The "is "is_ .tra_paaaalle field indicates whether the robot gripper can grasp the part. To ensure a stable and fixed grasping pose, only one part should be marked graspable. The interpretation of the "parameters" field depends on the selected geometry type:
"mesh": This part is a mesh generated by a text-to-3D model using a prompt. The parameters [1x, ly, 1z] represent the dimensions of the bounding box after scaling, corresponding to the lengths along the X, Y, and Z axes.
"cube": The part is cube. The parameters [ax, sy, sz.] specify the scale along the X,
Y, and axes.
"ball": The part is ball. The parameter single value [radius], representing the radius of the sphere.
"cyling :: The part is a cylinder. The parameters [radius, height] define the radius of the circular base and the height along the Z-axis.
"ring": The part is ring. The parameters [radius, thickness] describe flat torus, where radius the major radius and thickness is the cross-sectional thickness.
"tube": The part a tube. The parameters [radius, length, thickness] define
hollow cylindrical tube with given outer radius, length, and wall thickness.
The assembly function takes the list of parts input and uses several editing APIs to merge them into a complete tool. An example assembly function is shown below:
def
assembly (parts: Import necessary modules from trimesh import trimesh
list [dict]):
Part Create funnel_part parts [0]
funnel using text-to-3D generation
# Cenerate a funnel mesh from the prompt
funnel generate_3d 3d (funnel_part (tpppparrtppppppppp" "]p
Align the funnel so that its bounding is centered axis aligned
funnel rotate_to aolaggnn (funnel)
14

---

# Get the current bounding box dimensions
bbox [tt_axis_alignee bounddind_buudingg. (funnel)
dims [bbox [1] bbox [0], bbox [3] bbox [2] bbox [5] bbox [4]]
# We want the funnel's x-dimension (largest by our convention) to match the provided parameter target_dimg funnel_part ["parametera"]
Compute a uniform scale factor based on x-dimension
acale_ratio target_dims [o] / dima [0] if dims != else 1.0 funnel rescale (funnel, scale_ratio)
Part 2: Create the handle using a primitive cylinder handle_part parts[1] handle Drimitive""yylndee handle rotate_to, _ot_taiaaaaaaaaaa (handle)
handle_part ("parameters"))
Position the handle so that it attaches to the side of the funnel
# We assume the funnel is centered at the origin, so we shift the handle left along
handle move (handle, (-dims [0), 0, 0))
Optionally, we could adjust the vertical offset of the handle if needed, e.g., move slightly down handle move (handle, (0, [dims(2]))
# Combine the two meshes into one tool mesh tool_mesh = trimesh util. concatenate ( ([funnel, handle])
Export the assembled tool an file tool_mesh export (ypport(ttoo..ooo tool.
# Return the list of exported asset filenames return ('tool.obj")
A.2 An example of tool designer
Edits the shape parameters of the bottom cube to more closely resemble T-key
Proposer
A cube with a thin rod
Proposer
Instead a plain cube/rod, use an insert-and-twin key sized for the coin slot...
SD
Suggests making top cube (the graspable part) thinner to facilitate holding.
Critic
Critic
Adds bottom to the initial design, attempting create key-like structure
Edits the shape parameters of the top cube make thinner.
Proposer
Proposer
Swap the bottom cube for a T-key cam head that is narrow for insertion and wider when rotated ~900 to catch the inner rim.
Done. more modification suggestions.
Critic
Critic
Figure A 4-round conversation between Proposer and Critic for the Piggy task.
15

---

B APIs and Prompt Templates
B.1 Proposer
B.1.1 Proposer Prompt Template
Now you have a robotic task:
On the table, there is robot arm with parallel gripper. The task goal is DTASK_DESCRIPTOONES For detailed aetup, here a code:
$3D_ CDDCCOFFIGURTIONSS
You need design a help finish the task.
You have 2 APIs to generate 3D meshes:
[DESCRIPTIONS OF GENERATION APIS]
You have several APIs for 3D editing:
[DESCRIPTIONS OF MODIFICATION APIS]
To describe generated tool, you will use a jeon format. Here is an example: "jsoon
"name"; string,
"parts":
geom": in l'cube', 'ball', 'cylinder', 'ring', 'tube 'mesh']
"prompt": "geom" " 'mesh', there string representing text prompt for text-to-30 generation model; otherwise is empty string, "parameters"; if "geom" is 'mesh the parameters (1x, ly, 1z), the length in x/y/z directions; if not, the parameters should be the same as primitiv__scale in primitive
"is.graspable": True/False, if it is True, then robot can grasp this part, otherwise the robot arm cannot grasp.
}
{
}, asssemby"; complete python code to assemble all the parts above into the whole tool using 3D editing APIs provided.
mun
def assembly (parts: list(dict)):
the input paraneter parts here is list of dictionaries, you need to call primitive() () generate_3d manually with parameters in parts in the function, the above 3D editing APIs are used to assemble each part into the whole tool.
the function should export 3D assets of the tool obj files, each asset ahould be on the x-y plane. the function returns list of file names of the exported 3D assets.
into
16

---

"placement"; complete python code to place the tool in the scene
def placement() ()
self tool scene. add_ gs Morph Mesh
filename="tool "bo""
00,.00000000000000
euler= scale=(1,1,1),,
the filename here should match that saved in assemble_func, and the position, rotation and scale should be approprate to be here.
You should make sure:
The tool saved. All the editing functions in assembly fune return values, which should be received. SEREE__ROVIDED_PRRICIIE PRINCIPLESS
B.1.2 Generation APIs
The following 2 APIs are available for the Proposer to create 3D) geometry:
def primitive (primitive_name, primitive_scale)
Create 3D primitive mesh centered at origin.
Arga:
primitive_nan (str): One of { l'uube', 'ball', 'cylinder',
ring' 'tube') '}.
primitive_scale (list of float):
cube, [sx, sy, sz]
If ball, [radius] If cylinder, [radius, height] If ring, [radius, thickness] If tube, [radius, length, thickness]
Returns:
trimesh, Trimesh: The resulting mesh, centered at (0, 0, 0).
def generate_3dnnm (name)
Call a pretrained text-to-a model to generate single solid 3D
object.
Args:
name (str); the text prompt to generate 3D shape. Should be very simple, like several words.
Returns:
trimesh. Trimesh: The resulting mesh, randon position and of randon size.
Tips:
Remember to rescale and align the generated mesh, since the size and position would be random.
B.1.3 3D Editing APIs
The following 11 APIs are available for the Proposer to modify 3D geometry:
17

---

def cotate_to_aligee (mesh1)
Rotate and translate the mesh so that:
Its bounding box is centered at the origin (0,0,0) The largest dimension of the bounding box aligns with +X, The second-largest aligns with +Y,
The smallest aligns with +Z.
The meah remains axis-aligned and returned in-place.
Arga:
mesh (trimesh. Trimesh); The input mesh (modified in in-place).
Returns:
trimesh, Trimesh: The traasaddtranslated mesh, for convenience.
def get_position (mesh)
Return the position of the mesh as (x, 8, z).
Args:
mesh (trimesh.. Trimesh): The mesh
Returns:
*.* tuple: (x, y, 2)
def get_axis_aligee boundigg.o box (mesh):
Return the axig-eligned bounding box of the mesh as (min_x, min_y, min_z, max_x, max_y, max_z).
Args:
mesh (trimesh. Trimesh): The mesh
Returns:
tuple: (min_x, min_y, min_z, max_x, max_y, max_z)
def got_volume (mesh):
***
Return the volume of the mesh in cubic units.
Arga:
mesh (trimesh. Trimesh): A closed, manifold mesh
Returns:
float: The volume the mesh
def rescale (mesh, ratio):
Uniformly scale the mesh by the given ratio about the origin (0, 0, 0).
Args:
mesh (trimesh. Trimesh): The mesh scale (modified in in-place).
ratio (float): The scale factor to apply.
Returns:
trimesh Trimesh: The scaled mesh (for convenience),
def move (mesh, offset):
Translate the mesh by the given offset (x, y, z).
Args:
mesh (trimesh, Trimesh): : The mesh to move in-place.
offset (tuple or list of float): The translation offset (dx, dy, dz).
Returns:
trimesh Trimesh: The transformed mesh (for convenience).
def empty_grid(). ():
18

---

Create empty 256x256x256 boolean occupancy grid from -0.5 to +0.5 in each axis.
Returns:
dict: A dictionary containing:
'data'; nd.ndarray of shape (256, 256, 256), dtype-bool (all False initially).
'res': integer (256).
'minn ar-oy -0.5, -0.5))
"mmax_boudd" arrarra ([0.5, 0.5, 0.6]).
def add_mesh ( mesh)
Convert 'mesh' into volume of occupied voxels using an SDF signed-distance) test, then mark those voxels as True in 'grid'.
Args:
grid (dict): The grid dictionary from empty-grid().
mesh Trimmes))Trmesh) A triangular mesh (assumed to fit in [-0.5, 0.5]-3).
Returns:
dict: The updated grid, same reference as input.
def sub_mesh(grid (grid, mesh);
Convert 'mesh' into a volume using an SDF, then set those voxels to False
(subtract from the grid).
eaa
def cut-grid(grid))
Return tuo new grids of the same resolution (256x256x266) grid_bottom: occupies only z <
grid_up: occupies only >=
zeroing out the complementary region in each grid. Each returned grid has:
256,,266) 'data': a (256, (256,256,70 256) boolean array
same 'min_bound' and 'max_bound' as the original same 'res" as the original Args:
grid (dict): Must have keys 'data', 'res', 'ound' max_bound',
'data' is 3D boolean array: (256, 256 256,256)
Returns:
(dict, dict): (grid_up, grid_bottom)
def to (grid, do To_simplify-True,
Convert 3D occupancy grid into surface mesh using Marching Cubes.
Optionally simplify the mesh using Open3D's quadric decimation.
Args:
grid (dict); dictionary with keys:
'data': (256, 256 256) boolean array (True occupied).
'res": : int, resolution (e.g. 256).
'min_bound'; np. ap.array((x_mi,
y_min, 2_min]).
'oundd np. array ([x_max, y_max, z_max]).
do simplify (bool): Whether to perform mesh simplification default True).
target_nume festt_num_faoe faces (int): If simplifying, the target number of faces.
19

---

Returns:
trimesh, Trimesh: The extracted (and optionally simplified) mesh. If the grid is empty or no surface is found, faces might be empty.
B.2 Critic Prompt Template
Here the prompt template for the Critic agent:
Now you have a robotic task:
On the table, there is robot arm with parallel gripper. The task goal is to DESCRRIECCIP For more detailed scene setup, here is code:
$3D ADD_CONFIGURATIONS
Here are the aulti-vies figures of tool designed for the task. Does that look good? If so, reply DONE; not, give modification suggestions.
Here are some important aspects for you to examine if the tool looks good:
1. good tool should easy to grasp.
2. A good tool should be in good shape, and each part looks connected. PROVIDED_PRINNIPLESSS [RENDERED _FIGURES]]]]
$USER_ PROVEDEDR PRINCIPLESS
B.3 Tool User Prompt Template
Now you have robotic task:
On the table, there is a robot arm with a parallel gripper. The task goal is to DTASK_DESCRIPTIONS You have a tool designed for the task. For more detailed scene setup, here is the scene setup code and rendered image:
$3D CDD_CONFIGGRATIONS
Now you need to provide robotic manipulation trajectory using the following 3 APIs:
python
move (pos, euler) # move the gripper to the target position and orientation
grasp (obj, euler) auto calculate grasping pose, aligning given gripper orientation, to steadily grasp obj
release() open the gripper to the widest
Please write a manipulate() function with the APIs provided above. Note you can only move tools or rigid objects in the scene.
C Task Details
C.1 Dough Calabash
Task Description T: "make the dough into calabash liked shape"
20

---

Initial Scene Configuration Sy: dough pos) # cubic dough
Metric min(1,
CLIP CLIP__@OOE image, "calabash-shaped dough"))
C.2 Reach
Task Deseription 7:"reaching the cube outside workspace"
Initial Scene Configuration Sn; cube=Box pos0) cube out of robot arm 's reachability
Metric M: max (cube, pos- (oss-target_pos) norm () (posO-target norm
C.3 Flatten Dough
Task Description T:"flatten dough a height smaller than 0.03"
Initial Scene Configuration Sj: douie=Box(sizee0, pos0) a cubic dough (eweeightt..00)(eeiieeeeee
Metric M: 1 max
C.4 Hold a Phone
Task Description T:ekeep the phone face upfront rather than lying on the table"
Initial Scene Configuration Sy: phone=Mesh aephone.obj. pos, euler)) a phone Metric M: 1 (90-phone.
1S0-phone.euler euler [0])/90)
C.5 Fill Bottle
Task Description T:upour the cup of water into the bottle"
Initial Scene Configuration Sp:
bottle=Mesh( ( 'bottle.ob) obj'', , pos, euler) a bottle
cup=Mesh cwcuplccuuupooooo cup''bj', pos, euler) # cup
water=Cylinder (radius, height, pos)) a cup of water
Metric M: 1 max(0, (dough. Briggh..eighttO.comm 03)) (heightO-0.03))
C.6 Lift a Bowl
Task Deseription T:flift up the large bowl without touching inner surface"
Initial Scene Configuration Sy: bowl=Meshs ('bowl.obj', pos, euler) # large bowl Metric M: 1 m0, (0. (0.1-bowl. pos (0.)
C.7 Cut Dough
Task Description c:"cut the dough into two even parts"
Initial Scene Configuration Sp: dough= pos0) cubic dough
Metric M:
dough1, dough2 (Means fit (dough)
1 max (0.2- posugh1.pss-ouugee poss) norm /0.
C.8 Transport Water
Task Description T: "transport as more as possible water into the cup from tank"
Initial Scene Configuration Sp:
Sp: bow1=Mesh ( ('tank. obj', , pos0, euler0) # water tank
21

---

Critic Tool Designer
Task Description
Tool
Proposer
Critic
Tool Use
Joint Optimizer
Optimized Tool
Planner
Tool
CAM-ES
Tool User
Optimized Trajectory
Trajectory
Figure 8: A brief pipeline figure to explain the workflow.
Initial Scene Configuration Sy: cup=Mesh (ocieashhccu.ob)) oup' pos1, euler1) # an empty cup
water=Box (size0, pos0) # tank of water
Metric M: water inber_in_cupp cup. volumn / cup. volumn
C.9
Lift Piggy
Task Description T:"lift up the piggy bank"
Initial Scene Configuration Sp:
piggy=Mesh ('piggy.oom obj', pos0, euler0) a piggy bank with insertion hole on top of
Metric M: 1 max (0, (0 (0.3-piggy height)//.3)
D Societal Impact
Positive Impact. Our work contributes toward more capable and autonomous robotic systems by enabling them design and use tools tailored specific tasks. This has the potential greatly benefit sectors such as elder care, disaster response, and manufacturing, where adaptive, general-purpont robots could reduce human labor hazardous or repetitive environments. By making tool-use more accessible and efficient, our framework could also accelerate innovation in robotics education and research, fostering creativity and reducing the need for manual engineering. Moreover, the open- sourcing of our APIs and representations promotes transparency and reproducibility, encouraging responsible development within the research community.
Negative Impacts. As with many advances in automation and roboties, our system could contribute labor displacement in industries reliant manual semi-skilled tasks, particularly in basic manufacturing. Additionally, the misuse of autonomous tool design- for example, in adversarial contexts- raises concerns. Our system's 's reliance on large foundation models also inherits potential biases or inaccuracies present in those models, Careful deployment, human oversight, and continuous auditing are necessary mitigate these risks and ensure that such systems used for socially beneficial purposes.
E
Compute Resources
Our experiments involve two main components: querying large pretrained models and running optimization in simulation. We use API calls to models such as O3-Mini and Meshy tool design and initial tool use planning. These API-based components do not place any demand on local compute resources. For simulation and optimization, we run CMA-ES within the Genesis simulator, which supports both CPU-only and GPU-accelerated platforms. Our tasks do not require large GPU memory,
22

---

and many experiments were successfully conducted on single NVIDIA 2080 Ti GPU, and others were conducted on NVIDIA GeForce RTX 4090 GPUs. While more powerful GPUs can improve simulation efficiency, they are not essential for reproducing our results.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed description the pipeline in the Method section and the appendix, and we will release the code upon acceptance.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We clearly described the method and provided all necessary prompts. We plan release the data and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliiiggiddde/ooooooo /ooeeee/e/ooeubbbiiieoo ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We included these details, such optimization parameters and prompt tem- plates in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report error bars of different trials in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We conduct all experiments on devices with single GPU because of high efficiency, Experiments on CPU only devices are also feasible.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p///.aaurrss.ccc//p//llllla
Answer: [Yes]
The research conducted the paper conformed with the NeurlPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both positive societal impacts and negative societal impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in the paper are properly credited. We have ensured that their licenses and terms of use explicitly acknowledged and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We clearly described the usage of LLMs in method and provide all prompts needed in appendix.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
30


Reviews:
[{"review_id": "2lAfU3Bf9b", "reviewer": "Reviewer_rFNK", "summary": "The authors propose a pipeline for automatically generating suitable tools for robot manipulators in embodied agent tasks. The approach leverages LLM agents to collaboratively design initial tools using Constructive Solid Geometry (CSG) representations and generates initial action trajectories by minimizing trajectory length. To enable real-world applicability, the pipeline employs Evolution Strategies to jointly optimize both shape parameters and action trajectories. The effectiveness of the method is validated through both simulation and real-world experiments.", "questions": "In addition to the issues mentioned above, the reviewer have the following questions:\n- Is it possible to generate concave-shaped tools? Figure 5 only shows convex examples.\n- Why are only the move API parameters jointly optimized (line 203)? What about the grasp API parameters?\n- Why is the \"Dough Calabash\" cut success rate for \"retrieval\" only 21%, which is even lower than using no tool?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality\n\nThe paper addresses a highly relevant and innovative problem in embodied agents, representing a significant step toward generalizable embodied intelligence. The effectiveness of the proposed method is demonstrated through several simulation and real-world experiments. However, the number of experiments is limited, and the tasks are relatively simple, which restricts the comprehensiveness of the evaluation. Although the method is claimed to benefit long-horizon tasks, the evaluation is still conducted on individual tasks. A major concern is the diversity of tool generation; only a few qualitative empirical visualizations are provided, making it difficult to assess the method’s full potential and limitations.\n\nClarity\n\nThe paper clearly presents the challenges, motivation, and design, making it easy to follow and highly readable. The contributions are well articulated. Figure 2 effectively illustrates the pipeline, and the experimental results are presented clearly, demonstrating the method’s effectiveness both qualitatively and quantitatively.\n  \nSignificance\n\nThe inclusion of real-world experiments is a notable strength that enhances the significance of the work. However, some important issues remain unaddressed. The diversity of tool generation using the CSG representation is only partially demonstrated (as shown in Figure 5), and the performance boundaries of this representation are unclear. Additionally, CSG is a crucial step in tool generation and should be compared with other representations. The configuration of the robotic arm is also not discussed, suggesting that the method may be specific to a particular robot and may not generalize to other robotic arms.\n\nOriginality\n\nThe paper addresses an important and timely problem in embodied AI, which is also a key challenge for imitation learning in long-horizon tasks. The proposed solution is interesting, involving the interaction of two agents for tool design and the use of evolutionary algorithms to iteratively optimize both the tool design and action trajectories.", "was_revised": true, "final_justification": "Thanks to the additional information in the rebuttal stage, the reviewer raises the score of significance.", "rebuttal": "*We thank the reviewer for the positive and constructive comments. We address your concerns in detail below.*\n\n**W1. \"However, the number of experiments is limited, and the tasks are relatively simple, ...\"**\n\nThere's currently no large scale benchmark settings for tool generation and tool use tasks. Compared to previous work [1, 2, 3], our task settings are more diverse in quantity and semantic complexity, and we cover various deformable and fluid materials rather than only rigid objects.\n\n[1] Guo, Michelle, et al. \"Learning to design 3d printable adaptations on everyday objects for robot manipulation.\" 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.\n\n[2] Li, Mengxi, et al. \"Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[3] Liu, Ziang, et al. \"Learning to Design and Use Tools for Robotic Manipulation.\" 7th Annual Conference on Robot Learning.\n\n**W2. \"... the evaluation is still conducted on individual tasks.\"**\n\nOur approach aligns with common practice in robotics, where complex, long-horizon problems are often decomposed into manageable sub-tasks, as demonstrated by prior work like [4, 5]. \n\nOur primary focus lies in validating our method on these fundamental individual tasks, rather than task decomposition or high-level planning. To further illustrate our method's capacity for complex, integrated scenarios, we've included a complete long-horizon task demonstration in Figure 1.\n\n[4] Wang, Yufei, et al. \"RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation.\" International Conference on Machine Learning. PMLR, 2024.\n\n[5] Huang, Zhiao, et al. \"Diffvl: Scaling up soft body manipulation using vision-language driven differentiable physics.\" Advances in Neural Information Processing Systems 36 (2023): 29875-29900.\n\n\n**W3. \"A major concern is the diversity of tool generation, ...\" \"The diversity of tool generation using the CSG representation is only partially demonstrated ...\"**\n\nAs shown in Figure 5, we've demonstrated the diversity of generated tools for two distinct tasks. To further address this concern, we can include additional examples of diverse tools generated for other tasks in the appendix of the final version.\n\nCompared to prior work like [3], we've significantly extended tool diversity in two key ways: by introducing a text-to-3D generation module for increased component complexity, and by designing a set of tool design APIs for enhanced connection diversity that allow 3D parts to be assembled into a tool.\n\nAs for Constructive Solid Geometry (CSG), our tool representation is indeed inspired by CSG principles. However, we employ an implicit approach using 3D editing APIs for construction rather than explicit CSG operations. While CSG is a powerful method and widely used in CAD, its primary limitation is that it relies on a pre-defined set of primitives. We address this by complementing our approach with off-the-shelf 3D generation, bringing in complex, semantically meaningful meshes to fill in where CSG might fall short. We believe this mix gives us a solid balance between structured representation and generative flexibility.\n\n[3] Liu, Ziang, et al. \"Learning to Design and Use Tools for Robotic Manipulation.\" 7th Annual Conference on Robot Learning.\n\n**W4. \"The configuration of the robotic arm is also not discussed, ...\"**\n\nThe tool design is task specific, and the robot arm configuration is part of the task settings. For instance, a \"reaching\" task becomes trivial with a longer arm, just as a \"lifting bowl\" task might not require a tool if a dexterous hand is available. Currently, we mostly focus on robot arms with parallel grippers and leave the dexterous hands for future works. \n\nIt's important to note that our designed tool adapts to the robot arm's configuration during the joint optimization process. For example, if you were to use a shorter arm for the reaching task, our method would optimize for a longer tool to compensate, effectively adapting to the specific arm in use.\n\n**Questions**\n\n\n> **1. Is it possible to generate concave-shaped tools? Figure 5 only shows convex examples.**\n\nYes, our method is capable of generating concave tools. We can achieve this either by directly generating such shapes through the text-to-3D model (Meshy), as seen with the tool for \"Hold A Phone\" in Figure 3, or by explicitly creating them using a set minus operation within our tool design APIs. For instance, the \"Dough Calabash\" tool in Figure 3 was formed by subtracting a calabash shape from a cube, clearly demonstrating a concave design.\n\n\n> **2. Why are only the move API parameters jointly optimized (line 203)? What about the grasp API parameters?**\n\nThe *Release* API has no parameters to optimize. For the *Grasp* API, we exclude its parameters from joint optimization to reduce computational burden, as stable grasping can be immediately evaluated.\n\nFor the *euler* parameter in *Grasp* API, we specifically leverage the VLM's prior knowledge to guide grasping orientation, preventing awkward poses that could cause arm twisting, or self-collision during function. \n\nGiven that there's often little practical difference between similar grasp orientations (e.g., 90° vs. 89° or even 90° vs. 70°), our *Grasp* implementation samples and evaluates numerous poses within a range of the VLM's ideal. So the VLM's exact orientation is not strictly guaranteed. Optimizing this *euler* parameter could confuse the optimizer, leading to less stable results.\n\n> **3. Why is the \"Dough Calabash\" cut success rate for \"retrieval\" only 21%, which is even lower than using no tool?**\n\nTo clarify, the reported success rates for both \"retrieval\" and \"no tool\" are actually 0.0%. The values 0.56 and 0.21 represent the best score during 8 trials. The score here is the normalized CLIP scores between rendered image of the dough and the textual prompt \"calabash\". A score over 0.8 is considered a success for this task, and neither method reached that threshold.\n\nFor the \"no tool\" scenario, the robot's gripper itself acts as a tool, allowing for various manipulations like pressing and pinching. In the \"retrieval\" case, the system attempted to retrieve either a hammer or a knife. These tools proved even less flexible than the gripper for shaping the dough, leading to lower performance. Ultimately, creating a true calabash shape remains a significant challenge for both approaches.\n\n*We wish that our response has addressed your concerns, and convinced you of the merits of our work. If you have any more questions, please feel free to let us know during the rebuttal window. Thank you very much!*\n\n*We appreciate your suggestions and comments! Thank you!* \n\nBest,\n\nAuthors"}, {"review_id": "ZFVBH6HFSw", "reviewer": "Reviewer_d7mA", "summary": "The authors propose an automated pipeline that leverages vision-language models (VLMs) and physics simulation to design and use tools for robotic manipulation. This pipeline iteratively generates tool designs and corresponding robot actions through three main modules: (1) a Critic Tool Designer, where two VLM agents propose and evaluate tool designs; (2) a Tool Use Planner, which generates action plans using robot APIs; and (3) a Joint Optimizer, which refines both tool geometry and action trajectories.\n\nTools are constructed from a set of modular parts, represented using a constructive solid geometry (CSG)-like format. This representation improves the language model’s ability to interpret and reason about 3D geometries.\n\nThe authors conduct extensive experiments demonstrating that their method outperforms state-of-the-art approaches in tool design and use for robotic manipulation tasks.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The authors propose a novel method for robotic tool design leveraging vision-language models (VLMs). \n- They validate their approach through extensive experiments across multiple benchmarks.\n\nWeaknesses:\n- The Critic Tool Designer module consists of a Proposer and a Critic agent that iteratively generate and refine tool designs. However, the capabilities of these agents and their reasoning processes remain unclear. For example, the comment “the tool is long enough to avoid undesired contact” lacks supporting rationale. Please provide more detailed examples illustrating how VLMs or LLMs arrive at such suggestions. Additionally, presenting more diverse cases with corresponding reasoning chains behind specific tool modifications would better demonstrate the agents’ understanding and design capabilities.\n\n- In the Tool Use Planner, the system infers grasp orientation and plans movements based on visual input, with the VLM analyzing images to guide actions. However, it is not clear how reliably the VLM can determine grasp orientations, especially across varying object geometries. More comprehensive quantitative and qualitative results are needed to evaluate the VLM’s performance on different object types and shapes, and to justify its effectiveness in planning grasp and motion strategies.", "was_revised": true, "final_justification": "Thank you for the rebuttal information provided. I will maintain my original score.", "rebuttal": "*We appreciate the positive and insightful comments from you! We adress your concerns in details below.*\n\n**Weaknesses & Questions**\n\n**1. Detailed examples of Agents**\n\nTo address the specific example, the comment about \"the tool being long enough to avoid undesired contact\" was made in the context of the \"Transport Water\" task. The rationale behind this design is to ensure the robot's gripper, which holds the tool, does not have to directly enter the water tank. In general, a handle of sufficient length is necessary to maintain a safe distance from the water and other materials, such as food, preventing contamination and protecting the robot's end-effector.\n\nWe appreciate your suggestion to provide more detailed examples for a clearer understanding of our method's capabilities. We will add more detailed examples of chat history between Proposer and Critic to the appendix of the final paper, their reasoning chains. \n\nWe appreciate your suggestion to provide more detailed examples and reasoning chains. We will add more demonstrations, including chat histories between the Proposer and Critic, and their corresponding reasoning chains, to the appendix of the final paper. For immediate illustration, here is a summary of a tool design case for the \"Lift A Piggy\" task, where the main challenge is that the piggy is large, round, and not directly graspable by the robotic gripper:\n\n(Due to rebuttal rules, images cannot be included, so we provide textual summaries of the shapes and suggestions.)\n\nLift A Piggy Task: 4-Round Tool Design Modification Example\n\n1. Proposer's Initial Design: A cube with a thin rod, looks like:\n```\n     _________\n    /       /|\n   /_______/ |\n  |       |  |\n  |       | /\n  |_______|/\n       ||\n       ||\n       ||\n       ||  \n```\n2. Critic's Suggestion (Round 1): Suggests redesigning the piggy bank interface. Specifically: \"instead of a plain cube/rod, use an insert-and-twist key sized for the coin slot\". (Note: The Critic may offer multiple suggestions, but the Proposer typically acts on the first one or two.)\n3. Proposer's Modification (Round 2): Adds a bottom cube to the initial design, attempting to create a key-like structure:\n```\n     _________\n    /       /|\n   /_______/ |\n  |       |  |\n  |       | /\n  |_______|/\n       || \n       ||\n      _||__\n     / || /|\n    /___ / /\n    |____|/\n```\n4. Critic's Suggestion (Round 2 Continued): Suggests modifying the  newly added bottom cube: \"Swap the bottom cube for a T-key/cam head that is narrow for insertion and wider when rotated ~90° to catch the inner rim.\"\n5. Proposer's Modification (Round 3): Edits the shape parameters of the bottom cube to more closely resemble the T-key.\n```\n     _________\n    /       /|\n   /_______/ |\n  |       |  |\n  |       | /\n  |_______|/\n       || \n       ||\n       ||_\n      /||/|\n     /_ //\n     |_|/\n```\n6. Critic's Suggestion (Round 3 Continued): Suggests aking the top cube (the graspable part) thinner to facilitate holding.\n7. Proposer's Modification (Round 4): Edits the shape parameters of the top cube to make it thinner.\n```\n    _____________\n   /           /|\n  /___________/ |\n |           | /\n |___________|/\n       || \n       ||\n       ||_\n      /||/|\n     /_ //\n     |_|/\n```\n8. Critic's Final Assessment: Done. No more modification suggestions.\n\nAdditionally, we would like to direct you to the appendix B for the prompt templates of our agents, which offer further insight into how we guide their reasoning. We appreciate this feedback and will ensure the final version of the paper provides a more comprehensive view of our agents' design capabilities.\n\nThe following is raw generated data from the Proposer and Critic to illustrate this process: \n```\nProposer (round 1)\n{\n    \"name\": \"PiggyBank Lifter\",\n    \"parts\": [\n        {\n            \"geom\": \"cube\",\n            \"prompt\": \"\",\n            \"parameters\": [\n                0.04,\n                0.04,\n                0.04\n            ],\n            \"is_graspable\": true\n        },\n        {\n            \"geom\": \"cylinder\",\n            \"prompt\": \"\",\n            \"parameters\": [\n                0.0025,\n                0.05\n            ],\n            \"is_graspable\": false\n        }\n    ],\n    ......\n}\n\nCritic (Round 1)\nMODIFY\nSuggestions:\n1. Redesign the piggy-bank interface at the opposite end: instead of a plain cube/rod, use an insert-and-twist key sized for the coin slot (tapered lead-in, soft pads, depth-stop collar, and a 90° cam or T-head that expands under the rim) so the spherical, smooth bank can be lifted without external grasping.\n2. Add a dedicated gripper block at the top: a compact rectangular prism with two broad parallel faces (plus thin rubber pads or light knurling) so a parallel gripper can pinch it securely without slipping or rolling.\n3. Strengthen the rod and joints: use a thicker shaft (≈8–12 mm), make the head and shaft one piece or mechanically keyed/pinned (avoid glue), and add fillets/gussets at both transitions to prevent bending or loosening under twist-and-lift loads.\n4. Ensure alignment and balance: keep the shaft collinear with the insert head’s centroid so the load hangs vertically, place the gripper block centered above the shaft, and add simple orientation marks (“insert” vs. “locked”) to guide the twist.\n5. Provide torque features & safety: give the gripper a small transverse bar or shallow finger groove on the block for better twist leverage, include a detent/stop to hold the locked position, and round all external edges to avoid scratching the bank.\n\nProposer (round 2)\n{\n    \"name\": \"PiggyBank Secure Lifter\",\n    \"parts\": [\n        ......,\n        {\n            \"geom\": \"cube\",\n            \"prompt\": \"\",\n            \"parameters\": [\n                0.02,\n                0.02,\n                0.005\n            ],\n            \"is_graspable\": false\n        }\n    ],\n    ......\n}\n\n\nCritic (Round 2)\nMODIFY\n1. Insert-and-twist head: Swap the bottom cube for a T-key/cam head that is narrow for insertion and wider when rotated ~90° to catch the inner rim; include a tapered lead-in, soft sleeves, a depth-stop collar, and a positive rotation stop/detent so it stays locked during lift.\n2. Shaft & joint reinforcement: Thicken the shaft (≈8–12 mm), use a keyed/D-flat connection to both the plate and the T-head (avoid plain round press-fits), and add fillets or a gusseted boss where the shaft meets the plate to prevent bending under twist-and-lift loads.\n3. ......\n\nProposer (Round 3)\n{\n    \"name\": \"Secure PiggyBank Lifter\",\n    \"parts\": [\n        ......,\n        {\n            \"geom\": \"cube\",\n            \"prompt\": \"\",\n            \"parameters\": [\n                0.02,\n                0.005,\n                0.005\n            ],\n            \"is_graspable\": false\n        }\n    ],\n    ......\n}\n\nCritic (Round 3)\nMODIFY:\n1. Gripper interface upgrade: Replace the small top block with a broad, flat plate that fits your gripper’s maximum aperture; keep two clean, parallel edges for pinching, add thin rubber pads for friction, chamfer the rim to avoid snagging, and use weight-relief cutouts so the plate is light yet stiff.\n2. Improve the insert-and-twist head: Replace the tiny T-tab with a cam/T-key that is narrow for insertion and wider when rotated ~90° to catch the inner lip; include a depth-stop collar, soft sleeves for friction, rounded edges to avoid scratching, and a positive detent/stop so it stays locked under load.\n3......\n\nThe Proposer (Round 4)\n{\n    \"name\": \"Optimized Piggy Bank Lifter\",\n    \"parts\": [\n        {\n            \"geom\": \"cube\",\n            \"prompt\": \"\",\n            \"parameters\": [\n                0.12,\n                0.08,\n                0.02\n            ],\n            \"is_graspable\": true\n        },\n        ......,\n    ],\n    ......\n}\n\nCritic (Round 4)\nDONE\n```\n\n**2. Reliability of VLM Planner**\n\nOur tool-use planner is designed to efficiently solve simple tasks and provide strong initialization for complex ones. While grasp orientation is guided by a visual-language model (VLM) prior, the grasping pose itself is determined via sampling. We select the successful grasp with orientation closest to the VLM's suggestion. This approach ensures adaptability to varying geometries while leveraging the VLM's prior to accelerate sampling, without requiring perfect accuracy.\n\nIn our experiments, removing the planner made it nearly impossible to discover successful trajectories within the time limit for most tasks.\n\nTo further evaluate our grasping component, we tested the planner's ability to generate grasp orientations for 200 objects from a dataset. We measured the angular distance between the VLM-proposed orientation and the nearest feasible grasp found by our system. The average angular distance between our found grasp pose and the VLM's suggestion was 0.33. This compares favorably to an average distance of 0.87 when a working grasp pose was found via random sampling. These results confirm that the VLM provides useful priors, significantly improving our grasping pipeline's sampling efficiency.\n\n*We wish that our response has addressed your concerns. If you have any more questions, please feel free to let us know during the rebuttal window. Thank you very much! We appreciate your suggestions and comments! Thank you!* \n\nBest\n\nAuthors"}, {"review_id": "kBwFfDTOqY", "reviewer": "Reviewer_Kk5s", "summary": "The paper proposes a tool augmented VLLM framework, that designs tools for robotic agents as well as the trajectory for their use. Several components based on GPT3omini are used with a CMA-ES at the end to address LLM geometrical reasoning limitations.", "questions": "The main issues identified are with the clarity of the method and result interpretation.\n- What components are trainable and what components are shared?\n- What is the optimisation process?\n- Clarifincation on evaluation is needed", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well written, explains the problem well and provides a solution. \nHowever, there are several severe issues related to the clarity of the method.\n\n- It is unclear what components are trainable if any are at all and how they are trained. This needs to be made explicit.\n- Do the Proposed, Critic and Tool Planner share weights?\n- The tool user is not mentioned in the problem setting/method overview.\n- What are the APIs mentioned in 3.3? how are these obtained? are they open source libraries? these need better explanation.\n- Optimisation process is not explained at all in the joint optimiser. This needs to be significantly more elaborate. Preliminaries for the CMA-ES should be added to the main paper or appendix as it is an important component of the pipeline.\n- Table 1 results appear impressive, but some further investigation is necessary. What dataset is used? is this publicly available and if not, will it be made publicly available?\n- The best and average of 8 runs is used, however these results are not very meaningful as they are (best could be an outlier). It would be much better to show mean +- std and since multiple runs are computed, statistical significance of results would be in order.\n- This is the same for the ablation study.", "was_revised": true, "final_justification": "There are some novelty concerns post rebuttal, specifically as the method is proposing in practice a tool-augmented MLLM in a new domain (the authors themselves mention several in the related work and the rebuttal). However, the results are convincing and the motivation for using MLLMs in the task is very clear and intuitive.", "rebuttal": "*We sincerely appreciate your detailed and constructive feedback. We have carefully considered your suggestions and enhanced the clarity and depth of our explanations accordingly. Furthermore, we will open-source our framework to support continued progress in this research area.*\n\n**Weaknesses & Questions**\n\n> **1&2. It is unclear what components are trainable if any are at all and how they are trained. This needs to be made explicit. Do the Proposed, Critic and Tool Planner share weights?** \n\nOur approach is built as a training-free agentic framework. This means we don't require any training data or fine-tuning throughout our pipeline. Instead, we leverage the robust capabilities of GPT-o3, a SOTA model from OpenAI known for its strong reasoning abilities.\n\nThe Proposer, Critic, and Tool User agents in our framework are all instances of the same GPT-o3 model, which means they share the same underlying weights.\n\nHowever, each agent operates differently, as it is guided by a distinct input prompt tailored to define its specific role within our framework. Detailed prompt templates are provided in Appendix B of our paper.\n\nFor example:\n\n- The Proposer is prompted to design a 3D tool asset based on a given robotic task description and 3D scene configuration. It outputs code for generating a 3D tool asset. \n\n- The rendered multi-view images of this tool, along with the Critic's system prompt, are then provided to the Critic agent. The Critic provides natural language modification suggestions, which feed back to the Proposer in an iterative refinement process until no more suggestions are needed.\n\n- The Tool User agent, prompted with the task, scene configuration, and the final tool description, outputs a robotic manipulation trajectory using a set of specified APIs (move, grasp, release). This is a one-time process.\n\nThis strategy of repurposing a powerful, pre-trained model for various roles is a common and effective practice, as demonstrated in prior works like [1, 2, 3, 4].\n\n[1] Wang, Zihao, et al. \"Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents.\" Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023.\n\n[2] Sarch, Gabriel, et al. \"Vlm agents generate their own memories: Distilling experience into embodied programs of thought.\" Advances in Neural Information Processing Systems 37 (2024): 75942-75985.\n\n[3] Park, Joon Sung, et al. \"Generative agents: Interactive simulacra of human behavior.\" Proceedings of the 36th annual acm symposium on user interface software and technology. 2023.\n\n[4] Huang, Wenlong, et al. \"Inner Monologue: Embodied Reasoning through Planning with Language Models.\" Conference on Robot Learning. PMLR, 2023.\n\n> **3. The tool user is not mentioned in the problem setting/method overview.**\n\nOur pipeline consists of 3 components: a critic tool designer, a tool use planner, and a joint optimizer. Within the critic tool designer, there are two VLM agents, Proposer and Critic, that collaboratively design tools. Within the tool use planner, there is a single VLM agent called Tool User to function. Thanks for pointing out that we only mentioned Tool User in Section 3.3, not in the method overview. We'll make it clear in the revisions. \n\n\n> **4. What are the APIs mentioned in 3.3? how are these obtained? are they open source libraries? these need better explanation.**\n\nThe APIs are indeed a minimal set of three: Grasp, Release, and Move, as listed in Section 3.3. We designed and implemented these specific robot control APIs, drawing inspiration from approaches found in prior work such as [5, 6]. Yes, we plan to open-source the implementation of all three APIs, with a particular focus on the robust Grasp API, to facilitate reproducibility and future research.\n\n[5] Wang, Yufei, et al. \"RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation.\" International Conference on Machine Learning. PMLR, 2024.\n\n[6] Liu, Peiqi, et al. \"OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics.\" 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024.\n\n> **5. The optimisation process is not explained at all in the joint optimiser. This needs to be significantly more elaborate. Preliminaries for the CMA-ES should be added to the main paper or appendix, as it is an important component of the pipeline.**\n\nOur joint optimizer employs Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is a powerful evolution strategy for numerical optimization of non-convex problems. At its core, CMA-ES operates by iteratively updating a multivariate normal distribution from which new candidate solutions are sampled. The mean and covariance matrix of this distribution are adapted based on the performance of the sampled solutions, allowing it to efficiently converge on optimal parameters even in complex landscapes.\n\nIn our specific setting, the optimization process within the joint optimizer uses the task metric directly as the reward signal for CMA-ES (e.g., the distance between the cube and ideal position for the Reaching task; refer to Appendix C for more metrics). CMA-ES then optimizes two key aspects: the tool shape parameters and the robot control trajectories.\n\nWe appreciate this valuable feedback. We will expand on the optimization process in Section 3.5 of the final paper, while also including necessary CMA-ES preliminaries in the appendix for improved clarity.\n\n\n> **6. Table 1 results appear impressive, but some further investigation is necessary. What dataset is used? is this publicly available and if not, will it be made publicly available?**\n\nWe curated this benchmark, including 9 tasks encompassing a diverse range of materials and robotic skills. Yes, this benchmark, along with its detailed task configurations, will be made publicly available to the research community to support future work.\n\n\n> **7&8. The best and average of 8 runs is used, however these results are not very meaningful as they are (best could be an outlier). It would be much better to show mean +- std and since multiple runs are computed, statistical significance of results would be in order. This is the same for the ablation study.**\n\nWe appreciate the feedback regarding the statistical reporting of our results and ablation study. Here we present the mean and standard deviation of the P score of the main result and ablation study as follows:\n\n| Method \\ Task | Reach | Hold a Phone | Lift a Bowl | Lift a Piggy | Dough Calabash | Flatten Dough | Cut Dough | Fill a Bottle | Transport Water | Overall |\n| :------------ | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| **No Tool**       | 0.00±0.00 | 0.02±0.03 | 0.13±0.13 | 0.00±0.00 | 0.33±0.13 | 0.12±0.07 | 0.48±0.34 | 0.04±0.02 | 0.00±0.00 | 0.12±0.08 |\n| **Retrieval**     | 0.03±0.04 | 0.52±0.30 | 0.14±0.12 | 0.36±0.37 | 0.15±0.09 | 0.31±0.16 | 0.65±0.27 | 0.05±0.04 | 0.26±0.24 | 0.27±0.18 |\n| **Meshy**         | 0.46±0.20 | 0.79±0.20 | 0.06±0.06 | 0.62±0.32 | 0.61±0.22 | 0.56±0.27 | 0.53±0.34 | 0.58±0.27 | 0.00±0.00 | 0.47±0.32 |\n| **Ours**          | 0.59±0.36 | 0.75±0.22 | 0.71±0.32 | 0.90±0.20 | 0.56±0.22 | 0.69±0.36 | 0.48±0.38 | 0.51±0.34 | 0.47±0.32 | 0.63±0.31 |\n\n| Method | Ours w/o Text-to-3D | Ours w/o Optimizer | Ours w/o Tool Opt | Ours|\n| :------------ | :----: | :----: | :----: | :----: |\n| **Score** | 0.46±0.24 | 0.10±0.07 | 0.32±0.25 | 0.63±0.31 |\n\nOur initial choice to report success rates and best scores was because most task performances are binary. For such discrete outcomes, standard deviation could be less informative. The best score specifically aimed to highlight our pipeline's full potential. As expected with binary scoring, the standard deviation will be relatively high for many tasks.\n\n\n*We hope that the provided new experiments and additional explanations have convinced you of the merits of our work. We wish they had addressed your concerns and turned your assessment to the positive side. If you have any more questions, please feel free to let us know during the rebuttal window.*\n\n*We appreciate your time! Thank you so much!*\n\nBest,\n\nAuthors"}, {"review_id": "s0W2SWwQDl", "reviewer": "Reviewer_BMgf", "summary": "This paper introduces RobotSmith, a framework to design specialized tools for diverse robot tasks using foundation models and physics simulation. RobotSmith has three steps: (1) iterative tool design, where a proposer VLM creates tools via a specialized programmatic representation and a critic VLM provides feedback based on renderings of the tool; (2) planning, where another VLM calls high-level robot APIs to synthesize a trajectory; and (3) tool-trajectory joint optimization, where an evolutionary strategy refines the VLM's parameters by evaluating candidates in simulation with a task objective function. The authors validate their method on 9 diverse manipulation tasks, including those involving soft bodies and liquids, and show that RobotSmith outperforms baselines: no tool use, human tool retrieval, and mesh generation. Additionally, generated tools are 3D printed and applied to 2 tasks in the real world, demonstrating successful transfer.", "questions": "Questions\n1. The discussion on failure cases in Section 4.6 is very helpful, and I wonder if you could also include percentages for each type of failure. That is, which of these failure cases are most important to address for future work?\n2. How does your tool representation compare with MJCF or URDF? I understand a key distinction is the ability to generate geometries, and are there other key advantages? I wonder if these commonly-used formats could be helpful for VLM generation, since they already exist in the internet training data.\n3. Can you provide some more technical details on the main changes that allow RobotSmith to run in real? For instance, with the grasp API, how do you get the location of the object without ground truth simulator state?\n4. Following Weakness 1, can you provide more details on the evaluation procedure and success rate of the real-world trajectories?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The experiments show that RobotSmith's custom-designed tool representation is quite general-purpose, as it allows the VLM to generate many diverse geometries. The tasks that these tools complete is also very different, each requiring different geometric features.\n2. Some of the generated results are validated in real, demonstrating potential for scalable real-world tool design. The long-horizon pancake-making task is especially impressive.\n3. The combination of VLM generation with subsequent optimization is novel and addresses a key issue in VLMs lacking physical precision.\n4. This paper tackles the difficult and promising challenge of tool-design, which has the potential to significantly expand a robot's capabilities.\n5. The paper is written clearly, and the figures are overall illustrative.\n\nWeaknesses\n1. While the method is validated in real, it isn't evaluated over multiple trials. Section 4.4 seems to suggest the existence of successes in real, but does not elaborate on its effective success rate.\n2. The planner only uses three API functions (grasp, release, move), which seem limiting and may restrict it from more freeform trajectories (eg, wiping). The grasp API is also called on the entire object, and the grasp point is not exact; I wonder whether this may hinder the use of a tool due to the heuristic grasp selection choosing an incorrect location.\n3. Figure 2 seems a bit cluttered due to the many textual examples. It may be clearer to have two separate figures, an overview that more abstractly visualizes the method and another that shows examples in each step.", "was_revised": true, "final_justification": "The authors addressed my questions and limitations in their rebuttal. Explanations and justifications were clear and helped me better understand their design. The work is overall convincing and solid, and the only main weakness is the real-world results. However, the authors do demonstrate that their generated tools are kinematically and geometrically feasible in real (given privileged state), and as real-world policy synthesis is not the focus of this work, I believe the real-world experiments are sufficient.", "rebuttal": "*We appreciate the positive and insightful comments from you! We adress your concerns in details below.*\n\n**Weaknesses**\n\n> **1. While the method is validated in real, it isn't evaluated over multiple trials. Section 4.4 seems to suggest the existence of successes in real, but does not elaborate on its effective success rate. & Q4: Following Weakness 1, can you provide more details on the evaluation procedure and success rate of the real-world trajectories?**\n\nIn high level, our pipeline is designed to work in simulation to generate tool-use data, which would be used to train a policy that can be deployed to the real-world in the future.\n\nThus, our real-world experiments aim to validate that tools designed and trajectories optimized in simulation are structurally stable and physically feasible. Thus, we first build a simulated digital twin of the setup, perform tool design and trajectory optimization in simulation, and then 3D-print the tool and execute a selected successful trajectory in the real world.\n\nAlthough evaluating real-world robot performance isn't our main focus, we report the following success rates: *Dough calabash*: 9/20, *Hold a phone*: 20/20 (simple trajectory replay), *Pancake-making*: 1/4 (limited trials due to tedious scene resets).\n\nDespite that theoretically it should be all success, failures occur mainly due to:\n\n- **Shape variation**: Manual resets of deformable objects cause inconsistencies.\n\n- **Material changes**: Contact with water or dough (flour) alters surface friction, affecting grasp reliability.\n\n\n> **2. The planner only uses three API functions (grasp, release, move), which seem limiting and may restrict it from more freeform trajectories (eg, wiping). The grasp API is also called on the entire object, and the grasp point is not exact; I wonder whether this may hinder the use of a tool due to the heuristic grasp selection choosing an incorrect location.**\n\nThe *Grasp*, *Release*, and *Move* APIs are a deliberately minimal set of high-level commands. Their purpose is to let the VLM provide semantic priors for tool use and action planning, which are then refined into accurate trajectories by downstream optimization. Wiping can be approximated by a sequence of *Move* calls. While we could make *Move* more free-form (e.g., via RL), that would introduce many more parameters and significantly increase optimization time. Our design therefore trades some expressiveness for computational efficiency.\n\nFor grasp location, we want to clarify that sampling is not performed over the entire tool surface; it is restricted to parts explicitly designed for grasping (the *is\\_handle* label in Fig. 2 of the main paper). By design, each tool should include a functionally graspable part. As a result, grasps are semantically appropriate and unlikely to interfere with the tool’s function.\n\n\n\n> **3. Figure 2 seems a bit cluttered due to the many textual examples. It may be clearer to have two separate figures, an overview that more abstractly visualizes the method and another that shows examples in each step.**\n\nWe appreciate your advice regarding the clarity of Figure 2. We'll address this in the final version of the paper by adding a clean and brief figure that solely visualizes the overall pipeline more abstractly. The detailed examples, which currently contribute to the clutter, will then be separated into a distinct, dedicated figure. \n\n\n**Questions**\n\n\n> **1. The discussion on failure cases in Section 4.6 is very helpful, and I wonder if you could also include percentages for each type of failure. That is, which of these failure cases are most important to address for future work?**\n\nThanks for finding our discussion on failure cases helpful and for the excellent suggestion to include failure percentages. While our method aims to address the limitations of baseline approaches (such as their failure to retrieve appropriate tools or realize simple geometries for certain function, as seen in Figure 4), we have conducted a further analysis, and the failures in our pipeline can be broken down as follows:\n\n- Invalid Code Generation (32%): A significant portion of failures occurs when the Proposer generates invalid code (e.g., code missing return value captions). Although the Critic detects failures, the Proposer is not always able to correct itself within the limited number of back-and-forth rounds.\n\n- VLM Multi-modal Understanding (47%): The largest source of failure is from the limited multi-modal understanding ability of VLMs. This can lead to undetected failures, such as the generation of non-functional or unreasonable shapes. We believe this will be a primary area for future work and is likely to be mitigated as VLMs continue to improve.\n\n- Optimization Limitations (15%): A smaller portion of failures can be attributed to the limited computational budget for the optimization process, which may not always find the optimal parameters.\n\n- Other Cases (6%): The remaining failures are due to various other miscellaneous issues.\n\n> **2. How does your tool representation compare with MJCF or URDF? I understand a key distinction is the ability to generate geometries, and are there other key advantages? I wonder if these commonly-used formats could be helpful for VLM generation, since they already exist in the internet training data.**\n\nThanks for the excellent question that highlights a key design choice in our work. MJCF and URDF are highly structured formats designed to describe the kinematics and dynamics of articulated, rigid-body systems, such as robot arms. Our tool representation, in contrast, is designed to be highly expressive for generating novel, flexible tools with diverse geometries.\n\nOur tool representation is inspired by Constructive Solid Geometry (CSG), a well-established and powerful method widely used in CAD. This provides a key advantage in expressiveness. For example, it easily supports operations like set minus to create intricate concave shapes, which would be difficult to describe with the joint and link primitives of MJCF or URDF.\n\nIt's a good point that presence in internet training data could be helpful. We believe that given CSG's popularity as a design choice, VLMs also have a strong understanding of this representation. Therefore, our approach combines the VLM's high-level generative capabilities with our representation that is a better fit for creating complex, task-specific geometries.\n\n\n> **3.Can you provide some more technical details on the main changes that allow RobotSmith to run in real? For instance, with the grasp API, how do you get the location of the object without ground truth simulator state?**\n\nAs noted above, we execute an open-loop trajectory optimized in simulation just to prove the physics feasibility. The simulation provides ground-truth state information. To build the digital twin, we manually measure object positions relative to the robot arm—keeping real-to-sim transfer simple, as it's not our primary focus.\n\n*We hope that the provided new experiments and additional explanations have convinced you of the merits of our work. Please do not hesitate to contact us if you have other concerns.*\n\n*We appreciate your time! Thank you so much!*\n\nBest,\n\nAuthors"}]

Author rebuttal:

Meta-review: This paper develops an agentic approach to design and use tools for robotic manipulation, combining VLM-based tool design with evolutionary optimization. Reviewers were generally positive on the paper's clarity, the breadth of evaluation tasks (including deformables and fluids), the approach combining generative priors with physics-based optimization, and overall performance of the approach. The rebuttal clarified some aspects, added real-world success rates, and added ablations and additional analysis. In particular, this converted most of the initial concerns mentioned in the reviews to more minor limitations of the approach (e.g., minor novelty concerns, evaluation scale, and the extent of the real-world experiments). The authors are encouraged to incorporate the rebuttal and discussion when revising their paper.
Decision: Accept (poster)