VLMgineer: Vision-Language Models as Robotic Toolsmiths
Authors: George Jiayuan Gao
Tianyu Li
Junyao Shi
Yihan Li
Zizhe Zhang
Nadia Figueroa
Dinesh Jayaraman
Venue: ICLR 2026 Poster (2026)
Keywords: robotic manipulation
robotic tool use
vision language models
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce VLMgineer, a novel VLM-driven evolutionary framework that automatically co-design tools and actions to solve robotics task.

Abstract: Tool design and use reflect the ability to understand and manipulate the physical world through creativity, planning, and foresight. As such, it is often regarded as a measurable indicator of cognitive intelligence across biological species. While much of today’s research on robotics intelligence focuses on generating better control strategies, inventing smarter tools offers a complementary form of physical intelligence: moving the problem-solving onus into the tool’s geometry so that control becomes simpler. This motivates us to ask: can today’s foundation models offer useful priors to automatically invent—and effectively wield—such tools? We present VLMgineer, the first fully automatic framework designs tools and actions from scratch by harnessing the creativity of Vision–Language Models (VLMs) together with evolutionary search. We evaluate VLMgineer on a diverse benchmark of everyday manipulation scenarios that demand creative tool design and use. Across this suite, VLMgineer consistently discovers tools and policies that solve tasks more effectively and innovatively, transforming challenging robotics problems into straightforward executions. It also consistently outperforms VLM-generated designs from human specifications and existing human-crafted tools for everyday tasks. We further demonstrate that VLMgineer’s automatically designed tools and action policies transfer seamlessly to real-world task execution on a physical robot. To facilitate future research on automated tool invention, we will release our benchmark and code. Project Website: https://vlmgineer.github.io/.

Published as a conference paper at ICLR 2026
VLMGINEER: VISION LANGUAGE MODELS AS ROBOTIC TOOLSMITHS
George J. Gaot, Tianyu Li Junyao Shi, Yihan Li!, Zizhe Zhang' Nadia Figueroa, Dinesh Jayaraman GRASP Lab, University of Pennsylvania
ABSTRACT
Tool design and use reflect the ability to understand and manipulate the physical world through creativity, planning, and foresight. As such, often regarded as a measurable indicator of cognitive intelligence across biological species. While much of today's 's research on robotics intelligence focuses on generating better control strategies, inventing smarter tools offers a complementary form of physical intelligence: moving the groblem-solvin onus into the tool's geometry so that control becomes simpler. This motivates us to ask: can today's foundation models offer useful priors to automatically invent- and effectively wield- -scchhhhhhh tools? We present VLMgineer, the first fully automatic framework designs tools and actions from scratch by harnessing the creativity of Vision-LLanguage Models (VLMs) together with evolutionary search. We evaluate VLMgineer a diverse benchmark of everyday manipulation scenarios that demand creative tool design and use. Across this suite, VLMgineer consistently discovers tools and policies that solve tasks more effectively and innovatively, transforming challenging robotics problems into straightforward executions. It also consistently outperforms VLM- generated designs from human specifications and existing suman-crafted tools for everyday tasks. We further demonstrate that VT.Mgineer' automatically designed tools and action policies transfer seamlessly to real-world task execution on a physical robot. To facilitate future research on automated tool invention, we will release our benchmark and code. Project Website:
INTRODUCTION
Humans exhibit a remarkable ability to design and utilize tools, fundamentally extending their capabilities to accomplish tasks otherwise beyond their reach through creativity, planning, and foresight. This capacity for tool creation and usage represents one of our most distinctive cognitive adaptations, and therefore is widely regarded as marker of cognitive complexity. Achieving comparable versatility in robots demands a coupled approach: the shape of a tool and the motions that wield it should be co-designed
each constraining and enabling the other. Much of today's robotics research concentrates on enabling complex robot motions that use simple standard tools (Shi) al. 2023 Qi et al. [0024] Car al. [2244] Shaw al., 2024 Chen al.) 2024). In this work, pursue an alternative form of physical intelligence: inventing smarter tools that simplify downstream control - thereby shifting the primary problem-sol solving burden from devising control strategies to designing the tool's geometry.
State-of-the-ur vision language models (VLMs) possess vast and impressive common-sense sense rea- soning and creative abilities, alongside extraordinary capabilities in code generation, visual compre- hension, and in-context learning. When combined with evolutionary search methods, VLMS have successfully crafted human-level reward functions for reinforcement learning (Yu et al. 2023 Ma cet al. 2023), 3D graphics (Huang et al. 2024), articulations of in-the-wild objects (Le et al. 2024), intricate 3D sculptural designs (Goldberg et al. (2024) and developing advanced algorithms to solve mathematics and science problems (Romera- Paredes et al. 2024 Aglietti et al. 2024 Novikov et al.
2025).
and denote equal contribution. Correspondence: (gegao, tianyu) @ seas.upenn.edu

---

Published as a conference paper at ICLR 2026
Tool Design
VLMgineer
"The cupcake too away 6
Robot Action
"Now I have the cupcake @"
Given manipulation task that lies outside the robot's capabilities, VLMGINEER first prompts a vision language model to generate a tool and action. We then employ evolutionary search in simulation to refine the tool's geometry and synthesize the corresponding robot motion plan. Finally, the robot, equipped with the automatically designed tool, successfully completes the task.
Figure 1:
In the wake of these results, we ask: can today's 's VLMMs also guide the design of innovative and action-efficient physical tools for robots? We introduce VIMGINERR, an autonomous framework that leverages VLMs to jointly evolve both tool design and manipulation strategies for robots. Our method demonstrates unprecedented efficacy in developing specialized tools for diverse manipulation tasks, through an evolutionary search process guided by VLM- VLM- MeM-geerateed tool geometries and action plans. VLMGINEER is the first fully automatic framework for designing tools and actions from scratch: compared to prior more limited investigations tool design, that largely consider parameter optimization for manually designed parametric template, VLMGINEER works off-the- for new tasks without numan-in-the-loww steps such as task-specific templates, prompts, or examples. To facilitate future research and benchmarking, we also introduce RoBoTOOLD (oocBenc,, a comprchensive simulation suite comprising 12 diverse robotic tool-use manipulation tasks specifically designed to evaluate tool design and policy optimization methods.
In summary, we make the following contributions:
VLMGINEER, novel evolutionary optimization framework that automatically discovers innovative tools solve robotics task more efficiently.
ROBOTOOL BENCH, a comprehensive simulation benchmark consisting of 12 robotic tool-use tasks designed explicitly for evaluating robotic tool and policy designs.
Our fully autonomous approach not only outperforms designs generated with human specifi- cations and human-crafted everyday tools, but also produces tools and actions in simulation that seamlessly transfer to real-world task execution. When evaluated on MOBoTooLBENCEE VLMGINEER achieves an average normalized improvement of 64.7% over VLM-generated designs from human language specifications and outperforms existing human-crafted tools by average normalized improvement of 24.3%. Our results serve to validate both the physical design intelligence enshrined in VLMs pre-trained on web-scale data, demonstrate amplification of VLM physical creativity via evolutionary search beyond prompting, and present the promise of more adaptable and capable roboties systems that can ingeniously create and use tools.
RELATED WORK
Task-specific computational agent and tool design. Previous research has extensively investigated methods for optimizing robot morphology, end-effectors, and tool designs for robot manipulation through various computational approaches, ranging from model-based optimization (Allen et al.
2022). reinforcement learning (RL) (Li al. 2021). data-driven generative models (Wu et al. 2019 Ha et al. 2020; Xu et al. 2024). and differentiable simulation (Li et al.) 2023). Others have explored robot design for locomotion using evolutionary algorithms (Jelisaveic et al. 2019 Hejna III et al.
2021 Walker & Hauser 2021 [iiss 2023 Dong et al.) 2023a b). stochastic optimization (Exarchos) ct al. 2022), and graph search (Zhao et al., (020). However, these existing approaches typically require manual task-specific pre-definition of handful of optimization parameters, rely on fixed trajectories or pre-defined control policies, and tend to suffer from low sample efficiency. Prior efforts like Khan al. (2025) have demonstrated that VLMs can generate functional tool designs, often resembling existing objects found in online databases. In contrast, our core novelty lies in the finding that evolutionary search can elicit physical creativity from VLMs significantly beyond
2

---

Published as a conference paper at ICLR 2026
single-shot prompting. This process allows our system discover nonstandard, highly performant geometries that are not merely retrieved but iteratively optimized for specific tasks. As demonstrated quantitatively in Fig. | and Table I our evolutionary approach yields tools that are significantly more effective than those from the initial VL.--genrated population, confirming that the iterative refinement is crucial for grounding the design in physical performance. We also introduce VLM- driven approach that simultaneously optimizes both tool design and manipulation policies, enabling generalization across diverse manipulation tasks without requiring manual parameter specifications.
Robot learning for tool-based tasks. To learn effective tool usage, some have employed learned or simulated dynamics models for tool manipulation optimization (Xie al., 2019 Allen al. 2020 Girdhar et al., 2020; Lin et al.] (022a/b). Another prevalent approach involves learning tool and understanding the functions of objects and tool-object interactions (Fang et al. 2020 Qin et al. 2020 Brawer et al. [2200] Xu et al., 2021a Noguchi et al. 2021 Shi et al. 2023). Recently, large language models have been leveraged to employ creative tool use (Xu et al. 2023).
object affordances
While such methods typically assume that suitable tools already exist in the environment, we instead address the more practical scenario where a general-purposs robot must concurrently optimize both the tool's design and its manipulation strategies.
Joint optimization of morphology and control. Jointly addressing tool design and control problems has often involved formulating nonlinear programs to solve task and motion planning (TAMP) given predefined design parameter space, which are particularly effective for sequential manipulation over extended horizons (Toussaint et al. 2018 2021). However, given objective to deploy VLMGINEER in any arbitrary environment without manual specification of design parameters, we rely on the underestimated physical creativity of VLMs. Approaches using RL (Wang et al. 2023a bj
Luck et al. 2020, Yuan et al. (02T)) gradient-based optimization (Spielberg et al.] 2019), Bayesian
optimization. evolutionary algorithms (Cheney al., 2018||ertaa & Cheney 2024] Ringel et al.
2025). or combination of them (Lia et al. [2099 Schalf et al. [019:] Ha 2019; Bhatia et al.
2021 Pathak et al. 2019) have been proposed for joint morphology and control learning for robot locomotion tasks, in particular with soft or modular robots. Studies on joint robot or tool and policy design through RL (Chen et al. 2020; [iu et al. (023)) differentiable simulation (Xu et al. 202 and model-based optimization iKawaharazuka al. (2220) have also demonstrated effectiveness in tool manipulation. However, since these methods still all require manual specifications of the design space, they require significant human efforts to scale beyond few tasks.
Recent work has explored LLM-aided evolutionary search for robot design in conjunction with RL-based policy optimization in locomotion (Qiu et al.) 2024 [oong al. 2025), demonstrating the potential of using I.LMs to unlock more performant robot design. Unlike prior work, our work targets open- wren-world VL.Mguidded design of both tools and actions for manipulation without human-in-the loop parameter specification. VLMGINEER leverages the surprising physical creativity of VLMs to automatically create design solutions using evolutionary search. It can easily be scaled to wide range of tasks, and it is much more efficient in terms of samples, time, and compute than prior RL-based methods.
3 BACKGROUND
Evolutionary Methods. Evolutionary algorithms (Langdon & Poli 2013 Doncieux et al. 2015) have a long-standing history in solving optimization problems, inspired by principles of biological evolution and natural selection. They are particularly effective in black-box optimization with vast optimization spaces, such as open-ended design. At their core, these methods maintain a population of candidate solutions, which iteratively evolve through carefully designed mutation and crossover operators. Each iteration evaluates individuals against a fitness function, selecting those with higher fitness while discarding or replacing less successful candidates. To balance exploitation and exploration, crossover combines promising solutions into offspring, and mutation introduces novel variations. Evolutionary algorithms have proven effective across diverse domains such as program synthesis, symbolic regression, algorithm discovery, and even robot design. Nevertheless, their reliance on handcrafted mutation and crossover operators remains a significant limitation such operators are challenging design and often inadequately capture essential domain insights.
Large model-guided evolution. To improve the scalability, performance, and automation of evolu- tionary algorithms, recent work has integrated large models into the evolutionary process, automating
3

---

Published as a conference paper at ICLR 2026
Environment Setup
teds 10625, +1aaarpet CeeTe 2pps we2d, cb5
10.75.
75,
Designs
Designs
Candidate Sampling
CCC
Simulation [+ Evaluation
VLMgineer
Task Description
PyBuliet
The tne roboti dedge five to tej??
VLM
Design Agent
Q
Evolved Sampling
System Instructions
Figure VLMGINEER takes unmodified environment source code, environment image, environmen- tal description, and task description context zero-shot generate tool and action designs from a VLM. then iteratively refines its tool and action designs through a loop candidate sampling, simulation-barn based evaluation, and evolution improvement.
mutation and crossover operations. Leveraging the extensive world knowledge and inductive biases inherent in large models allows for more efficient evolution of candidate solutions and also eliminates the necessity of manually defining allowed mutation operations. Moreover, some approaches exploit the rich semantic understanding of large models to provide nuanced, semantic feedback beyond simple numerical fitness scores. Specific implementations of these principles in evolutionary algo- rithms vary according to the domain. For instance, Ma et al. (2023) employs large language models (LLMs) guide evolutionary reward design in reinforcement learning. Eureka generates population of candidate reward functions from raw environment code, evaluates RL agents trained with these rewards using task-specific filness function, and selects the best-performing candidates. It then employs LLM-guided in-context reward mutation, proposing improved functions based on textual feedback. Drawing inspiration from this, we investigate whether vision-language models (VLMs) can similarly provide inductive biases to guide the evolutionary design of robotic tools and actions.
METHOD
VLMGINEER builds upon previous Large Model-guided evolution methodologies to per- form tool-action co-design. Specifically, VLMGINEER consists of three algorithmic com- ponents: (1) We prompt the VLM to generate diverse population of potential candidate tool- action samples given raw environment code, task description, and system instructions as context. (2) We evaluate each of the design samples via task fitness functions and retain those with the top-k rewards. (3) We iteratively prompt the VLM to produce novel tool- sample offspring via guided tool mutation and crossover, pro- gressively improving tool and action designs.
Algorithm 1 VLMGINEER:
Evolutionary Tool
and Action Co-Design with VLMs
Require: Environment code E, image render task description daa, fitness function F, initial prompt PROMPT, Vision-Language Model VLM
1: Hy Hyperparameters: Number of evolution eycles population size K, top-k selection threshold for itcrations do do
2:
3:
// Sample K designs
4:
D1, D2, Dx VLM(E, dusk, PROMPT)
5:
Evaluate design candidates
81 = F(Di).... .8 = F(Dx)
7:
// Selection
8:
Select top-k designs
with high-
est 8j
This overall VLMGINEER algorithm is summa- rized in Algorithm!] Please refer to Appendix for for the full prompts.
Evolution
10: PROMPT
PROMPT
EVOLUTION PROMPT((D,, D,.,))
11: end for
Joint tool and action candidate sampling. While previous approaches of large model- guided evolutionary robot design (Qiu et al.) 2024 Song al. 2025) typically optimize robot
12: return Final design D" arg max F(D) across iterations
4

---

Published as a conference paper at ICLR 2026
BringCuke
DislosgsCube
Get her Seeerghheee righeree
Ni ghobjjet
1d ftllox
MoveRalll 11
OneBook
SecroCoal
SnatchCcokie
DLexayCogs
Figure 3: VLMGINEER produces innovative tool designs and their corresponding actions across 12 diverse tasks in RoBoTOOLBENCE that are challenging to perform using a general-purpon robot arm and gripper.
morphology alone, relegating action or control optimization to subsequent evaluation stage. Our approach prompts the VLM to simultaneously generate paired tool designs and corresponding action strategies in single inference step. Our key insight behind joint tool-action sampling is that it allows for tighter coupling between tools and their associated actions. Rather than sequentially optimizing the tool geometry first and then actions afterward, simultaneous optimization leverages the VLM's inductive biases to smoothly navigate the joint Concretely, within each evolution cycle, VLMGINEER prompts the VLM to propose n distinct tool designs along with candidate action plans per tool, resulting in corresponds kind crude VI.M-gudded policy optimization, which merely selects the best among the generated action plans. Compared to policy optimization via RI. (Ma et al.) 2023 Song et al.) 2025 Qiu et al. 2024), our action sampling approach, albeit simple, significantly accelerates iteration cycles and reduces computational overhead by exploiting the insight that appropriately designed tools inherently simplify and enhance action plans. We also empirically observed this strategy to outperform other forms of feedback, such videos object-centric signals, whose low accuracies tend to degrade performance and limit meaningful improvement.
design space towards the Pareto frontier.
total tool-action pairs. This
Specification of crossover and mutation. A critical part of how VLMGINEER enables effective tool design evolution is the utilization inductive in-context crossover and mutation. We define inductive in-context crossover and mutation as the process of prompting VLMs introduce random, free-form tool mutations and crossovers, conditioned on previous elite tool candidates, and guided by the model's 's learned inductive biases for producing better task-solving tools. We use the prompt below to perform inductive in-context crossover and mutation: "Your design decision is part of genetic algorithm for tool creation, where each new design produced either by mutation cuaation-channing exactly one aspect (e.g., adjusting component' 's dimension or adding/removing component) by crossover, combining elements from two existing designs. All resulting mutations and crossovers should plausibly enhance task success while preserving design diversity."
Tool representation format. Selecting an appropriate representation for tools balancing abstrac- critical for effective optimization. Prior works have represented objects and tools as meshes (Nair et al. 2020), CAD (Thomas et al. 2018) or blocks (Goldberg al.) 2024). These representations, however, either introduce excessive complexity and optimization challenges or lack sufficient expressiveness. Inspired by prior work (Le et al. 2224).
tion, design flexibility, and manufacturabillyy
represent tools in Unified Robot Description Format (URDF). The structured, modular nature of

---

Published as a conference paper at ICLR 2026
Franka Grioper
LLM expert
Layparson
Pobatics expert
Curs
feward
BringGube
CleanTable
DistodgeCube
ElavteePlat
GatherSpatres
HighCbjer
Liftior
MoveBall
GheBook
ScoreGoal
SnatshCookie
Tureyyaag
Taak Average
Figure 4: Comparison of rewards for the Franka Gripper, 3 Human Prompt experiments, and our proposed method across tasks. Darker bars indicate the average reward over five runs, while paler bars indicate the best reward. All results here were using gemini-2 -2. pro wrwpppeeeeee-eee 03 03-25.
URDF, analogous to code blocks, aligns seamlessly with vasione models' (VL.Ms) strengths in code understanding and generation. Concretely, prompt the VLM to generate URDF-defined tool designs as modular blocks that can be directly integrated into a designated end-effector link of the robot model.
Action representation format. Building on recent work that leverages VLMs for action gener- ation (Di Palo & Johns 2024 Yin et al., 2023), prompt the model explicitly output action sequences in the form of an N 7 array, where N denotes the number of waypoints. Each row encodes 6-DoF pose for the robot end-effector, along with gripper open/close command. In this work, we intentionally use discrete waypoints to show that smarter tools could reduce the need for sophisticated policies. When needed, however, our framework is feasible for dynamic force-based actions (e.g. force vector or wrench estimate for force-based reasoning) and can warm-start expressive closed-loop policies.
5 ROBOT TOOL DESIGN BENCHMARK
We propose comprehensive simulation benchmark ROBOTOOL.BE.S BEnCH designed explicitly for evaluating robotic tool and policy design. PABoTToOLBBnCC comprises 12 object manipulation tasks designed to be challenging for the conventional robot morphology to complete. These task environ- ments are visualized in Fig. 3 For several tasks (BringCube, CleanTable, Gatherrpheeeer ScoreGoal), we took inspiration from the subset of RLBench (James et al. 2020) tasks that involve tool use note, however, that we expect that automated tool design will replace and improve the original tools from RLBench. Several other tasks (HighObjecct Eltt,,Eeee ElevatePlate are inspired by prior works computational co-design (Liu et al. 2023) that study task-specific design parameter optimization discussed in See 2 Still more task environments are inspired by everyday home scenarios (Li Box,, MoveBall, OneBook, Snat chCookie, TurkeyLegs). Finally, Dis SeislodgeCii Cube is inspired by tool design behavior previously observed in the Caledonian crow (Jacobs et al. 2016), which used tools to retrieve objects confined spaces. We adopt the Franka Panda robot arm as the standard morphology attach tools to, and implement our environ- ments using PyBullet (Ellenberger 2018-200 2018-2019). For more details of the each task, please refer to Appendix A.2
6 EVALUATION
Our experiments are designed to provide a comprehensive analysis of VLMGINEER'S 's capabilities. We aim to answer the following questions:

---

Published as a conference paper at ICLR 2026
Q1: Can VLMGINEER effectively discover innovative tools and the actions use them across
diverse set of manipulation tasks?
Q2: How does VLMGINEER 's autonomous co-design compare to tool designs specified to VLM
by human users with different expertise?
Q3: How important is the evolutionary framework VLMGINERR'S performance?
Q4: Can the co-designed tools and actions be successfully transferred from simulation to solve
tasks in the real world?
6.1 PERFORMANCE ON ROBOTOOLBENCH
Baselines. To showcase VLMGINEER 's ability to generate creative and effective tools and usage actions, we compare our method with the following baselines: (1) Franka Gripper: We evaluate the performance of the vanilla Franka Panda two- two-finger gripper without additional tools on RoBOTOOL- BENCH to highlight the inherent limitations of the robot's 's default morphology; these tasks are after all explicitly designed be very hard impossible to perform without the right tools. We derive the no-tool action policy by prompting the VLM to follow an action-sampling procedure analogous to our proposed method, minus the use of any tools.
(2) Human Prompts: For these baselines, we ask humans to specify tool design to the VLM in natural language, following which it attempts to generate that tool and several action plans, in our method. There no evolutionary search. We evaluate on humans with varying expertise: "Robotics expert" (a graduate student researching robot learning). "LLM expert" graduate student researching LLMs), and "Layperson" (an undergraduate student with no relevant research experience). (3) RLBench Tools: We evaluate four original tools from the tasks we adapted from RLBench, which are often natural everyday tools for the tasks considered. While RLBench tools are existing tools for everyday tasks, they might not have been explicitly optimized for maximum task success. However, as illustrated Fig. [] they represent sensible and common designs used in practice, making them practical and meaningful benchmarks. For further details of our experimental setup, see Appendix A For consistency, all evaluations were wwwrroopreveew--Oe-
done using gemini-
Note that a key distinction between VLMGINEER and other studies in the Related Work section is that the other studies all involve substantial manual parameter tuning or predefined parametric tool designs, which are fundamentally different from VIMGINEER 's fully automated approach. Hence, direct apples-to-apples comparisons would be challenging. In fact, VLMGINEER could serve as strong prior for these related works.
Evaluation Metrics. To assess the quality of tool-action design after each execution, we define the evaluation metric as Task Rewards, which is a set of pre-defined task reward functions R S r E [0, 1] that unique each task, where S is its environmental state and r is normalized reward. These rewards are designed to evaluate the progress made in the task by certain tool-action pair.
Results. The results are summarized in Fig. VLMGINEER works consistently well across tasks, in terms of both average and best rewards. We dive into interesting individual method comparisons now. As expected, the default Franka Panda two-finger gripper fails the majority of these tasks. What is perhaps more noteworthy is that VIMGINEER outperforms human-prompting. This true even for expert humans across all tasks and on both metries (better peak performance and also more reliable). While human prompts occasionally produced strong solutions, their results were less consistent and efficient. In tasks like Cl eeeeTaablee and ScoreGoal, both approaches reached similar peak rewards, but our method did so with significantly shorter paths. For further analysis, Fig. 5 shsowws example designs from human- promanng and VLMGINEER. Human designd tools (left column) generally offer suitable forms for task completion; however, VIMGINEER (right column) creates more specialized features that enhance performance. For instance, in task ScoreGoal. method produces long and bent shapes facilitating simpler, more efficient motions, which the robot just need to move very little along one axis to hit the puck. On the other hand, the straight tool designed from human prompts would require more careful control of the puck. In Gathe rSphere, re, our design includes a scoop with side protection and an overhead stripe structure, effectively preventing spheres from bouncing away.
VLMGINEER tools also outperform the RLBench original tools. On the four RLBench- tasks, evaluated the standard RI Benchh Tools (Fig. middde column). As shown in Fig. 6 across
7

---

Published as a conference paper at ICLR 2026
Human
RLBench
Ours
BringCube
ScoreGoai
Gather
Figure 5: Qualitative comparison of human-designed, RLBench, and VLMGINEER tools on three tasks: BringCube (top row), ScoreGoal (middle row), and GatherSpheres (bottom row).
ALDenrch
Reward
0.6
0.2
0.0
BubngGube
CleanTable
SphrerSpheren
SporeGoal
Tank Average
Before Eva.
After Evo.
Figure 7: Qualitative examples of evolution- ary refinement. The process makes intuitive and effective improvements to initial tool de- signs for tasks like GatherSpheres (top) and MoveBall (bottom).
Figure 6: Performance comparison against stan- dard RLBench tools. VLMGINEER consistently outperforms the original, human-crafted tools across four relevant tasks.
every task, VLMGINEER not only attains the highest possible reward but does so more reliably (on average) than RLBench Tools. Qualitatively inspecting the tools further highlights the advantages of our method. The RLBench tools, originally designed for similar but distinct tasks, often underperform due to less optimized features. For example, in BringCube, the RLBench's 's simple stick provides insufficient lateral control, resulting in inconsistent cube manipulation. Our method's cage-like structure reliably locks and moves the cube closer, achieving significantly higher rewards.
8

---

Published as a conference paper at ICLR 2026
6.2 EVALUATING THE ROLE OF EVOLUTIONARY SEARCH
To isolate the contribution of our evolutionary framework, we conducted an ablation study. Across three trials for each of the three chosen ROBOTOOL BENCH tasks, tested VLMGINEER'S standard evolutionary process (4 iterations of 2000 samples) against a brute-force baseline that used VLM sampling for 8000 evaluations, ensuring an identical sample budget for both methods. The results, presented in Table | show a clear advantage for the evolutionary approach. On average, the evolutionary search strategy outperformed the sampling baseline by a significant 119.2%
This quantitative advantage can be understood through the qualitative nature of the evolutionary process, We consistently observed evolution making intuitive and effective enhancements initial designs. As illustrated in Fig. 7 for the Gat herSpheres task, an open scoop was refined with guardrails to prevent spillage. Similarly, for MoveBal 1, an open-ended pusher was augmented with hugging rim to improve control. This suggests that iterative refinement is key mechanism that allows VLMGINEER to discover robust and high-peeforming tool designs that are difficult to find through simple sampling.
Table I:
Mean normalized reward (0-1) for Evolution vs. VLM Sampling under an Equal Sample Budget. The structured search of VLMGINEER substantially outperforms the brute-force sampling baseline, highlighting the critical role of the evolutionary framework.
Method
ElevatePlate
RetrieveHigh
CleanTable
Average
VLMGINEER (Evolution)
0.925 0.007
1.000 0.000
0.888 0.018
0.938 (+119.2%)
VLM Sampling (Baseline)
0.317 = 0.254
0.501 = 0.705
0.466 + 0.124
0.428
6.3 SIM-TO-REAL TRANSFER AND WORLD VALIDATION
To validate that our VIMGINEER generated design and actions work in real robotic scenarios, we selected three Mals- evatee ate, and Gat GatherSpheres: res-to transfer zero-shot to Franka robot. After rumning VLMGINEER in simulation, replicated the simulated environment in the real world, 3D printed the best manufacturable tool, mounted the tool on the robot, and played the associated action plan. The only minor modification we apply is to trim any portion that overlaps with the tool mounting head, a standard post-pros post-processing step. We present snapshots of our real-world experiment execution in Figure More details of our sim-to-real transfer process and real can be found in Appendix [A.9 In our real-robot experiments, for each task, we recorded the normalized rewards by executing the co-design for 5 runs to account for any environmental variance during execution, and obtained an average normalized reward for each task's 's winning co-design. The average normalized rewards from sim-to-real transfer experiments for MoveBall, ElevateP Pltte, e, and Gathers r Spheres are 0.959, 0.761, and 0.713, respectively. Overall, we observe VLMGINEER s tools and actions effectively translate to real-world scenarios. Please refer to our website for the real-world videos.
7 CONCLUSION
We propose VLMGINEER, the first fully autonomous framework for co-optimizing tool design and tool use actions by leveraging the creativity of a VLM. By evaluating on 12 different simulation tasks, we demonstrate the capability to design and use tools to solve challenging robotic manipulation problems. Our results show that VL.GGINERR outperforms baselines that either use no tools or take design specifications directly from humans. We demonstrate that co-designed tools and actions successfully transfer to a real robot, solving three representative tasks with high performance. Our ablation studies further confirm that our evolutionary framework is critical component, providing significant performance boosts over unstructured sampling.
Limitations and Future Works. While VLMGINEER demonstrates significant advancements, several limitations remain: (1) Although we have shown successful sim-to-real transfer, broader validation across more dynamic real-world scenarios is needed. (2) Robot actions represented as

---

Published as a conference paper at ICLR 2026
Figure 8: Snapshots of the real-world tool-action rollout on Franka robot. The three rows correspond to three tasks: MoveBall, Eleva ElevatePl late, and GatherSpheres.
discrete end-effector poses, limiting the handling of complex dynamic tasks. (3) Tool representations in URDF are constrained to simple geometries, and comprehensive evaluation of more complex, articulated tools is needed. (4) VLMGINEER currently optimized for individual tasks, and we have not explored multitask optimization generalization. See App. A.8 for failure modes. Future work should focus on enhancing action representations, exploring richer tool designs, and pursuing multitask generalization. (5) Current VLMGINEER pipeline does not consider manufacturing consid- erations and limitations. Future work needs to explore ways of incorporating these constraints into the optimization for designs that could be more readily manufactured. (6) One might ask whether is practical to design new tools for every task. However, we cannot assume that a convenient tool already exists in the environment for every new tools suited to task is a general capability provided by VLMGINEER. We acknowledge ledge that integrating existing or previous tools beneficial, and future integration with methods capable of selecting between existing and newly generated tools could enhance practicality and scalability. (7) VLMGINEER requires lewelevel and cettromment-speciit information, including raw environment code and task description as input. While many other works have attempted to build digital twins of the real world (Torne et al., 2024) or training policies in simulation that are transferable the real world (Ma et 2023), ) we instead investigate much less studied aspect: automating the design of tools and how to wield them. Our focus is on designing a fully autonomous framework for this co- design problem, minimizing any task-specific manual design. We leave integrating well-established modules, such as digital twin constructions and sim2real policy training, to future work.
task. Instead, the ability to design and fashion
8 REPRODUCIBILITY STATEMENT
We aim to make our results straightforward to replicate. The benchmark tasks, environment details, and dense reward definitions are specified in A.2 The major evaluations are performed in a widely available simulator, PyBullet. The full algorithmic pipeline is illustrated in Figure and and described in Algorithm| [ with hyperparameters summarized in Appendix Appendix(A.3.6) We include the complete set of prompt templates and composition rules used in all experiments in Appendix A..4 Finally, we indicate the paper that the benchmark and code will be released facilitate future research.
ACKNOWLEDGMENTS
We thank our anonymous reviewers, who provided thorough and fair feedback that improved the quality of our paper. This work was supported by NSF CAREER Award RRR-2443721, NSF CARÉER Award 2239301, NSF SLES 2331783, ONR N00014-22-1-2677, and DARPA TIAMAT HR001 12490421.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco JR Ruiz, Alan Malek, Alexis Bellot, and Silvia Chiappa. Funbo: Discovering acquisition functions for bayesian optimization with funsearch. arXiv preprint 77xiv:24406.08824, 2024.
Kelsey R Allen, Kevin A Smith, and Joshua B Tenenbaum. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings the National Academy of Sciences, 117(47):29302-29310 2020.
Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez Pe- ter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.0072 2022.
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym: A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34:2201-2214 2021.
Jake Brawer, Meiying Qin, and Brian Scassellati. A causal approach to tool affordance learning. In 2020 IEEE/RS international conference on intelligent robots and systems (IROS), pp. IEEE, 2020.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Benchmarking in manipulation research: Using the IEEE Robot. Autom. Mag., 22(3):36-5,, September 2015.
object and model set.
Arvind Car, Sai Sravan Yarlagadda, Alison Bartsch, Abraham George, and Amir Barati Farimani. Plato: Plamning with Iims and affordances for tool manipulation, 2024. URL https:://axx
hxg/abs//409. 11580
Tao Chen, Eric Cousineau, Naveen Kuppuswamy, and Pulkit Agrawal. pelling: study in constrained dexterous manipulation, 2024. URL https:://aax....c////////// 07884
peeling: A
case
Tianjian Chen, Zhanpeng He, and Matei Ciocarlie. Hardware as policy: Mechanical and compu- tational co-optimization using deep reinforcement learning. arXiv preprint arXiv:2008.044: 04460, 2020.
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphol- ogy and control in embodied machines. Journal of The Royal Society Interface, 15(143):0017033) 2018.
Norman Di Palo and Edward Johns. Keypoint action tokens enable in no-context imitation learning in robotics. In Proceedings of Robotics: Science and Systems (RSS), 2024.
Stephane Doncieux, Nicolas Bredeche, Jean-Baptiste Mouret, and Agoston E Eiben. Evolutionary robotics: what, why, and where to. Frontiers in Robotics and A1, 2:4, 2015.
Heng Dong, Junyu Zhang, Tonghan Wang, and Chongjie www.hanngnnmmtyyaaaaaaaaa Sywmetry-aware robot design with structured subgroups, 2023a. Utt/stppse//axxxx....///b////333
Heng Dong, Junyu Zhang, and Chongjie Zhang. robot design, 2023b. URL https: http:////wr.v..oo////////111111
ebbeeninn for coarse-to-fine
Benjamin Ellenberger. bobbulee-gym 2018-2019.
Pybullet gymperium.
ht.ps:://github.com/bbe
Ioannis Exarchos, Karen Wang, Brian H Do, Fabio Stroppa, Margaret M Coad, Allison M Okamura, and C Karen Liu. Task-specific design optimization and fabrication for inflated- -eeee soft robots with growable discrete joints. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7145-7151. IEEE, 2022.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei. and Silvio Savarese. Learning oriented grasping for tool manipulation from simulated self-superviision The International Journal of Robotics Research, 39(2-3):022-266, 2020.

---

Published as a conference paper at ICLR 2026
Rohit Girdhar, Laura Gustafson, Aaron Adcock, and Laurens der Maaten. Forward prediction for physical reasoning. arXiv preprint 1rXXi:2006.107344, 2020.
Andrew Goldberg, Kavish Kondap, Tianshuang Qiu, Zehan Ma, Letian Fu, Justin Kerr, Huang Generative design-for-robot-
Huang, Kaiyuan Chen, Kuan Fang, and Ken Goldberg. Blox-net: assembly using vlm supervision, physics simulation, and a robot with reset, 2024. URL https
//arxiv. wttpv//..rr//b////000./ .00009..7110 17128
David Ha. Reinforcement learning for improving agent design. Artificial life, 25( 2019.
Huy Ha, Shubham Agrawal, and Shuran Song. Fit2Form: 3D generative model for robot gripper form design. Conference on Robotic Learning (CoRL), 2020.
Donald Hejna III, Picter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. arXiv preprint 1rXiv:2102.131000 2021.
lan Huang, Guandao Yang, and Leonidas Guibas. Blenderalchemy: Editing 3d graphics with vision-language models. In European Conference on Computer Vision, pp. 297-314. Springer, 2024.
Ivo Jacobs, Auguste von Bayern, and Mathias Osvath. A novel tool-use mode in animals: New caledonian crows insert tools to transport objects. Anim. Cogn., 19(6): 124--1222, November 2016.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
Milan Jelisavcic, Kyrre Glette, Evert Haasdijk, and AE Eiben, Lamarckian evolution of simulated modular robots. Frontiers in Roboties and AI, 6:9, 2019.
Kento Kawaharazuka, Toru Ogawa, and Cota Nabeshima. Tool shape optimization through backprop- agation of neural network. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8387-8393. IEEE, 2020.
Muhammad Haris Khan, Artyom Myshlyaev, Artem Lykov, Miguel Altamirano Cabrera, and Dzmitry Tsetserukou. Evolution 6.0: Evolving robotic capabilities through generative design. arXiv preprint arXiv:2502.17034 2025.
William B Langdon and Riccardo Poli. Foundations of genetic programuning. Springer Science & Business Media, 2013.
Long Le, Jason Xie, William Liang, Hung-Ju Wang. Yue Yang. Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Articulateannything: Automatic modeling of articulated objects via vision- languaee foundation model. arXiv preprint arXiv:2410.13882. 2024.
Mengxi Li, Rika Antonova, Dorsa Sadigh, and Jeannette Bohg. Learning tool morphology for contact- rich manipulation tasks with differentiable simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1859-1865 18699 IEEE, 2023.
Yunfei Li, Tao Kong, Lei Li, Yifeng Li, and Yi Wu. Learning to design and construct bridge without blueprint. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2398-2405. IEEE, 2021.
Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calan- dra. Data-efficient learning of morphology and controller for microrobot. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2488-2494 IEEE, 2019.
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint ar Xiv:22003. /7755, 2022a.
Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, and David Held. Planning with ppatial-eemporal abstraction from point clouds for deformable object manipulation. In 6th Annual Conference on Robot Learning, 2022b. URL https 7/openrevies eww wwt.:iiwwweeeeeeeeeeeeeeeeeeeeee nwt./forum fo
12

---

Published as a conference paper at ICLR 2026
Ziang Liu, Stephen Tian, Michelle Guo, C. Karen Liu, and Jiajun Wu. Learning to design and use tools for robotic manipulation. 2023. URL pottts://arxiie org/ aws.assss.... 00754
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 854 PMLR, 30 Oct-01 Nov 2020. URL https::/prrceed:: ngs m m www.mmrrppp. press v 00/ wckk0uc html
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310. 221.21110,1 2023.
Alican Mertan and Nick Cheney. Investigating premature convergence in co-optimization of mor- phology and control in evolved virtual soft robots, 2024. URL /tLL::tt////////ooorr///
24C2. 09231
Lakshmi Nair, Nithin Shrivatsav, and Sonia Chernova. Tool macgyvering: A novel framework for combining tool substitution and construction. arXiv preprint 100::8888..668, 2020.
Yuki Noguchi, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Tool as embodiment for recursive manipulation. arXiv preprint arXiv:21 2021.
Alexander Novikov, Ngån Vū, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Push- meet Kohli, and Matej Balog. Alphaevolve: coding agent for scientific and algorithmic discovery, 2025. URLpttss: arxiv. orq abs 2006.55066... 13131
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019.
Carl Qi, Yilin Wu, Lifan Yu, Haoyue Liu, Bowen Jiang, Xingyu Lin, and David Held. Learning generalizable tool-use skills through trajectory generation. IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), 2024.
Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, and Silvio Savarese Keto: Learning keypoint representations for tool manipulation. 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7278-7285. IEEE, 2020.
Kevin Qiu, Krzysztof Ciebiera, Pawel Fijalkowski. Marek Cygan, and Lukasz Kuciński. Robomorph: Evolving robot morphology using large language models. arXiv preprint arXiv:2407.08622 2024.
Ryan P. Ringel, Zachary S. Charlick, Jiaxun Liu, Boxi Xia, and Boyuan Chen. Text2robot: Evolu- tionary robot design from text descriptions, 2025. /BLnttps:///axxx..oo/////// 19963
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 40088-475,000 2024.
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning to construct and control agents using deep reinforcement learning. 2019 international conference on robotics and automation (ICRA), pp. 9798 -9005990..011 IEEE, 2019.
Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, and Deepak Pathak. Bimanual dexterity for complex tasks. In 8th Annual Conference on Robot Learning, 2024.
Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long- clasto-plane -lasttc object manipulation with diverse tools, 2023. URL https: http:s/:///aaxxxx///a./. 2306 14447
13

---

Published as a conference paper at ICLR 2026
Karl Sims. Evolving virtual creatures. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pp. 699-706. 2023.
Junru Song, Yang Yang, Huan Xiao, Wei Peng, Wen Yao, and Feifei Wang. Laser: Towards diversified and generalizable robot design with large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojeiech Matusik, and Daniela Rus. Learning- in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations. Advances in Neural Information Processing Systems, 32, 2019.
Garrett Thomas, Melissa Chien, Aviv Tamar, Juan Aparicio Ojea, and Pieter Abbeel. Learning robotic assembly from cad, 2018. URL https: /ttppp://ariiii www. w00. 00
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A -oa-tt--im-to--rall approach for robust manipulation. arXiv preprini arXiv:2403.33949, 2024.
Mare Toussaint, Jung-Su Ha, and Ozgur S Oguz. Co-optimizing robot, environment, and tool design via joint manipulation planning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6600-6606. IEEE, 2021.
Mare A Toussaint, Kelsey Rebecca Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and systems foundation, 2018.
Kathryn Walker and Helmut Hauser. Evolution of morphology through sculpting in voxel based robot. In Artificial Life Conference Proceedings 33, volume 2021, pp. 27. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info .... 2021.
Yuxing Wang, Shuang Wu, Haobo Fu, QIANG FU, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Curriculum-basee co-design of morphology and control of voxel- soft robots. In The Eleventh International Conference on Learning Representations, 2023a. URL https
//openrevew.. net.
Yuxing Wang. Shuang Wu, Tiantian Zhang, Yongzhe Chang, Haobo Fu, Qiang Fu, and Xueqian Wang. Preco: Enhancing generalization in co-design of modular soft robots via brain-body pre-training In Conference on Robot Learning, pp. 478-498 PMLR, 2023b.
Yizhe Wu, Sudhanshu Kasewa, Oliver Groth, Sasha Salter, Li Sun, Oiwi Parker Jones, and Ingmar Posner. Imagine that! leveraging emergent affordances for 3d tool synthesis. arXiv preprint arXiv:1909. 13561, 2019.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang. Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: simulated part-based interactive environment. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2020.
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. arXiv preprint 19455:1904.05538 2019.
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martin Martin, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Deep affordance foresight: Planning through what can be done in the future. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 6206-6213. IEEE, 2021a.
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojeiech Matusik, Shinjiro Sueda, and Pulkit Agrawal. An end-to-end differentiable framework for contact-aware robot design. In Robotics: Science and Systems XVII, RSS2021. Robotics: Science and Systems Foundation, July 2021b. doi: 10.15607/rss.2022 202. xvvi.00..
URL http: http:///xxxooo.r/////// 100 015560//S53 2021 XVII
000
Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, and Ding Zhao. Creative robot tool use with large language models, 2023.
14

---

Published as a conference paper at ICLR 2026
Xiaomeng Xu, Huy Ha, and Shuran Song. setamics-guider diffusion model for robot manipulator design. arXiv preprint arXiv:2402.15038, 2024.
Yida Yin, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig, In-context learning enables robot action prediction in Ils.. In ICRA, 2025.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https: /tttpps://axxv...e/ http:/iwwwwggg/////aaaa/aaaa https///aaa...orrr/aa///////
/
10897
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. Arxiv preprint arXiv:2306. 08647, 2023.
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris Kitani. Transform2act: Learning a transform- and-control policy for efficient agent design. arXiv preprint arXiv:2110.03659) 2021.
Allan Zhao, Jie Xu, Mina Konaković-Lakoove Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Robogrammar: graph grammar for terrain-optimizer robot design. ACM Trans. Graph., 39(6), November 2020. ISSN 0730-0301. doi: 10.1445/3444685..444 34416853.311111... URL
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 BASELINE DETAILS
A.1.1 HUMAN-PROMPTED DESIGNS EXPERIMENT IMPLEMENTATION
Each participant underwent the following experimental procedure for each task: (i) We provided a screenshot of the environment and description of the task, accompanied by brief Q&A session to ensure the participants understood the task. (ii) Participants then had five minutes to write prompt in English specifying their desired tool design and robot action. We instructed participants be as descriptive as possible while focusing on both the design of the tool and how the robot should use it to accomplish the task. (iii) we integrated their prompt into our standardized request to the VLM (by adding instructions as shown in Appendix A.4.9). generating 5 tool described in URDF format along with batch of 10 samples of action waypoints for each tool. (iv) The VLM outputs were then evaluated in our simulation environment using the same reward metrics described in Section 6. (v) Finally, we evaluated and recorded the best-perfonee rest-performing tool and action pair based on the task reward metric for each participant. For case study. obtained prompts from three humans coming from three different backgrounds, including an LLM expert (a student with extensive research experience in LLM), robotics expert (a student with extensive research experience in robotics), and layperson (with no technical background). This case study will serve as an initial attempt on the concept. In the future, plan recruit more human subjects conduct human study experiments on larger sample population.
A.1.2 No-ToOL EXPERIMENT IMPLEMENTATION
In the no-tool baseline experiment, we evaluate the robot's performance without any additional tool attachment. The Franka Panda robot uses its original two-finger gripper perform the task, with the VLM generating action waypoints for the robot end effector pose and gripper open/close, totaling 7 degrees of freedom. The prompt for this baseline is adapted from our proposed prompt by removing the tool design component and associated instructions, while retaining the task description and action generation requirements. We use 5 agents with each generating 10 samples of action waypoints, evaluated using the same metrics introduced in Section 6. The complete no-tool prompt is provided in Appendix A.4.8
A.1.3 RLBENCH EXPERIMENT IMPLEMENTATION
In the RLBench experiment, we evaluate the robot's performance with tools from RLbench. We assume the tool is already attached to the end effector without considering the picking step. The tool are sealed to adapt to our tasks which are similar to the ones in RLBench. The prompt for this baseline is also adapted from our proposed prompt by removing the tool design component and associated instructions. We use 5 agents with each generating 10 samples of action waypoints, evaluated using the same metrics introduced in Section 6. The complete no-tool prompt provided in Appendix A.4.10
A.2 ROBOTOOLBBECH DETAILS
In this section, we provide detailed descriptions of each task and their corresponding dense reward functions in ROBOTOOL BENCH.
16

---

Published as a conference paper at ICLR 2026
BringCube
In this task, red cube on the desk which is out of the reach the robot is needed to be brought closer to the target zone.
The reward measures how close the cube is to the target as fraction of its starting distance, and scales it to 0-1,
CleanTable
In this task, the colorful cubes representing dusts need be pushed away from the robot into circular target zone marked by the green boundary.
The reward reflects, on average, how far each cube has been pushed toward the goal circle, and scales 0-1.
DislodgeCube
In this task, a red cube confined within white, transparent pipe in front of the robot, which has two exits: one opening faces the robot (along negative X) and the other at the front-right corner (along negative Y). The objective is to dislodge the cube through either opening. The reward captures the cube's progress toward either of the two pipe exits by computing two separate, normalized (on 0-1 scale) "distanceeto-exit" scores and then taking the better one.
ElevatePlate
In this task, a white plate placed on the desk in front of the robot needs be securely lifted up. The reward measures how far the plate has moved from its starting position to the desired lifted position, and scales to 0~1.
17

---

Published as a conference paper at ICLR 2026
GatherSpheres
In this task, an open three-walled container filled with small purple spheres is placed before the robot. The objective is to gather and elevate many spheres as possible above 0.3 m.
The reward captures, on average, how high the spheres have been lifted up to specified cap, and scales it to 0-11
ARSIE BEOPT
HighObject
In this task, a green cube sits on top shelf. The objective is to place it inside the beige box positioned between the shelf and the robot. The reward combines hard "in-box" check with smooth distance-based signal and a bonus for lowering the cube off the shelf, and scales 0-1.
LiftBox
In this task, brown box on the desk in front of the robot must be lifted above a height threshold of 0.25 m. This reward measures how much the box has moved toward its target (lifted) position, and scales it to 0-1.
MoveBall
In this task, a red ball on the desk must be moved from the robot's left side its right side. The reward balances two objectives, getting the ball toward the right-side target and keeping its speed in check, and scales it to 0-1.
18

---

Published as a conference paper at ICLR 2026
OneBook
In this task, book holders with five books between them are in front of the robot. The objective is to pull out the middle (3rd) book while keeping the others in place. This reward balances two goals, pulling out the middle book and keeping the others perfectly still, and scales 0-1.
ScoreGoal
In this task, a hockey puck and goal are placed on the ground far from the robot. The objective place the puck inside the goal.
The reward gives full credit once the puck is entirely inside the goal's 3D bounding box, and otherwise scales linearly with how much closer the puck is, horizontally, the goal than it was the start, and scales it to 0-1.
SnatchCookie
In this task, a transparent jar of cookies sits on the desk in front of the robot. The objective take at least one cookie from the jar. The reward checks whether any cookie has been lifted out of the jar, and otherwise gives partial credit, from 0 to 1, based on how high the tallest cookie has been raised.
TurkeyLegs
this task, a silver pot with handles on both sides, full turkey legs, sits on the desk in front of the robot. To the pot's 'ss left (robot's 's perspective) is chef's box. The objective is to transfer all turkey legs into the box without moving the pot.
The reward combines two checks, keeping the pot out of the box and getting each turkey leg into the box, by multiplying, and scales it 0-1.
19

---

Published as a conference paper at ICLR 2026
A.3 VLMGINEER IMPLEMENTATION DETAILS
A.3.1 DESIGN AGENTS CONTEXT
This section describes the context that applies to single design agent. The design agent is provided with task context as illustrated in Fig. 2, which includes (1) the environment code, (2) a screenshot of the environment, (3) a brief task description, and (4) total text prompt composed by the prompts in Appendix A.4 We also provide task-agnostic context, including (i) environment base class, which provides basic task-agnostic functionalities, (ii) environment runner, which establishes the context for how we will use the output tool and action, and (ii) a URDF of the Franka Panda without the tool to indicate where to attach the tool. All task environments are implemented child classes of the environment base class, ensuring they inherit the fundamental functionalities while allowing for task-specific implementations.
A.3.2 DESIGN AGENTS QUERIES
In VLMGINEER, we query VLM for designs by first initializing Ragents number of agents in parallel with the same prompts. For each agent, we prompt it to generate nooool number of tool designs, and Raction number of action waypoint samples that correspond to each tool design. Therefore, the total number of tool-action pairs that are generated via one complete query is Ragents nooot X Paction The prompt we use to specify this behavior to each agent is presented in Appendix A.4.2
We explicitly choose this style of querying to maximize time efficiency and design diversity: (1) time efficiency is achieved by reducing the querying algorithmic complexity by using parallel VLM agents. (2) Empirically, we found that design diversity achieved when we balance dependence and independence between design decisions. Specifically, when a single VLM agent auto-regressively generates Notool X Raction tool-actions pairs, having later design outputs be conditioned on previous design outputs can encourage diversity within that conditional distribution. However, in order to sample from many distinct conditional distributions, as this provides additional diversity, we found that parallel VLM queries that share no history can help with that. Ultimately, found that optimizing time efficiency and design diversity led to better and faster initial samples as well as evolutions.
See Appendix A.3.6 for details on the values we used for these parameters for benchmarking.
A.3.3 DESIGN AGENT OUTPUTS
As part of our prompt to the VLM to query for designs, we specified our desired tool and action formats. For our tool design requirements, please refer to Appendix A.4433 for details. For our action design requirements, please refer to Appendix A..44 for details. Notably, these prompts are separated into with & without the Franka gripper usage. is it also important to note that the number of waypoints the VLM outputs is completely decided by the VLM itself. During VLMGINEER'S 's initial sampling. some design agents are asked to design tools for the gripper, some are not. This ensures the full capabilities of the default morphology are used. The two types of tools are also specified with different required attachment locations: gripper-using tools are asked be attached to the two Franka gripper fingers, and non- cenn--ipper--ssiin tools are asked to be attached to a "virtual joint", which is a joint we set up positioned at the flange of the Franka end effector make the attachment process more standardized.
A.3.4 SIMULATION EVALUATION
From the previous section, we obtain a list of tool-action pairs in the form of URDF designs and action waypoints, respectively. To use these for simulation evaluation, we first merge the tool URDF without modification into blank Franka Panda URDF (a blank Franka URDF will contain gripper if gripper usage is enabled, and otherwise will not). For the action waypoints, which are inherently sparse, we implement linear interpolation for the position trajectory and SLERP (Spherical Linear Interpolation) for the orientation trajectory. The Pybullet simulation then executes these interpolated actions in the designated task environment. Finally, the environment returns result metrics for each run with both choosing evolution candidates and for producing the quantitative evaluation of the design performance. To speed up the evaluation, kein samples are evaluated in parallel Pybullet simulations time.
the corresponding samples. allowing for
20

---

Published as a conference paper at ICLR 2026
A.3.5 EVOLUTION
After evaluating all previous tool-action pairs, we perform selection as follows: (1) For every task, we define two parameters to control the behavior of selection: rewardsave and ktop. (2) Using these parameters, we first select the ktop number of tool-action pairs with the highest task rewards, and then keep only the pairs that have a reward higher than the rewardsave threshold, resulting in a set of winner tool-action pairs. We found this selection mechanism empirically allows for the best signals for evolution.
We then take this winner tool-action pair set and feed it as context into the next design agent query. These previous designs are introduced to the VLM by the "evolution mission introduction prompt" in Appendix 1.4.1 where the VLM asked to perform mutation and crossover on the previous tools via the rules specified in [A.4.7] These evolved design samples will be fed into the simulation for evaluation, and the cycle will continue. We define a final Aiteration parameter to control the number of iterations that this cycle would go for.
See Appendix A.3.6 for details on the values we used for these parameters for benchmarking.
A.3.6 VLMGINEER BENCHMARKING DETAILS
Table 2: Benchmarking Parameters for Different Tasks
Task Name
nagent
Thoool
Faction
kesp
rewardsave
ksim
BringCube
20
10
5
0.6
3
100
Table
20
10
10
5
0.6
3
100
DislodgeCube
20
10
10
5
0.6
3
100
ElevatcPlate
20
5
0.6
3
100
GatherSpheres
20
10
5
0.6
100 3
HighObject
20
10
10
5
0.5
3
100
LiftBox
30
15
15
5
0.1
3
100
MoveBall
20
10
5
0.6
3
100
OneBook
0.4
ScoreGoal
20
10
10
5
0.4
3
100
SnatchCookie
5
5
5
5
0.3
3
100
TurkeyLegs
30
10
15
5
0.2
4
100
When benchmarking VLMGINEER against RoBoTOOL.BENC, we used a different set of parameters for each task, detailed in Table. We used gemin mominni-e --prropreviee 000--003--0 3--2-55 as our VLM model throughout the entire experiment, and ran PyBullet evaluations on an AMD Ryzen 9800X3D 8-Core Processor CPU with 64 GB of RAM. On average, one of VLMGINEER one of these tasks should take around 30 minutes. Below, we explain each hyperparameter.
Naagoot: the number of parallel VLMgineer evolutionary design agents. Each agent independently performs design sampling and evaluation. The top-k designs across all agents are selected and passed to the next evolutionary iteration. We empirically found that employing multiple parallel agents enhances both diversity and runtime efficiency, as a single agent often becomes trapped in local minima and runs more slowly.
Phwat: the number of tool samples generated by each evolutionary agent in each iteration.
"aacioon the number of action samples generated per tool in each iteration by each evolutionary agent. Note that tool and action samples are generated simultaneously in one pass by the VLM, resulting in total of Toool X Tlaction samples per iteration per evolutionary agent.
kiep' the number of top-performing samples selected from all samples generated by all agents at the end of each iteration.
rewardawe: in addition to lippp we apply a minimum reward threshold to determine whether a sample should be retained.
Alttetionn the total number of evolutionary iterations conducted.
kim: the number of simulation evaluation steps performed for each sampled design.
21

---

Published as a conference paper at ICLR 2026
Empirically, observed that using too few agents constrained the diversity of generated tools. Thus, we consciously selected a sufficiently large number of agents within our computational budget. For parameters such koo and rewardave. samples reduces diversity. We further present experimental ablations to illustrate the sensitivity of VIMgineer's performance variations in Raction and Poool samples.
experiments indicated that selecting too few top
Our analysis reveals that VLMgineer's performance consistently improves as the number of generated samples increases. The table below reports the average reward (0.0-1.0) and standard deviation across
4 benchmark tasks for varying sample sizes. Number of samples corresponds to Nagent X Thool X Naction
A.4 FULL PROMPTS
In this section, we provide all VLMGINEER prompts. We show individual prompt components in section A.4.7 We then describe we compose these prompts for different experiments in section A.4.8 A.1.1 We also include a environment code sample input in section A.4.1I For details of their usage. please refer Appendix A.3
A.4.1 MISSION INTRODUCTION
Initial sampling mission introduction prompt:
You are robotics hardware and controls expert. You operate with boldness and brilliance in the physical realm. You work with robot arm that sits in the origin of your environnent. You will be presented with some robotic tasks, and will be asked to design tools and actions to complete the task. Your goal is not to complete the task to perfection in one fell swoep. Instead, your meta-goal is to generate a wide range of differentiated good solutions over time, where one of them will inevitably succeed.
Evolution mission introduction prompt:
You are a robotics hardware and controls expert. You operate with boldness and brilliance in the physical realm. The goal is to create tools and actions to complete given task. You will be given list previously generated tool designs via JSON with URDF. Your geal is to evolve the tool designs via mutation and crossover, and generate the new best actions for the evolved tools. This will be done in way that is similar to genetic algorithms, and will specified in detail in the "Evolutionazy Process" section belcw.
22

---

Published as a conference paper at ICLR 2026
A.4.2 PROCEDURE INSTRUCTION
The procedure you will follow:
1. Receive Envi ronment Descriptions: The user will provide some detailed envirenment descriptiens, robotic task instructions, and initial image of the workspace
from the overhead camera.
Describe the Scene: Analyze the environment. Write down the spatial relationship, including by not limited to the position, orientaticn, dimension, and geometry of all the objecta in the scene. Use all the information provided to you, including all text, code, and images.
3. Create Strategies and Designs: You will need to create Noot tool that you can use to complete the task. For each of the tools you designed, you must generate Paction set of action waypoints that you
to complete the task. Specifically, for total of Rooo times, do the following steps:
(a) First, write down completely different, out-of-the-pox
tool design to tackle the task. Make it unlike any other tool design you made in your other strategies.
(b) Create these tools following the "Tool specification" section below.
(c) For this tool, write the following down: (1) The spatial relationshi (pose transformation
between the end-effector
and each component of the tool; (2) The 3D space that eacn tool component will take up when connected to the robot; The usage of each component of the tool when carrying out the
task.
(d) Use your provious analysis to tweak any obvious issues with the position, orientation, and dimension your tool design.
(e) Next, using your knowledge of the tool and your in depth analysis regarding the intricate 30 spatial relationships
between the tool and its environment, create Naction number of different step by step action plans to enable to effective tool use (See more in "Desired Action Criteria Definitions"). very wary about how objects interact with each other
(f) Transform your step-by-stop action plan into waypoints adhering to the "Action Specifications". During this transformation, think about the inherent nature of controlling robots with waypoint control and the difficulty that may present.
A.4.3 TOOL SPECIFICATIONS
Tool specification prompt without the of Franka Grippers:
23

---

Published as a conference paper at ICLR 2026
(Tool Specifications Your design of the tool must follow these rules: (1) You must only use 3D rectangles for each component; (2) Your tool will be outputted in URDF block format, which should be directly added to the end of panda URDE file, before the robot closing declaration; (3). Make sure your tools weigh very little in the JRDF file, where each tool part should weigh no more than a few grams (these weights do not have to be realistic, it is just for the robot inverse kinematics Your design will be a single rigid tool, which should be attached directly to the "panda "panda_ virtual" link, which you can safely assume to
have easier time converging).
(4)
have the same orientation as the world frame. (5) Any attachments you design shculd geometrically be directly connected to their parent links As a general observation, you perform better when the tools you design are complex and intricate.
the URDF (there should be no gaps in between!) (6)
Tool specification prompt with the use of Franka Grippers:
(Tool Specificatiens Your design of the tool must follow these rules: (1) You must only use 3D rectangles for each component; (2) Your tool will be outputted in URDE block format, which should be directly added to the end of a panda URDF file, before the robot clesing declaration; (3) Make sure your tools weigh very little in the URDF file, where each tool part should weigh no more than a few grans (these weights do not have to be realistic, it is just for the robot inverse kinematics to have easier time converging).. (4) Your design will be a pair of attachments the robot gripper fingers (waich allows the tool to be actuated with the robot grippez); You should attach the left attachment to leftfingr" and the right attachment to (5) Any attachments you design should gecmetrically be di rectly connected their parent links in the
(there should ba no gaps in between!) (6) As general observation, you perform better when the tools you design are complex and intricate.
A.4.4 ACTION SPECIFCCAIIOSS
Action specification prompt without the use of Franka Grippers:
(Action Speci fications) Your tool-using action will be Nx6 numpy array of action waypoints, where is the number of waypoints, and each waypoint angle orientations). per waypoint. Your waypoints will be carried out by the EnvRunner class. It is important to stress this: the action waypoints are controlling the robot end-effector "pandk_virtual" link: this means you have to carefully take into account the dimensions of the tool and the thickness of its parts when designing effective waypoints. Again, you can safely assume the end-effector has the orientation the world frame upon initialization (see frame clarificaticn again details)!
of dimension (xyz position roli-pitch-yaw euler
Your action needs to be precisely six numbers
Action specification prompt with the of Franka Grippers:
24

---

Published as a conference paper at ICLR 2026
(Action Specifications, Your tool-using action will be Nx7 numpy array of acticn waypoints, where N is the number of waypoints, and each waypcint is of dimension 7 (xyz position roll-pitch-yaw culer binary gripper open/close state in integers [0 for open, for closed]). Your action needs to be precisely seven numbers per waypoint. Your waypoints will be carried out by the EnvRunner class. It is important to stress this: the actior waypoints are controlling the robot end-effector "panda_virtual" link: this means you have to carefully take into account the dimensions of the tool and the thickness of its pazts when designing effective waypeints. Again, yeu can safely assune the end-effector has the same crientation the world frame upon initialization (see frame clarification again for details)!
angle orientations
A.4.5 ACTION DIVERSITY SPECIRICATION
(Desired Action Criteria Definitions) For the description below, we will call single sequential set of waypoints in a single rollout as one "actien set". For each toel you created, the goal is to and motion Task success is optimized when an action set able to complete the task successfully. Motion differentiation is optimized zed when there exists a large variance in the motion taken across all action sets you design for the same tool. A large variance motion is defined the tool, the 30 space. Think about how
generate naction action sets that optimize the task differentiation.
each time step, is
located at a different location a tool can be used to interact with the object from many different sides, angles, and ways. When both conditions are met, you have successfully designed good set of actions sets.
A.4.6 FRAME CLARIFICATIONS
(Frame Clarification) the world frame, front/back is along x axis, left/right is along the axis, and up/down is along the 2 axis with the following directions: Positive x: Towards the front of the table. Negative x: Towards the back of the table. Positive y: Towards the left. Negative y: Towards the right. Positive z: Up, towards the ceiling. Negative z: Down, towards the floor. In terms of orientation, starting from the origin frame, Positive retation about the x-axis: tilting the ead-offector hoad to the left. Negative rotation about the x-axis: tilting the end-effecter head to the right. Positive rotation about the y-axis: tilting the end-effector head down. Negative rotation about the y-axis: tilting the end-effector head up. Positive rotation about the z-axis: retating the end-effector head counter-clockvise. Negative retation about the z-axis: rotating the ead-effector head clockwise.
25

---

Published as a conference paper at ICLR 2026
A.4.7 EVOLUTIONARY INSTRUCTIONS
(Rvalutionary Process) Your design decision is part of a tool design genetic algerithm. For each of the Roat tool designs, choose to either mutate crossover. Specitically, tool mutation defined as one change to a single randomly selected previous tcol design. Mutation changes include:
(1) Changing the dimension, location, orientation of single component of the tool.
(2) Adding, removing, or replacing a single component of the tool.
Crossover is defined as the process of combining two randomly selected previous tool designs to create a new tool design. Combination is defined as!
(1) Selecting components from two previous teol designs and combining them to form new tool design.
All mutation and likelihood of task success, yet all decisions must different and diverse.
decisions must potentially increase the
A.4.8 No TOOL INSTRUCTIONS
You are a robotics hardware and controls expert. You operate with boldness and brilliance in the physical realm. You work with robot that sits in the origin of your environment. You will be presented with some robotic tasks, and will asked to design actions to complete the task.
The complete prompt is composed together with instructions from A.4.2 A.4.4
A.4.5
and
A.4.6
A.4.9 HUMAN SPECIFICATION INSTRUCTIONS
You are a helpful robotica hardware and controls expert. You have robot arm that sits in the origin of your envirenment. You working with colleague as team design tools actions for a robot to complete task. Your colleague will provide you with a design and action instructions in the form of natural language instructions. Your goal is to
your colleague's a design and action instructions to output URDF and action waypoints for the ropot to use. You should not use your own knowLedge to design the tool and action, but rather follow your hunan colleague's instruction. Here is the human colleague's prompt: (human_prompt)
The complete prompt is composed together with instructions from
A.4.2
A.4.3
A.4.4
A.4.5
and
A 4.4.6
26

---

Published as a conference paper at ICLR 2026
A.4.10 RLBENCH INSTRUCTIONS
You are helpful robotica hardware and controls expert. You have ropot that sits in the erigin of your environment. You working with colleague as team to design tools and actions for a robot to complete a task. Your colleague will provide you with a design in the format of a URDF, which is attached for you as tool.txt, Your goal is to use your colleagues URD# to come up with an action plan for the robot use.
The complete prompt is composed together with instructions from
A.4.2
A.4.4
A.4.5
and A.4.6
A.4.11 SAMPLE ENVIRONMENT CODE
from envs. envs.base_on base__evv import BascEnv
from models. models.primativee oube import Cube import numpy as np import pybullet
class BringCubeCloserknny (BaseEnv)
def iiiit
self, **wargs,
):
self wtw.s_ttrtttbeeeeeee pos np. np.array ([1.0, 0, 0.035]] delf.target_cube_po np.array ([0.6, 0, 0.035]
super () (innttt_ **kwarg,
15
def _setup_etttt trcru__evvreementtee (self): Load plane
planePos 10, 0, -0. 626]
planeOri - iettuuaterniionrroooultt ([0, 0, np.pi/2]
22
self.plane p.. LoadURDD p"ylane/plane.urree urdd",
planePos, planeOri, useFixedBase-Tuu
channegDDaamis (self. -1, restitution=0. .55
Load table
tablePos 10.6, 0, 62.625]
tableOri p.getquatern rom ((O, 0, 01)
29
self.table p. LoadURDD "taale/table.ure urdÉ", tablePos, tableori, useFixedBase True)
Load cube cubeSize 0.07 sube Cube (cubeSize, sclf.start_cubo_poo -ge__shape ()
36
Load visual goal se foal = p. LoadURDF ("visual www.aaalggaagggaa.aago /rra//.allll.,,
37
self. salffttrget_cubbepees...aaaaa ([0, 0, 0. 0.035]), useri SeerixedBasee-rre
if self blnndeeleeerrrrordeerrr self.blender..recrdeer..c
ote shelf.table, "table")
htw.baccddd..roooeeeeeogogeeeeee (self. goal, "goal")
(self.cube, "cube")
44
def reward fself)
cube pos, p. oetaeeesittaaaiiiiietttttttooeoe current_cistance np. ( caree np. ar.ar (cube_po)), pos), ord=1) initial_distance np. np..i car.mttoe...rrgtt.cuue.....
27

---

Published as a conference paper at ICLR 2026
self. self.start_ pos, ord=1)
Normalize reward to 0 at initial position and 1 at target rormali max (0,
distancce)
54
return normalized_reward
def reset (self)
super () reset ()
Reset cube position
p. Reeeiaeeeessstonnnnrrioitttoooot
self.cube,
self.start_cube.
P. p.gettuuaeerniinnromuuu ([O, 0, 0])
}
A.5 TOOL DESIGN GALLERY
In this tool design gallery, we take the opportunity to display tools from a few tasks that seemed to have allowed VLMGINEER the most creative freedom. These are tool designs that not presented elsewhere in the paper. We believe this illustrates VLMGINEER'S impressive physical creativity and problem-solving capabilities.
Task Name
BRINGCURE
CLEANTABLE
28

---

Published as a conference paper at ICLR 2026
Task Name
ELEVATEPLATE
SPHERES
MOVEBALL
A.6 AN EXAMPLE OF ACTION SEQUENCES
Each action set is 6-(or 7-)dimensional vector representing [x, y. z] position and [roll, pitch, yaw] orientation (and optional gripper command). The number of action waypoints is not fixed; it varies
29

---

Published as a conference paper at ICLR 2026
across trajectories and is fully determined by the VLM, not by any preset specification. An example shown below.
:
0.6, 0.0, 0.4, 0.0, 0.0, 0.0 1,
0.6, 0.0, 0.32, 0.0, -0.2, 0.0
0.8, 0.0, 0.32, 0.0, -0.2, 0.0
5
0.8, 0.0, 0.32, 0.0, 0.2, 0.0 1,
7
0.8, 0.0, 0.7, 0.0, 0.2, 0.0
8
},
A.7
STATISTICAL SIGNIFICANCE ANALYSIS
Franka Grisper
LLM expert
Layparson
Robatics expert
Curs
Reward
BringSube
CleanTable
DistoggeCube
ElavleePiato
SatherSphares
HighCyjed
LiftBor
MoveBall
GheBook
ScoreGoal
SnatshCooki
TurkoyLag
Taak Average
Figure 9: Comparison of the mean and standard deviation of reward generated by VLMGINEER, numan-prompted designs, and Franka Gripper across tasks. Error bars represent standard deviation (clipped to the range [0, 1)).
Fig. 9 presents our primary quantitative results, including standard deviations across 5 runs. Across all 12 tasks. VLMGINBER consistently surpasses all baselines, exhibiting notably low variation between trials. This indicates that VLMGINEER reliably produces high-performing and stable tool-action designs.
In contrast, results obtained from prompted designs not only yield significantly lower per- formance but also show greater variations across runs. We attribute this discrepancy to several factors. First, human-specified tools often require more intricate control strategies; even if capable of completing task, these designs tend to be less resilient to suboptimal or imperfect executions. By comparison, VLMGINEER M-MGGNeER-generrteee
generated tools typically exhibit greater robustness action imperfec- tions. Second, human prompts sometimes suffer from specification ambiguity or misalignment with the VLM. There be discrepancies between human intent and the VLM's s internal representation and physical modeling capabilities. By automating the design process, VI.GINNER avoids these alignment issues, resulting in more effective and precisely realizable solutions.
A.8 VIMGINEER'S COMMON FAILURE MODES
In the VLMGINEER pipeline, common failures include physically infeasible initial tool designs. For example, generated tools sometimes penetrate the environment due to inaccurate simulation physics or intersect with the robot itself at the beginning of the task. More careful prompting and pess-processing help mitigate these. Furthermore, additional practical failure modes emerge when attempting real-worlh deployment, such as tools being too thin for reliable 3D printing and excessively large for single-piece fabrication.
30

---

Published as a conference paper at ICLR 2026
A.9 SIM-TO-REAL TRANSFER PROCESS
We selected three simulated task environments in the RoboToolBench, MoveBall, ElevatePlate, and GatherSpheres, the basis of this real-world experiment. We selected these three environments for how easy they would be to construct their real-world equivalent. For each task in the simulation, we would first attach our custom- aastom-made Franka end-effector mounting head the robot at the location where the tools were specified to be mounted via our prompt. This allows our tool to be directly integrated with the mount, which makes real-world tool attachment simple. After getting winning candidate, we inspect the design and trim it using bounding box. The bounding box will remove the portions that overlap with the attachment. After that, we run script to automatically convert the mount-attached tools to single URDF file and convert it directly to printable STL file using Blender. We import the STL file into our 3D printer's software (Bambu Lab), perform typical 3D printing processing (placing, adding support, slicing), and directly send the job to the 3D printer (Bambu Lab PIS). After printing and removing the supports (if present), since our tools have mounts attached, we directly attach the tool real-world task environment already set up, we simply send the VLMgineer optimized end-effector action waypoints to our Franka position controller for execution. After the action trajectory is carried out, we measure the relevant object states to determine the reward obtained by this tool and action tuple.
Franka Panda robot end effector via screws. With the respective
A. COMPARISON OF DIFFERENT VLMS
In Table and Table 5 we provide the performance comparison across different VLM models. Table 4 compares Gemini-2.5-pro and GPT-o3 on average and best-run reward, and shows that Gemini-2.5 --ro clearly leads over GPT--33; Table comppres the performance across Gemini family, which shows that 2.5-pro the strongest one, while 2.5-flash mid-tier and 2.0-flash trails, and the step from fl5-f to 2.5-pro yields obvious gains.
Table Performance Comparison Across Different VLM Models.
Model
Avg. Reward Top Reward
Gemini-2.5 2.5-pr 0.6054 GPT-o3 0.3775
0.8222 0,5436
Table 5: Performance Comparison Across the Gemini family.
Model
Avg. Reward Top Reward
2.5-pro 0.6054
0.8222
2.5-flash
0.3393
0.4481
2.0-flash
0.0686
0.0796
A.11 COMPARISON OF IMAGE FEEDBACK
To evaluate our design choice of using purely the tool-action reward signal for evolution, we ablated our method against a variation of our method that added each tool-action execution's last frame to the next VLM evolution query as feedback signal. We tested this method variation on three tasks, ElevatePlate, GatherSpheres, and MoveBall, running this and the baseline 500 samples per iteration with 3 evolution iterations. Note that this is total smaller sample size than our main experiment, which used about 2000 samples per iteration. We see in Table the inclusion of visual feedback leads to an small decline 5.4% in average reward. This quantitative result supports our qualitative observation that VLMs often struggle to accurately ground fine-gna inn-grained physical progress from raw visual observations, leading to noisy selection signals that can hamper the evolutionary search.
A. 12 COMPARISON OF SINGLE-SAMPLE ITERATIVE REFINEMENT
To evaluate the effectiveness of our population Dppued evolutionary search, we compare against a simple iterative refinement baseline that represents natural alternative approach for VI.M-driven
31

---

Published as a conference paper at ICLR 2026
Table Ablation study on feedback modality: Comparison of VLMgineer with and without image feedback across three tasks.
Method
Elevate Plate Gather Spheres Move Ball Average
VLMgineer Img Feedback VLMgineer w/o. Img Feedback (Ours)
0.30 0.28
0.45 0.56
0.80 0.82
0.52 0.55
tool action co-design. In this baseline, the VLM maintains single tool design each iteration rather than diverse population of candidates. At each step, the VLM receives the current tool design, its associated action sequence, and the scalar reward feedback same in the proposed evolutionary approach. The VLM then directly refines the design based on this feedback, generating an improved tool and N=10 action plan. Unlike our approach, which leverages population diversity through mutation and crossover operators to explore the design space broadly before converging on solutions, iterative refinement baseline performs greedy local search from a random initial guess, producing lower performance. We tested the baseline in three tasks: FlevatePlate, GatherSpheres, and MoveBall, with each running refinement for 30 times. The running procedure for this baseline is exactly the same as main proposed method. We use the same setup for proposed method in the image feedback ablation A. This experiment shows that our evolutionary approach outperforms iterative prompt refinement. Iterative refinement is easy stuck local minimum with bad initial proposal while evolutionary approach could expand exploration, leading to higher performance.
Table 7: Ablation study on evolutionary vs. single sample iterative prompt refinement
Method
Elevate Plate
Gather Spheres
Move Ball
Average
Iterative Refinement 0.0 0.28
0.07 0.56
0.76 0.82
0.28
Evolutionary (Ours)
0.55
A.13 LICENSES
The cardboard box asset used in LIFTBOX environment is from PartNet-Mobility Dataset Xiang et al. (2020). Their terms of use are stated here: (sapien. edu/about
The book assets and the book holder used in the ONEBoOK environment came from the YCB Dataset Calli et al. (015)) This dataset is under the CC BY 4.0 license.
The goal frame and net assets used in the SCOREGOAL environment came from the Meta-World Benchmark Yu et al. (019). This benchmark is under the MIT License.
The transparent jar asset used in the SNATCHCOOKIE environment came from cgtrader] 3D CAD model website. This asset is under the "Royalty Free No Ai License", detailed here
The cookie assets used the SNATCHCOOKIE environment came from sketchfab 3D CAD model website. This asset is under the CC BY 4.0 license.
The turkey leg assets used in the TURKEYLEGGS environment came from sketchfall 3D CAD model website. This asset is under the CC BY 4.0 license.
A. THE USE OF LARGE LANGUAGE MODELS (LLMS)
We mainly use L.LMs to polish our writing. The use LLMSs this work could not regarded as contributor.
32


Reviews:
[{"review_id": "MTAbjAgkEI", "reviewer": "Reviewer_Cg95", "summary": "The paper proposes a fully automatic framework, VLMgineer, that designs both tools and corresponding actions from scratch by drawing on the generative capabilities of Vision–Language Models (VLMs) together with evolutionary search. The system jointly discovers tool geometries and manipulation strategies to solve tool-use tasks, and it is evaluated on a custom benchmark (RoboToolBench) of everyday manipulation scenarios where creative tool design is considered necessary. While the idea of co-optimizing tool design and use is conceptually interesting, the practical benefits and generalizability of the approach remain unclear. The experimental setup lacks strong baselines, and the model relies too heavily on task-specific and highly detailed inputs, which limits its applicability and generalization to broader scenarios.", "questions": "In addition to the issues presented in the weakness section: \n\n- The paper compares VLMgineer’s co-designed tool and action against the original, pre-existing tools from RLBench. Since the RLBench tools have established, fixed designs and VLMGINEER can co-optimize both tool and action, is this a fair comparison? Specifically, the robot cannot improve the pre-established tools in the RLBench environment setup, which limits the optimization space compared to VLMGINEER's full co-design approach. Could the authors clarify the intent behind this specific comparison?  \n\n- The VLMgineer framework takes \"System Instructions\" as input, alongside the raw environment code and task description. For clarity and to better assess the generalizability of the VLM prompt, what exactly do these system instructions entail? Given that the model already leverages a Vision-Language Model, wouldn't it be more generalizable if the inputs were ideally limited to high-level, natural inputs like the environment image and a simple task goal to maximize generalizability and reusability?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Using evolutionary strategies to generate tools, rather than a pure VLM solution, is a strength that introduces an iterative refinement process, helping to generate diverse and more functional designs.   \n\n- RoboToolBench provides a set of distinct environments tailored for evaluating the performance of tool-use strategies.  \n\n- The method yields more effective designs and improved performance for the given task compared to human-prompted designs.", "weaknesses": "- The tools are designed as permanent end-effector attachments (or new grippers), rather than as distinct, portable tools that would be grasped by the robot. This design choice fundamentally limits the system’s generalizability and applicability to real-world scenarios where a robot must use a variety of pre-existing, non-integrated tools.  \n\n- The overall evaluation lacks sufficient diversity and strength in its baselines. The comparison to the default two-fingered Franka Panda gripper is weak, as these tasks are explicitly designed to be unsolvable without custom tools. While the work includes a comparison to RLBench tools on four tasks, it omits comparisons against recent, more sophisticated computational co-design or tool-learning baselines (e.g., one of the studies reported in the Related Work section), which would provide a more robust validation of VLMgineer's effectiveness. The comparison against a small-sample group of three human designers is also subjective.  \n\n- The VLM requires low-level and environment-specific information, including raw environment code, task description, and explicit system instructions as input (Figure 2). This reliance on low-level, internal data severely limits the method's generalizability and reusability, as a truly general-purpose system should ideally operate with high-level, natural inputs like an environment image and a simple task goal.  \n\n- The authors claim to have empirically observed their action sampling strategy outperforms other forms of feedback (line 212); however, no quantitative analysis or representative baseline is provided in the paper to substantiate this critical statement.  \n\n- The authors claim that optimizing both the tool’s design and its manipulation strategy is a more practical solution than using already existing tools in the environment for a general-purpose robot (line 111). However, this assertion is questionable regarding real-world practicality, scalability, and generalization. Continuously generating and 3D printing new tools for each task is costly and time-consuming (line 437), limiting its applicability. An approach focused on learning when to use existing tools and when to generate/form new ones  would contribute more significantly to generalizable robotics and open-world applications.  \n\n- For clarity and reproducibility, the paper should include an example of the generated action sequences (the 7 waypoints) and a discussion of their corresponding usage and limitations.  \n\n- It would be better to present the algorithm in the main paper rather than in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Zh2mZTvbS", "reviewer": "Reviewer_UWYg", "summary": "This paper presents VLMGINEER, a system that uses a VLM to invent new tools for robots. It uses an \"evolutionary search\" process where the VLM generates, tests, and improves tool designs and the robot actions needed to use them. The main idea is to create tools that make the robot's task feasible or easier. The paper also introduces ROBOTOOLBENCH, a new 12-task simulation benchmark for this problem. Results show this method performs better than a standard robot gripper and tool designs specified by human experts. The authors also successfully transferred three of the simulated designs to a real robot.", "questions": "- The sim-to-real transfer was open-loop. How robust are these policies to minor perturbations in the real-world setup (e.g., a 1cm shift in object position)?\n- The action representation is an $N \\times 7$ array of waypoints. How is $N$ (the number of waypoints) determined? Is it a fixed hyperparameter per task, or does the VLM decide on the number of waypoints? This seems like a critical parameter that would heavily influence the performance.\n- For example, how often does the VLM produce an invalid URDF, a non-manipulable tool, or a design that violates a constraint? How does the pipeline handle such errors?\n- Number of trials for each human subject, what information is presented, what training the participants received, and how the participants were recruited?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper studies the challenging problem of automatic tool-action co-design. This is a significant and creative step beyond typical VLM-based planning, which generally assumes a set of pre-existing tools. This task requires a high degree of physical common-sense reasoning, and the paper demonstrates that VLMs can provide useful priors for this.\n\n- The proposed method is well-explained. It effectively uses the VLM's semantic and physical knowledge to guide the search, successfully automating the process that would typically require significant hand-engineering.\n\n- The ROBOTOOLBENCH test suite is a valuable contribution. Its 12 tasks are well-design and could be useful for future research.", "weaknesses": "- The framework's reliance on thousands of simulation-based evaluations for its evolutionary loop (e.g., 8000 samples for the ablation ) makes the full design process impractical for direct real-world deployment. The paper demonstrates sim-to-real transfer on only 3 of the 12 tasks. More critically, these transfers rely on open-loop execution of discrete waypoints, which is known to be brittle and highly sensitive to any real-world physics or calibration error. This remains a major gap between the simulated success and a real-world \"robotic toolsmith.\"\n- The set of baselines is a significant weakness. The primary comparisons are against \"Human Prompts\" and existing \"RLBench Tools\". While outperforming the \"Human Prompts\" baseline is interesting, it mainly proves that evolutionary search is superior to a single-shot (human + VLM) guess. The related work (Section 2) extensively discusses prior computational co-design methods, including those using RL , differentiable simulation , and model-based optimization. Liu et al. (2023) is even cited as inspiration for a task. However, none of these state-of-the-art computational methods are included as quantitative baselines. Without them, it is impossible to situate VLMGINEER's performance relative to the actual prior art in computational tool design.\n- The \"Human Prompts\" baseline, which is central to the paper's claim of outperforming human specifications, is not a rigorous study. The appendix (A.1.1) reveals this was a \"case study\" with only three participants: one \"LLM expert,\" one \"robotics expert,\" and one \"layperson\". This sample size is far too small to draw any generalizable or statistically meaningful conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "33PV5sMd5q", "reviewer": "Reviewer_gsFY", "summary": "This paper introduces VLMGINEER, an autonomous framework that uses vision–language models (VLMs) together with evolutionary search to co-design both robotic tools and action policies from scratch. Unlike prior work that assumes a fixed toolset or relies on human-specified design parameters, VLMGINEER automatically generates tool geometries (in URDF) and corresponding action waypoints, iteratively refining them through simulation-based evaluation. To benchmark this setting, the authors propose ROBOTTOOLBENCH, a suite of 12 manipulation tasks that are infeasible using a standard gripper. Experiments show that VLMGINEER consistently outperforms human-designed tools, VLM-generated tools from human prompts, and RLBench baselines, with strong improvements in both success rates and reward metrics. The framework also demonstrates successful sim-to-real transfer on a physical Franka robot.", "questions": "- **Quantitative benefit of joint tool–action sampling.**  \n  Have you evaluated whether jointly sampling tool geometry and actions in a single VLM pass outperforms a sequential approach (i.e., generating tools first, then actions with the VLM)? Providing such a comparison would help validate this core design decision.\n\n- **Justification for action representation.**  \n  Given that VLMs often struggle with spatial grounding and precise low-level control, what motivated the choice of directly predicting waypoint sequences instead of using motion primitives or parameterized action abstractions? Please clarify this design choice or include ablation experiments.\n\n- **Separating tool-quality vs. action-quality effects.**  \n  Since execution relies on VLM-generated waypoints, how do you distinguish failure cases caused by suboptimal tool design from those stemming from imperfect action generation? Disentangling these factors is important for interpreting the contribution of the evolutionary pipeline.\n\n- **Clarity in Figure 6.**  \n  The meaning of the light blue and light orange shaded regions is not obvious from the figure or its caption. Please provide a clearer explanation of what these regions represent.\n\n- **Evolutionary search vs. simpler iterative refinement.**  \n  Have you compared your evolutionary algorithm against a simpler iterative prompting baseline (e.g., repeatedly asking the VLM to refine the tool design without population-based evolution)? Such a comparison would help isolate the specific advantages of evolutionary search.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Novel use of inductive in-context crossover and mutation:**  \n  The paper introduces an interesting concept of using VLMs to perform crossover and mutation entirely in-context without predefined operators. This is a creative and promising perspective that differentiates the approach from traditional evolutionary robotics.\n\n- **Clear demonstration of tool-design capability:**  \n  The proposed framework is able to zero-shot generate both tool geometry and corresponding action strategies, showcasing VLM-based physical reasoning beyond conventional policy learning.\n\n- **Benchmark contribution:**  \n  ROBOTOOLBENCH could serve as a useful benchmark for future research on tool co-design and automated physical reasoning.", "weaknesses": "- **Citation formatting reduces readability.**  \n  The manuscript embeds citations directly in narrative form, which disrupts flow and clarity. Using standard citation commands (e.g., `\\citep{}`) would improve readability and maintain academic consistency.\n\n- **Missing key related work: *RobotSmith*.**  \n  The paper does not reference *RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills*, one of the closest prior works on tool co-design. Including this would strengthen the literature coverage and contextualize the contribution.\n\n- **Overstated novelty claims.**  \n  The claim that “VLMGINEER leverages the surprising physical creativity of VLMs...” suggests first-mover novelty. However, works such as *Evolution 6.0: Evolving Robotic Capabilities Through Generative Design* have similarly explored creative, generative physical design via LLMs. The authors should clarify what is genuinely new here and frame contributions accordingly.\n\n- **Limited evaluation on long-horizon tool-use tasks.**  \n  ROBOTOOLBENCH primarily evaluates single-stage manipulation tasks. Including long-horizon, multi-step tool-use scenarios with meaningful constraints would better demonstrate reasoning depth and practical utility.\n\n- **Ambiguity in Table 1 metrics.**  \n  It is unclear whether reported values represent normalized reward, success rate, or another metric. Clearly defining the evaluation metric and explaining its interpretation would improve transparency.\n\n- **Lack of manufacturability considerations.**  \n  Although real-world results are included, manufacturability constraints are not integrated during tool generation. Some designs (e.g., for *ScoreGoal* and *BringCube*) appear unnecessarily complex compared to human-designed solutions. Incorporating manufacturability into the optimization process would improve real-world relevance.\n\n- **Baseline design comparison could be expanded.**  \n  Current baselines rely primarily on human-prompted tool designs, which may introduce expertise-dependent bias and variance. Including recent LLM-based baselines—such as *RobotSmith* or simple LLM-generated tool-and-action plans without evolutionary refinement—would provide a stronger and clearer comparison, better contextualizing the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHYApKBJL6", "reviewer": "Reviewer_5kyU", "summary": "This paper presents VLMgineer, a VLM-aided evolutionary algorithm that jointly optimizes tool design and action strategy to solve challenging manipulation tasks beyond the capabilities of conventional robot morphologies. It is validated that the inherent physical creativity of vision-language models, when paired with iterative evaluation and refinement, can yield interesting tool-action pairs that outperform human-designed counterparts. The paper also introduces a comprehensive simulation benchmark, RoboToolBench, for evaluating tool design performance, which could facilitate further investigation.", "questions": "1.Could the authors describe how much does the proposed approach rely on the spatial reasoning capabilities of VLMs, and how much on the privileged geometric information, in order to directly generate action waypoints? \n\n2.In many cases expert knowledge could still serve as useful informative priors. I wonder whether the authors have tried utilizing the human-specified or RLBench tools as an initial population and further evolve them with VLMgineer. Would this yield additional performance gains, or instead impose human biases that constrain more creative search?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper addresses an important research problem, i.e. free-form evolutionary optimization of tool design to enhance the capabilities of robotic systems in challenging scenarios. An innovative VLM-based approach is presented that combines the physical creativity of VLMs with evolutionary computation to yield high-performing tool-action pairs. \n\n2.The paper is well-written and easy to follow. The abundant illustrations and qualitative results greatly help with readers’ understanding. \n\n3.Full prompts and implementation details are provided to ensure reproducibility. The authors also provide an actionable procedure for transferring the co-optimized solutions onto physical robot arms, increasing real-world practicality.", "weaknesses": "1.In the proposed method, the VLM is prompted to directly generate discrete action waypoints, instead of using reinforcement learning. From my understanding, this would more or less rely on the privileged geometric information provided in environment source codes (such as object positions and orientations). The authors could discuss in more detail the specific information provided regarding environment setup, and what is the implication and generalizability of such control strategies for less structured, real-world settings where only images and textual instructions are available. \n\n2.The tool use setting considered in this work is somewhat simplified, where the tool is attached to the robot arm and fixed all along. Since in this case the robot morphology and tool could be considered as a whole, “augmented” morphology, the authors are suggested to discuss what are the unique challenges to be solved for this particular problem setup, compared with previous morphological design studies that also leverage large models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)