Learning to Design and Use Tools for Robotic Manipulation
Authors: Ziang Liu
Stephen Tian
Michelle Guo
Karen Liu
Jiajun Wu
Venue: CoRL 2023 Poster (2023)
Keywords: tool use
manipulation
design

Abstract: When limited by their own morphologies, humans and some species of animals have the remarkable ability to use objects from the environment toward accomplishing otherwise impossible tasks. Robots might similarly unlock a range of additional capabilities through tool use. Recent techniques for jointly optimizing morphology and control via deep learning are effective at designing locomotion agents. But while outputting a single morphology makes sense for locomotion, manipulation involves a variety of strategies depending on the task goals at hand. A manipulation agent must be capable of rapidly prototyping specialized tools for different goals. Therefore, we propose learning a designer policy, rather than a single design. A designer policy is conditioned on task information and outputs a tool design that helps solve the task. A design-conditioned controller policy can then perform manipulation using these tools. In this work, we take a step towards this goal by introducing a reinforcement learning framework for jointly learning these policies. Through simulated manipulation tasks, we show that this framework is more sample efficient than prior methods in multi-goal or multi-variant settings, can perform zero-shot interpolation or fine-tuning to tackle previously unseen goals, and allows tradeoffs between the complexity of design and control policies under practical constraints. Finally, we deploy our learned policies onto a real robot. Please see our supplementary video and website at https://robotic-tool-design.github.io/ for visualizations.

Learning to Design and Use Tools for Robotic Manipulation
Ziang Liu", Stephen Tian", Michelle Guo, C. Karen Liu, Jiajun Wu
Department of Computer Science Stanford University, United States
ziangliu, tians@stanford edu
Abstract: When limited by their own morphologies, humans and some species of animals have the remarkable ability use objects from the environment to- ward accomplishing otherwise impossible tasks. Robots might similarly unlock range of additional capabilities through tool use. Recent techniques for jointly optimizing morphology and control via deep learning are effective at designing lo- comotion agents. But while outputting a single morphology makes sense for loco- motion, manipulation involves a variety of strategies depending on the task goals at hand. manipulation agent must be capable of rapidly prototyping specialized tools for different goals. Therefore, we propose learning designer policy, rather than single design. A designer policy is conditioned task information and outputs tool design that helps solve the task. A design-conditioned controller policy can then perform manipulation using these tools. In this work, we take a step towards this goal by introducing a reinforcement learning framework for jointly learning these policies. Through simulated manipulation tasks, we show that this framework more sample efficient than prior methods multi-goal or varianv settings, can perform zero-shot interpolation or fine-tuning to tackle previously unseen goals, and allows tradeoff's between the complexity of design and control policies under practical constraints. Finally, we deploy our learned policies onto a real robot. Please see our supplementary video and website at bebesignngittee github..ii... io/ for visualizations.
Keywords: tool use, manipulation, design
Introduction
Humans and some species of animals are able to make use of tools to solve manipulation tasks when they constrained by their own morphologies. Chimpanzees have been observed using tools to access food and hold water [1], and cockatoos are able create stick-like tools by cutting shapes from wood [2] with their beaks. To flexibly and resourcefully accomplish range of tasks comparable to humans, embodied agents should also be able to leverage tools.
But critically, while any object in a human or robot's environment object is directly a useful aid for the task goal hand. How can robotic system also acquire the extraordinary ability of animals to create appropriate tool to help solve task after reasoning about scene's 's physics and its own goals? In this work, investigate not only how agents can perform control using tools, but also how they can learn to design appropriate tools when presented with a particular task goal, such as target position or object location.
potential tool, not every
Prior works have studied joint learning of agent morphologies and control policies for locomotion tasks [3, 4, 5, 7J. However, these approaches optimize designs for a single, predefined generic goal, such maintaining balance or forward speed. In this work, take step towards agents that can learn to rapidly prototype specialized tools for different goals or initial configurations.
"indicates equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

We propose tackling this challenge by learning de- signer and controller policies, that are conditioned on task information, solely from task rewards via reinforcement learning (RL). We find that when trained with a multi-stage Markov decision process (MDP) formulation, these policies can be efficiently learned together in high-dimensional combined space. We frain agents on multiple instantiations of each task so that they can learn to produce and ma- nipulate designs best suited to each situation.
Figure 1: A robot may need to use different tools to fetch out-of-reach book (blue) push into the bookshelf (pink). should rapidly prototype the tool it needs.
We argue that there are three important properties of an embodied agent that designs and uses tools in re- alistic settings. Firstly, it must be able to learn design and control policies without explicit guidance, using signals specified only on task progress. Fur- thermore, it should form specialized tools based on the task goal at hand, as motivated by Figure 1. Finally, it should adjust to real-world constraints, rather than creating infeasible designs.
Our main contribution is learning framework for embodied agents to design and tools based on the task at hand. We demonstrate that our approach can jointly learn these policies in sample- efficient manner from only downstream task rewards for a variety of simulated manipulation tasks, outperforming existing stochastic optimization approaches. We empirically analyze the generaliza- tion and few-shot finetuning capabilities of the learned policies. By introducing a tradeoff parameter between the complexity of design and control components, our approach can adapt to fit constraints such as material availability or energy costs. Finally, we demonstrate the real-world effectiveness of the learned policies by deploying them on real Franka Panda robot.
2 Related Work
Computational approaches to agent and fool design. Many works have studied the problem of optimizing the design of robotic agents, end-effectors, and tools via model-based optimization [8, 91. generative modeling [10, 111, evolutionary strategies [3, 5]. stochastic optimization [12], or reinforcement learning [13]. Li et al. [14] use differentiable simulation to find parameters of a tool that are robust to task variations. These methods provide feedback to the design procedure by executing pre-defined trajectorics or policies, or performing motion planning. In contrast, we aim to jointly learn control policies along with designing tool structures.
In settings where the desired design is known but must be assembled from subcomponents, geome- try [15] and reinforcement learning [16] have been used to compose objects into tools. In this work, we address the fabrication stage of the pipeline using rapid prototyping tools (c.g. 3D printing).
Learning robotic tool use. Several approaches have been proposed for empowering robots to learn to use tools. Learning affordances of objects, how they can be used, is common paradigm [17, 18, 19, 20]. Noguchi et al. [21] integrate tool and gripper action spaces in Transporter-sty framework. Learned or simulated dynamics models [22, 23, 24, 25] have also been used for model- based optimization of tool-aided control. These methods assume that helpful tool is already present in the scene, whereas we focus on optimizing tool design in conjunction with learning manipulation, which is a more likely scenario for a generalist robot operating for example in a household.
Joint optimization of morphology and control. One approach for jointly solving tool design and manipulation problems is formulating and solving nonlinear programs, which have been shown to be especially effective at longer horizon sequential manipulation tasks [26, 27]. In this work, we aim to apply our framework to arbitrary environments, and so we select a purely learning-based approach at the cost of increasing the complexity of the search space.
Reinforcement learning and Bayesian optimization (BO) [28] based approaches have also been ap- plied to jointly learn morphology and control. These include policy gradient methods with either separate [29] weight-sharing [30] design and control policies or methods that consider the agent
2

---

design as computational graph [31]. Evolutionary or BO algorithms have been combined with control policies learned via RI. [32] to solve locomotion and manipulation tasks. Luck et al. [4] use an actor-critic RL formulation with a graph neural network (GNN) value function for improved sam- ple efficiency. Pathak et al. [33] learn to design modular agents by adding morphology-modilyye actions to an MDP. Yuan et al. [7] provide generalized formulation with multi-stage MDP and GNN policy and value networks. These methods have demonstrated promising performance lo- comotion tasks. In this work, we focus on developing tools for manipulation, where the challenge is learning designer and controller policies that can create and operate different tools depending on the task variation at hand, and it is less feasible to use actuated joints as design components,
3 Problem Setting
Our framework tackles learning tool design and use for agents to solve manipulation problems, without any supervision except for task progress. We represent the agent's environment as a two- phase MDP consisting of the "design phase" and "control phase". We use environment interactions from both phases to jointly train designer policy and controller policy.
At the start of each episode, the environment begins in the design phase, visualized at the top of Figure 2. During the design phase, the action Od E Ap specifies the parameters of the tool that will be used for the rest of the episode. The environment state Sy E Stask consists of vec- tor of task observations: the positions and ve- locities of objects in the scene and the Cartesian end-effector position of the robot if present.
Designer Fullay An
Toal design dii
Iritial state-fg
god ulservading
Contrel Policy
Control
Oleervation di. Rewadd
After a single transition, the MDP switches to the control phase, illustrated in the lower half of Figure 2. During the control phase, the ac- tions a. E AA represent inputs to position or velocity controller actuating the base of the pre- viously designed tool. The control phase state is the concatenation of the task observation 8t E Stask and the previously taken design action d.- That is, the control state space Sc = Stask X Ap. The agent receives rewards r; at each timestep t based on task progress (e.g., the distance of an object being manipulated to the target position). The control phase continues until the task is solved or time limit is reached, and the episode then ends.
Figure 2: Solving a task using learned designer and controller policies. During the design phase, the de- signer policy observes the task at hand and outputs the parameters for tool. In the control phase, the con- troller policy outputs motor commands given the tool structure, task specification, and observation.
To learn design and control policies for multiple goals, we condition our policies on supplied parameter from a goal or task variation space G. In this paper, we choose goals that represent the final position of an object to be manipulated or the number of objects that must be trans- ported. The objective is then to find the optimal goal coal-conditioned designer and controller policies Th (a]s, g). (als,,) g) that maximize the expected discounted return of a goal-dependent reward function R(s,a.g).
Instantiating Our Framework
Next, we concretely implement our framework toward solving a series of manipulation tasks. Specif- ically, we select tool design space, policy learning procedure, and auxiliary reward function.
Tool design space. The design space significantly impacts the difficulty of the joint design and control optimization problem. When the set of possible designs is large but many of them are unhelpful for any task, the reward signal for optimization is sparse. Thus, we would like to select a design parameterization that is low-dimensional but can also enable many manipulation tasks. Furthermore, we prefer designs that are easy to deploy in the real world.
Toward these goals, we consider tools composed of rigid links. While simple, find that this pa- rameter space includes tools that are sufficient to help solve variety of manipulation tasks. They
3

---

can also be easily deployed in the real world on soft robots [12] or through rapid fabrication tech- niques like 3D printing. However, we note that our framework not limited to this design space.
For example, the parameterization used in our 2D environments consists of three links attached end- to-end as shown in Figure 2, where tool is represented by a vector [[ 12,42, 0,, O2] IR An, where / represents each link length and represents the relative angle between the links.
Policy learning. Similarly to Yuan et al. [7], we interactively collect experience in the environment using the design and control policies, where each trajectory spans the design and control phase. We then train the policies jointly using proximal policy optimization (PPO) [34], a popular policy gradient method. We adopt the graph neural network (GNN) policy and value function architecture from Yuan et al. [7]. When training in goal-conditioned environments, supply the policies with randomized goals sampled from the environment for each interaction episode.
Auxiliary reward. An embodied agent that creates and uses tools in the real world must also consider resource constraints. Many prior co-optimization procedures assume that actuated joints or body links can be arbitrarily added the agent morphology. However, as an example, when an agent solves manipulation tasks in household environment, it may not have access additional motors building materials. On the other hand, when possible, constructing larger tool may reduce the amount of power expended for motor control, especially if a task must be completed many times.
We enable our framework to accommodate preferences this trade-off between design material cost and control energy consumption, proxied by end-effector velocity, using parameter a that adjusts an auxiliary reward that is added the task reward at each environment step:
Timddeof K [(1 (a dused
(dddueed + (1 - a.) Cused
ddax
Crax
where is a scaling hyperparameter, a E [) controll the emphasis on either the control or design component, dased and dmax represent the utilized and maximum possible combined length of the tool components in the design respectively, and Cused and Gmax represent the control velocity at the current step and the maximum single-step control velocity allowed by the environment. Intuitively the agent favors using less material for tool construction when is large, and less energy for the control policy when is small. Except in Section 5.3, we use K 0 to isolate this reward's 's effects.
Experiments
For our experiments, we introduce three 2D manipulation environments in the Box2D simulator [35] and three 3D environments PyBullet [36]. These tasks showcase the ad- vantages of using different tools when there does not exist single tool that can solve all instances. The six tasks are shown in Fig- ure 3. For each task, we initialize the de- signed tool by matching a fixed point on the tool to a fixed starting position regardless of the goal. During the control phase, we sim- ulate scenario in which robot has grasped the tool and manipulates it, via end-effector velocity control in 2D or position control in
(a) Puah
(b) Catch balls
(c) Scoop
Fetch cube (c) Lift cup (f) Scoop (3D) Figure 3: Simulated manipulation environments.
3D. All 2D tasks use the 3-link chain parameterization described in Section 4. A short description of each task is as follows, with additional details in the Appendix:
Push (2D): Push a round puck using the tool such that stops at the specified 2D goal location.
Catch balls (2D): Use the tool to catch three balls that fall from the sky. The agent's 's goal is to catch all three balls, which start from random locations on the x-y plane.
Scoop (2D): Use the tool scoop balls out of reservoir containing 40 total balls. Here specify goals of scooping {1, 2..,, 7} balls.

---

Push
6 random Catch bells
Scoop
3.0
3.5
3.0
2.5
5.0
2.0
2.0
E.0
1.5
1.5
3.0
0.5
0.5
0.0
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4,0
0.0
0.5
1.0
1.5
2.0
Steps
1e7
Steps
1e7
Steps
1c7
Fetch cube
Lift cup
Scoap (3D)
7.5
6.0
netum
6.0
4.0
4.0
2.0
2.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5 Stops
3.0
3.5
4,0
0.0
0.5
1.0
1.5
2.0 Stops
2.5
3.0
3.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
le7
166
Steps
1e7
Ours
Ours (shared arch.)
CMA-RL
HWasP-minimal/Ha 2018
CMA-ES
Figure 4: Learning curves for framework, prior methods, and baselines. Across all tasks, framework achieves improved performance and sample efficiency. Shaded areas indicate standard seeds on methods, except the scoop (3D) task where we use 3 seeds due computational constraints.
Fetch cube (3D): Use the tool to retrieve an object randomly positioned beneath an overhang. This task is challenging because the end effector is restricted 0.8m X 0.2m region of the z-y plane avoid collision with the overhang. The tool is three-link chain where each link is box parameterized by its width, length, height, and relative angle to its parent link.
Lift cup (3D): Use the tool to lift a cup randomized geometry from rest to a certain height. This task requires careful tool design match cup geometry. The tool is four-link fork with two prongs symmetrically parameterized by their separation, tilt angle, width, length, and height.
Scoop (3D): An analog of the 2D scoop task (with the same goal space), but the tool in 3D is a six-link scoop composed of a rectangular base plate parameterized by its length and width, and four rectangular side plates attached to each side of the base plate, parameterized by their height and relative angle to the bottom plate. A dimenddinnsionn handle is attached one side plate.
In our experiments, we analyze whether the instantiation of our framework on these manipulation tasks has the following four desirable properties:
Can our framework jointly learn designer and controller policies in sample-efficient manner, using just rewards based on task progress?
Do learned designer and controller policies generalize or enable fine-tuning for unseen goals?
Can the adjustable parameter enable agents to trade off design and control complexity?
Can designer and controller policies learned by our framework be deployed on a real robot?
5.1 Evaluating sample efficiency
Sample efficiency is critical for a performant joint tool design and control learning pipeline, as many sampled designs will be unhelpful for any goal or initialization. As prior joint optimization works do not handle situations where diverse designs may be produced depending on the particular task variation, compare to the following prior methods and baselines (details Appendix B.2):
Bi-level optimization (CMA-RL): This follows the common bi-level paradigm for joint opti- mization [4]. We use CMA-ES [37] to perform outer-loop stochastic design optimization and learn design--onditioned policies with PPO in the inner loop, as the reward signal for design.
Hardware-as-Policy Minimal [31]/Ha [30]: The variant of HWasP without differentiable physics assumptions (HWasP-nn (HWasP-minimal) and the method from Ha [30] both perform policy-gradicn optimization of single set of design parameters together with the control policy jointly using RL.
5

---

0.7
10
10
0.8
Return
6-
cutout
fine tune from scratch
0.35
outer
81 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Rato
Stopa
(a) Initialization ranges and zero-shot per- formance when cutting out 60% the area of the entire possible training region.
(b) Returns for policies trained with varying relative cutout region area.
(c) Fine-tuning performance compared to learning from scratch across target goals.
Figure 6: We find that our policies can solve instances of fetch cube task unseen at training time either zero-shot or by rapid finetuning. In (a), we plot the goal regions along with zero-shot policy performance. Areas within the dotted yellow borders and outside the teal region are unseen during training. The region within the teal border (but outside cutout regions) is the training region. Training curves averaged over 3 seeds; shaded regions show standard emror.
Single-trajectee CMA-ES baseline: We optimize a set of design and control actions for an episode independent of the goal or starting task state, demonstrating the required policy reactivity. Ours (shared arch.): An ablation of our framework that uses single policy network with design and control heads, demonstrating the importance of separate designer and controller architectures.
In Figure 4, we compare the learning curves of our method and demonstrate results for all six tasks. Our method strongly outperforms the prior methods and baselines, achieving superior final performance in fewer samples. It does by producing specialized tools to solve each task instance, while the other methods optimize for single design across all task variations. The ablation shows the importance of learning separate designer and controller policies.
We present qualitative examples of tool designs out- putted for different goals the push task in Fig- ure 5. The designer policy outputs range of tools depending on the goal location.
Tool designs outputted by single learned designer policy for the push task the goal position, in green, varies. The range out- putted designs enable the agent to push the ball in the desired direction.
Figure 5:
5.2 Generalization to unseen task variations
In simulation, agents can experience millions of trials for range of task variations. However, when deployed to the real world, designer and controller policies cannot be pre-trained on all possible future manipulation scenarios. In this section, we test the ability of our policies to generalize to task variations unseen during training. For these experiments, we focus on the fetch cube task because has an initial pose space that be manipulated in semantically meaningful way. We train policies using our framework by removing region of the space, which we call the "cutout" region. (see Figure 6a). Then, we evaluate the generalization performance of learned policies on the initial poses from the cutout region and outside the training region in two scenarios.
subset of initial poses from the entire initial pose space
Zero-shot performance. In the first scenario, we test the ability of the designer and controller policy tackle previously unseen initial pose directly. Using policy trained the initial pose space with cutout region removed, evaluate the zero-shot performance unseen initial poses.
In Figure 6a, we visualize the zero-shot performance of our design and control policies on initial poses across the entire environment plane, finding that our policies are able to solve even task varia- tions outside the training region boundaries. We also analyze how decreasing the number of possible training poses affects generalization performance. In Figure 6b, show the returns over the train- ing region, cutout region, and the regions outside of the training region, as the size of the cutout region for training poses varies.

---

Figure 8: Real world rollouts for the fetch cube (top) and 1ift cup (bottom) tasks, For fetch cube, we show the task success threshold in green and the volume covered the overhang yellow. The design policy generates specialized tools, and the control policy adjusts its strategy to the tool to complete each task.
When the cutout region is very small, the performance of our learned policies on seen and unseen poses is similar. As the area the cutout region increases, the performance on unseen goals degrades gracefully and can still solve a significant portion of unseen tasks.
Fine-tuning performance. Sometimes, task variations cannot be achieved zero-shot by designer and controller policies. In this section, we test whether our design policies and con- troller policies can still serve as good instantia- tions for achieving these variations. We test this by pre-training policies with our framework the entire training region and fine-tur tne-tuning them to solve initial poses outside that region.
1.0
a- 0.7
452
133
0.0
Control/Design ratio (b) Tools produced at dif- with different a.
(a)
In Figure 6c, we show the results of the fine- tuning experiment. We find that even for poses that are far away from the initial training region, policies are able learn to solve the task within handful of gradient steps, and is much more effective than learning from scratch.
ferent values.
Figure 7: Examples of tools generated by varying the tradeoff parameter a. As increases, the created tools have shorter links at the sides decrease material us- age. With lower a, large tools reduce the control pol- icy't 's required movement.
5.3 Trading off design and control complexity
In this section, we aim to determine whether our introduced tradeoff parameter (defined in Equa- tion 1) can actualize preferences in the tradeoff between design material cost and control energy consumption. For this experiment, we focus on the catch balls task, because the tradeoff has an intuitive interpretation this setting: a larger tool can allow the agent to catch objects with mini- mal movement, while smaller tool can reduce material cost but requires longer trajectory with additional energy costs. We train four agents on catch balls with different values of
In Figure 7a, we plot the ratio dwee/das where represents the combined length of all tool links and Goad /amee is the per-step control velocity. dmax and Cmax indicate the maximum tool size and control velocity allowed by the environment. We find that this ratio indeed correlates with a, which indicates that agents that are directed prefer saving either material or energy are doing so, at the cost of the other. We also visualize the outputted tool designs in Figure 7b.
5.4 Evaluation on real robot
Next, we provide a demonstration of our pipeline on a real robot by transferring learned policies in of the 3D environments, fetch cube and lift cup, directly the real world. We use Franka Panda 7-DoF robot arm equipped with its standard parallel jaw gripper (Figure 11). Five RealSense D435 RGBD cameras perform object tracking using AprilTags [38].
7

---

Fabricating tools. In order to evaluate our policies in the real world, we fabricate the tools that are outputted by the design policy via 3D) printing. Specifically, based on an initial state observation and/or goal, we convert tool parameters from design policy outputs into meshes and construct them from PLA using consumer Ender and Ender 5 fused deposition modeling (FDM) 3D printers. See Figure for examples of printed tools generated by our policy.
Real robot evaluation. For the fetch cube and 1ift cup tasks, we evaluate policies on four initial cube positions and four cup geometries respectively. For each task instance, we fabricate the designed tool and evaluate control over five trials. We report success rates for each tool in Table 1, finding that our policies successfully transfer their performance to the real world.
Figure shows qualitative examples of real rollouts. In the top row, the robot designs an 'L'-shaped hook. Because the end- effector has constrained workspace due to the risk of colli- sion with the overhang, the robot employs two-stage strategy first hooking the block from under the overhang, and then using the backside of the tool to drag closer to itself. In the second row, the robot uses a -shpeed tool quickly retrieve the nearby block and finally pushes it to the goal with the grip- per fingers when it is within reach. This exemplifies that our framework flexibly allows an agent to use its original morphol- ogy in combination with designed tools when needed. Finally, in the lift cup task, the design policy selects appropriate dis- tances between the tool arms and an angle of approach to hold the cup securely when elevated.
Figure 9: 3D ar-printed tools for fetch cube (top) and lift cup (hattom).
Tool number
2
3
4
For fetch cube, we further analyze the performance of the control policy when using tools for initial positions that they were not directly designed for. Specifi- cally, we take the four generated tools and evaluate how well the control policy can them to solve 12 tasks 12cm 85.6cm grid of initial cube positions.
Fetch cube (single init.) Fetch cube (grid inits.)
5/5
5/5
5/5
5/5
10/12
4/12
6/12
5/12
Lift cup
5/5
5/5
5/5
5/5
Table Real world evaluation performance. "Sin- gle init" denotes the evaluation of a tool on the initial environment state it was designed for. "Grid of inits" evaluates the same tool on range of initializations.
The results are presented in Table 1. While the control policy can reuse tools to solve new tasks, no tool can solve all the tasks. Tool solves the most tasks, but tool solves both tasks that tool 1 cannot. We also find that the policy takes a greater number of timesteps to solve each task with tool 1 compared tools specialized for those initializations. These experiments demonstrate that diverse set of tools is indeed important for different variations of this task.
6 Conclusion
We have introduced framework for agents to jointly learn design and control policies, as step towards generalist embodied manipulation agents that are unconstrained by their own morphologies. Because the best type of tool and control strategy can vary depending on the goal, we propose to learn designer and controller policies to generate useful tools based on the task at hand and then perform manipulation with them. Our work is a step towards building embodied agents that can reason about novel tasks and settings and then equip themselves with the required tools solve them, without any human supervision. Such systems may lead the way towards autonomous robots that can perform continuous learning and exploration in real-world settings.
Limitations. In this work, we focus on rigid, non articuaied tools composed of linked primitive shapes. promising direction for future work is to explore other parameterizations.
addition, we do not address fabrication: we use 3D-printing for prototyping and do not consider the prob- lem of constructing tools from set of available objects. Finally, we focus on tool geometry, but consideration of tool stability and applied forces could lead to improved real-world performance.
8

---

Acknowledgments
We thank Josiah Wong as well as anonymous reviewers for helpful feedback on earlier versions of the manuscript, Samuel Clarke for recording the video voiceover, and the Stanford Product Real- ization Lab for 3D printing resources and advice. The work is in part supported by ONR MURI N00014-22-1-27400 the Stanford Institute for Human-Centered AT (HAI), Amazon, Autodesk, and JPMC. ST and MG are supported by NSF GRFP Grant No. DGE-1656518.
References
[1] Goodall. Tool-using and aimed throwing in community free-living chimpanzees. Nature, 201, 1964.
[2] A. M. Auersperg, S. Borasinski, I. Laumer, and Kacelnik. Goffin's cockatoos make the same tool type from different materials. Biology Letters, 12, 2016.
[3] K. Sims. Evolving virtual creatures. Proceedings of the 21st Annual Conference on Com- puter Graphics and Interactive Techniques, SIGGRAPH '94, page 15-22, New York, NY, USA, 1994. Association for Computing Machinery. ISBN 0897916670.
[4] K. S. Luck,
Amor, and R. Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages
2019.
[5] D. Hejna, III, P. Abbeel, and L. Pinto. Task- agnostic morphology evolution. In International Conference Learning Representations, 2021.
[6] A. Gupta, S. Savarese, S. Ganguli, and L. Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 2021.
[7] Y. Yuan, Y. Song, Z. Luo, W. Sun, and K. M. Kitani. Transform2act: Learning a transform- and-control policy for efficient agent design. International Conference on Learning Repre- sentations, 2022.
[8] K. Kawaharazuka, Ogawa, and C. Nabeshima. Tool shape optimization through backprop- agation of neural network. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8387-8393, 2020.
[9] K. R. Allen, T. Lupeaaa K. Stachenfeld, A. Sanchez-Gonzalez, P. Battaglia, Hamrick, and T. Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv: Arxiv-2202.00728, 2022.
[10] Y. Wu, S. Kasewa, O. Groth, S. Salter, L. Sun, O. Jones, and Posner. Imagine That! lever- aging emergent affordances for 3D tool synthesis. arXiv preprint arXiv: Arxiv-1909, 13561, 2019.
[11] H. Ha, S. Agrawal, and S. Song. Fit2form: 3D generative model for robot gripper form design. In 4th Conference on Robot Learning, CoRL. 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA. volume 155 of Proceedings of Machine Learning Research, pages 176- 187. PMLR, 2020.
[12] Exarchos, K. Wang, H. Do Stroppa, M. Coad, M. Okamura, and C. Liu. Task-specific design optimization and fabrication for inflated-bean soft robots with growable discrete joints. In 2022 International Conference Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 7145-7151. IEEE, 2022.
[13] Y. Li, T. Kong, L. Li, Y. Li, and Y. Wu. Learning design and construct bridge without blueprint. In IEEE/RSJ International Conference Intelligent Robots and Systems. IROS 2021, Prague, Czech Republic, September 27 Oct. 1, 2021, pages
IEEE, 2021.

---

[14] M. Li. R. Antonova, D. Sadigh, and J. Bobg. Learning tool morphology for contact-rich manipulation tasks with differentiable simulation. arXiv preprint arXiv:2211.02222) 2022.
[15] I.. Nair, Shrivatsav, and S. Chernova, Tool macgyvering: novel framework for combining tool substitution and construction. arXiv preprint arXiv: Arxiv-2008. 10633,,0 2020.
[16] S. K. S. Ghasemipour, S. Kataoka, B. David, D. Freeman, S. S. Gu, and I. Mordatch. Blocks assemble! learning to assemble with large-scale structured reinforcement learning. In Interna- tional Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 7435-7469. PMLR, 2022.
[17] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task- oriented grasping for tool manipulation from simulated self-supervision. In Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018.
[18] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7278-7285. IEEE, 2020.
[19] J. Brawer, M. Qin, and B. Scassellati. A causal approach to tool affordance learning. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8394- 8399, 2020.
[20] D. Xu, A. Mandlekar, R. Martin-Martin, Zhu, S. Savarese, and Fei-Fei. Deep affordance foresight: Planning through what can be done in the future. In IEEE International Conference on Robotics and Automation (ICRA), pages 6206-62111 IEEE, 2021.
[21] Y. Noguchi, T. Matsushima, Y. Matsuo, and S. S. Gu. Tool as embodiment for recursive manipulation. arXiv preprint arXiv: Arxiv-21 /2.00359 2021.
[22] K. R. Allen, K. A. Smith, and J. Tenenbaum. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 2019.
[23] A. Xie, F. Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. In Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019.
[24] R. Girdhar, L. Gustafsonn, Adcock, and van der Maaten. Forward prediction for physical reasoning. arXiv preprint arXiv: Arxiv-2006.1077 2020.
[25] X. Lin, Z. Huang, Y. Li, J. B. Tenenbaum, D. Held, and C. Gan. DiffSkill: Skill abstraction from differentiable physics for deformable object manipulations with tools. In International Conference on Learning Representation (ICLR), 2022.
[26] M. Toussaint, K. R. Allen, K. A. Smith, and B. Tenenbaum. Differentiable physies and stable modes for tool-use and manipulation planning. In Proc. of Robotics: Science and Systems (R:SS), 2018.
[27] M. Toussaint, J.-S. Ha. and O. S. Oguz. Co-optimizing robot, environment, and tool design via joint manipulation planning. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2021.
[28] T. Liao, G. Wang, B. Yang, Lee, K. Pister, S. Levine, and R. Calandra. Data-efficient learning of morphology and controller for microrobot. In 2019 International Conference Robotics and Automation (ICRA), pages 2488-2494, IEEE, 2019.
10

---

[29] C. B. Schaff. D. Yunis, A. Chakrabarti, and M. R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages 9798-9805 IEEE, 2019.
[30] D. Ha. Reinforcement learning for improving agent design. arXiv:1810.03379. 2018.
[31] T. Chen, Z. He, and M. Ciocarlie. Hardware as policy: Mechanical and computational co- optimization using deep reinforcement learning. Corl, 2020.
[32] J. BBatii,, Jackson, Y. Tian, J. Xu, and W. Matusik. Evolution gym: A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34, 2021.
[33] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros. Learning control self-assembling mor- phologies: A study of generalization via modularity. arXiv preprint arXiv: Arxiv-1002.05546, 2019.
[34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.06444 2017.
[35] E. Catto. Box2D, URL https://box2d.org//
[36] E. Coumans and Y. Bai. PyBullet, a Python module for physics simulation for games, robotics and machine learning. http::/ attp://pybullet..rrg 2016-2019.
[37] N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation, pages 312-317, 1996. doi: 10. 1 RTT./////EE..99.........
[38] E. Olson. AprilTag: robust and flexible visual fiducial system. In 2011 IEEE International Conference Robotics and Automation, pages 3400-3407, 2011.
[39] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Emestus, and N. Dormann. 3table-Basclines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8 2021. URL http://jmlr otg/papers/vv22/22 10000-3334..00 html.
[40] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier. Polymetis. facebookresearch. github. ww/fiio//oooooo http:/aairr/ollmmess/e...... 2021.
https:// //

---

A
Environment Details
Here we provide additional details for our simulation environments. An unabridged version of the description from Section 5 is as follows:
Push (2D): Push round puck using the tool such that it stops at the specified goal loca- tion. The goal space is a subset of 2D final puck locations G C IR² and the control action space A E R² specifies the z and y tool velocities.
Catch balls (2D): Use the tool to catch three balls that fall from the sky. The agent's goal is to catch all three balls, which start from varying locations on the z-y plane. We use a I-dimensional control action space that specifies the z velocity of the tool at each step.
Scoop (2D): Use the tool to scoop balls out of a reservoir containing 40 total balls. Here we specify goals of scooping x (1, 2, ...,) 7] balls. The control action space Ac R specifies the velocity of the rigid tool in æ and y directions, along with its angular velocity.
Fetch cube (3D): Use the tool to retrieve an object randomly positioned beneath a ver- tical overhang. This task is additionally challenging because the position of the robot end effector is restricted to rectangular region in the z-y plane of dimensions 0.8m 0.2m to avoid collision with the overhang. The tool is three-link chain where each link is a box parameterized by its width, length, and height. The design space also includes the relative angle between two connected links, with a total of n = 11 parameters. The control action space Ac E R represents a change end-effector position.
Lift cup (3D): Use the tool to lift a cup of randomized geometry from the ground into the air. This task requires careful design of the tool to match cup geometry. The tool is four-link fork with two prongs parameterized by the separation, tilt angle, width, length, and height of the prongs. The same parameters are applied to both prongs to maintain symmetry. The handle dimensions are fixed. The design space has n = parameters. Ac E F represents a change in end-effector position.
Scoop (3D) A 3D analog of the 2D scoop task. This task has the same goal space as the 2D scoop task, but the tool in 3D is a six-link scoop composed of a rectangular bottom plate parameterized by its width and length, and four rectangular side plates attached to each side of the bottom plate. Each side plate parameterized by its height and relative angle the bottom plate. A handle with fixed dimensions is attached one of the side plates. There effector pose.
10 total design parameters. Aç R6 represents change in end-
Our task selection is motivated by real-world tasks that a robot may need to perform for example in a home robot setting. Examples include:
Fetching objects (fetch cube): retrieving fallen misplaced objects e.g. underneath sofas, tables, or chairs in homes, tight spaces like inside cars, enabling scene "resets" from states where objects are lost for continuous robotic learning settings
Scooping (scoop 2D, 3D): manipulating granular materials such as rice, beans, cereals or measuring and transferring liquids for cooking
Pushing (push): home robotics applications when robots are impeded by obstacles such as countertops, tables, beds; industrial applications to aligning and grouping objects for robotic packing
Lifting objects (lift cup): Transporting objects that are too large for parallel jaw gripper for a suction gripper to stably grasp, for example, pots and pans, garbage cans, kitchen appliances; or risky to grasp directly, for example, in high temperature industrial welding and forging
12

---

B Experimental Details
B.1 Training Hyperparameters Architecture Details for Our Framework
In Tables 5, 6, 7, 8, 9, and 10, we provide detailed hyperparameters for our framework for each environment. Unless otherwise specified, we use the neural network architectures for the design policy, control policy, and value function from [7].
Robot arm base
Robot arm base
Tool
(b) Tool
Robot arm base Tool 3
Robot arm base
(d) Tool
Figure 10: Heatmaps of success and failures for real world fetch cube trials where fixed tools used for range of initializations. Recall that we test the policies on a positions that span a total dimension of 12cm x 85.6cm, for a total of 12 trials per tool. This means that evaluations were performed every 6em along one dimension and 28.53 (8.53em along the other. The grid here is top-down view of the robot workspace and directly maps to the set of 2D initial cube locations tested each tool, where the base of the robot is at bottom of each diagram. Here green indicates success, red indicates failure, and orange indicates failure where the final cube position within 5cm success.
grid of initial
B.2 Training Hyperparameters Architecture Details for Baselines
For the CMA-ES baseline, we perform hyperparameter sweeps for fair comparison with our frame- work. For the CMA-RI. baseline, we use the same set best performing hyperparameters for the outer CMA-ES loop. The tested hyperparameter configurations for each baseline are listed in Ta- ble Except model architecture differences, we use the same optimization hyperparameters for Ours, Ourseshared arch.), and HWasP-minimal Note that we control for the number of network parameters in the "shared arch" ablation notably, used MLP policies implemented in Stable Baselines [39] and ensure that the number of trainable network parameters is either the same or strictly larger than in our method across all tasks.
B.3 Computational Resources
We train each of our policies using single GPU (NVIDIA RTX 2080Ti TITAN RTX)
and 32 CPU cores. The total wall clock training time varies per environment from
13

---

Method
Hyperparameters
Values
Population Size
10, 24, 100, 1000
Initial Stdev
0.1, 1.0, 10.0
Center Learning Rate
0.01, 0.1, 1.0
CMA-ES
Covariance Learning Rate
0,01, 0.1, 1.0
Rank p Learning Rate
0.01, 0.1, 1.0
Rank One Learning Rate
0,01, 0.1, 1.0
Poicy Net
(256, 256, 256, 256, 256)
Value Net
(256, 256, 256, 256, 256)
Learning Rate
3e-4
CMA-RL
Batch Size
(50000, 20000(3D scoop))
Minibatch Size
2000
Poicy Net
(256, 256, 256, 256, 256)
Value Net
(256, 256, 256, 256, 256)
Ours(shared Learning Rate
lc-4
arch.)
Batch Size
(50000, 20000(3D scoop)) 2000
Minibatch Size
Table We tune over these values for hyperparameters baseline methods. Bolded values indicate the best performing settings for CMA-ES, which we use in our comparisons.
hours for the Catch Balls environment to 24 hours for the Scoop(3D) environment. We detail the training/inference time for our model and baselines on the fetch cube task:
Ours Ours (shared arch) CMA-RI. HWasP-minimal/Hta 2018
CMA-ES
Training time 2.12c+2
7.29c+2
1.18e+3
2.98c+2 1.75e-3
3,46c+2
Inference time 1.68e-3
5.24c-4
5.32c-4
N/A
The training time represents the wall clock time in minutes needed to train the model for 10? environ- ment steps. The inference time denotes the wall clock time seconds needed to perform one forward pass through the model. These results are generated using NVIDIA Titan RTX GPU and Intel Xeon Gold 5220 CPU.
B.4 Generalization
to Unseen Goals Experiment Details
For Fetch cube, the rectangular region of initial poses is defined by x E [-0.395, 0.395] and y E 0..4, The cutout region corresponds to two disconnected rectangular patches contained in the training region defined by æ1 [-0.350, -0.045).
Figure 11:
Real world setup. We use Franka Panda arm and five RealSense D435 cameras for tracking.
cube pictured. used for the fetch cube task has a side length of 5cm.
3/1 [0.434, 0.666] and x2 E [0.045, 0.350), // [0.434, 0.666] respectively.
Zero-shot performance. We train six policies using our framework where the cutout region re- moves a fraction of the total training area equal to 0.1, 0.2, 0.4, 0.6, 0.8, and 0.9 respectively.
Fine-tuning performance.
For the fine-tuning experiment, four initializations that find policies do not complete successfully zero-shot: ((-0.167, 0.367), (-0.129,0.8. 0.357), (0.433,004493) (0.415, 0.610))
specifically select
B.5 Trading off Design and Control Complexity Experiment Details
We train four agents independently catch balls, setting the value of a, the tradeoff reward parameter defined in Equation I, to 0, 0.3, 0.7, and 1.0 respectively.
14

---

B.6 Real Robot Experiment Details
For our real-world experiments, we use a Franka Emika Panda arm. We control the robot using an impedance controller from the Polymetis [40] library.
Tools are 3D printed using polylactic acid (PLA) commercially available Ender 3 and Ender 5 printers with nozzle diameter O, 4.4m. We print using layer height of 0.3mm and 10% infill. We perform slicing using the Ultimaker CURA software.
We roll out each policy for 100 environment steps or until success detected.
For the fetch cube task, measure the success based whether the center of mass of the 5cm cube is closer than 0.5m from the base of the robot. Please see Table 3 for per-tool details. The tool images are shown in Figure 9, from left to right: Tools 4, 2, 3, respectively.
Tool
Initial cube position (x, y, z)
Tool
(-0.110, -0.803, 0.025)
Tool
(-0.339, -0.588, 0.025)
Tool 3
(0.155, -0.731, 0.025)
Tool
(0.211, -0.633, 0.025)
Table 3: Initial cube positions corresponding to tools fabricated in real experiments.
For the lift cup task, we measure success based on whether the cup has been lifted higher than 0.4m off of the plane of the workspace. Please see Table 4 for per-tool details. The tool images are shown in Figure 9, from left to right: Tools 1, 3, 4.
Tool
Cup geometry parameters (length/width, height)
Tool
(0.3, 0.6)
Tool 2
(0.3, 0.9)
Tool
(0.5, 0.8)
Tool 4
(0.9, 0.6)
Table 4: Cup geometry parameters corresponding to tools fabricated in real experiments. Note that the length and width parameters share a single value.
We also present detailed results for the fetch cube experiments using tools generated for specific initial position for a range of initializations. Recall that we test the policies on a 3 x 4 grid of initial positions that span a range 12cm x 85.6cm, for a total of 12 trials per tool. We plot the successes and failures for each tool according to geometric position in Figure 10. We can see that the control policy is able use each tool solve the task for several initializations, but each tool is specialized for particular regions.
15

---

Hyperparameter
Value
Tool Position Init.
(20, 10)
Control Steps Per Action
Max Episode Steps
150
Slack Reward
-0.001
Tool Length Ratio
(-0.5, 0.5)
Tool Length Init.
(2.0, 2.0, 2.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-1.0, 1.0)
Tool Angle Scale
90.0
Control GNN
(64, 64, 64)
Control Index MLP
(128, 128)
Design GNN
(64, 64, 64)
Design Index MLP
(128, 128)
Control Log Std.
-1.0
Design Log Std.
-2.3
Fix Design & Control Std.
True
Policy Learning Rate
2c-5
Entropy
0.01
Value Learning Rate
lc-4
KL Divergence Threshold
0.005
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table 5: Hyperparameters used for our framework on the push task.
Hyperparameter
Value
Tool Position Init.
(20, 10)
Control Steps Per Action
Max Episode Steps
150
Slack Reward
-0.001
Tool Length Ratio
(-0.5, 2.0)
Tool Length Init.
(2.0, 1.0, 1.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-1.0, 1.0)
Tool Angle Scale
60.0
Control GNN
(64, 64, 64)
Control Index MLP
(128, 128)
Design GNN
(64, 64, 64)
Design Index MIP
(128, 128)
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design & Control Std.
True
Policy Learning Rate
2e-5
Entropy
0.01
Value Learning Rate
lc-4
KL Divergence Threshold
0.002
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table 6: Hyperparameters used for our framework on the catch balls task.
16

---

Hyperparameter
Value
Tool Position Init.
(15, 10)
Control Steps Per Action
5
Max Episode Steps
30
Slack Reward
-0.001
Tool Length Ratio
(-0.7, 0.2)
Tool Length Init.
(6.0, 3.0, 3.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-0.1, 0.7)
Tool Angle Scale
90.0
Control GNN
(64, 64, 64)
Control Index MLP
(128, 128)
Design GNN
(64. 64, 64)
Design Index MLP
(128, 128)
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design Control Std.
True
Policy Learning Rate
2e-5
Entropy
0.01
Value Learning Rate
3e-4
KL Divergence Threshold
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table 7: Hyperparameters used for framework the scoop task.
Hyperparameter
Value
Tool Position Init.
(0.0, 0.5, 0.02)
Control Steps Per Action
10
Max Episode Steps
100
Slack Reward
-0.001
Success Reward
10.0
Box Dimensions Min
(0.005, 0.05, 0.005)
Box Dimensions Max
(0.015, 0.1, 0.02)
Tool Angle Min
(-90.0, -90.0.
Tool Angle Max
(90.0, 90.0, 90.0)
Control Action Min
(-1.0,-1.0, -1.0, -0.2, -0.2, -0.2, -0.2)
Control Action Max
(1.0, 1.0, 1.0, 0.2, 0.2, 0.2)
Control Action Scale
0.1
Control GNN
(128, 128, 128)
Control Index MLP
(128, 128)
Design GNN
(128, 128, 128)
Design Index MLP
(128, 128)
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design & Control Std.
False
Policy Learning Rate
le-4
Entropy 8
0.0
Value Learning Rate
3e-4
KL Divergence Threshold
0.5
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table Hyperparameters used our framework on the fetch cube task.
17

---

Hyperparameter
Value
Tool Position Init.
(0.0, 1.2. 0.05)
Control Steps Per Action
10
Max Episode Steps
150
Slack Reward
-0.001
Success Reward
10.0
Box Dimensions Min
(0.005, 0.02, 0.01)
Box Dimensions Max
(0.01, 0.1, 0.03)
Tool Angle Min
(-30.0, -30.0)
Tool Angle Max
(30.0, 30.0. 30.0)
Control Action Min
-1.77 1.1 (1.0, 100, 1.0, 1.57, 1.57, 1.57)
Control Action Max
Control Actioin Scale
0.1
Control GNN
(128, 128, 128)
Control Index MLP
(128, 128)
Design GNN
(128, 128, 128)
Design Index MLP
(128, 128)
Control Log Std.
0.0
Design Log Std.
-1.0
Fix Design Control Std.
True
Policy Learning Rate
2c-5
Entropy
0.01
Value Learning Rate
3e-4
KI. Divergence Threshold
0.5
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
5
Table 9: Hyperparameters used for our framework the lift cup task.
18

---

Hyperparameter
Value
Tool Position Init.
(0.0, 0.05, 0.1)
Control Steps Per Action
10
Max Episode Steps
100
Slack Reward
-0.001
Success Reward
10.0
Box Dimensions Min
(0.04, 0.005, 0.02)
Box Dimensions Max
(0.08, 0.005,
Tool Angle Min
(-15.0,-15.0,-1553
Tool Angle Max
(15.0, 15.0. 15.0)
Control Action Min
-1
Control Action Max
(1.0, 10,, 1.0, 1.57, 1.57, 1.57)
Control Action Scale
0.05
Control GNN
(128, 128, 128)
Control Index MLP
(128, 128)
Design GNN
(128, 128, 128)
Design Index MLP
(128, 128)
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design Control Std.
False
Policy Learning Rate
lc-4
Entropy
0.01
Value Learning Rate
3e-4
KI. Divergence Threshold
0.5
Batch Size
20000
Minibatch Size
2000
PPO Steps Per Batch
5
Table 10: Hyperparameters used for our framework on the 3D scoop task.
19


Reviews:
[{"review_id": "xOsMgB63P2e", "reviewer": "Reviewer_D5az", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors present learning a robot that learns both to (1) design a tool and afterward (2) to use the tool in order to solve a manipulation task in hand. The manipulation tasks are pushing an object, catching a ball, scooping balls, fetching cubes, and lift cups. In training, they divide the training steps into two phases: \"design\" and \"control\". In the design phase the agent optimizes the tool at hand. In the control phase the agent learns how to use the tool. The agent maintain two separate policies for the the two phases. Some points were considered by the authors in order to enable efficient learning such as low dimensional design space but still enabling wide range of tasks, relevant algorithm (PPO) with relevant representation (GNN), and finally, good auxiliary reward. The algorithm was tested both in simulation and on real robot. Comparison to other tool design algorithms was done as well. The results showed that validity of the algorithm, even on unseen tasks.", "summary_of_recommendation": "I really thank the authors for the paper and appreciate their work. The decision is not trivial in this case. The work is fair and the problem is interesting. But I'm not sure the work addresses (with the current experiments, as detailed above) the bar of CoRL. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) Can you provide more tasks that can be solved with this approach? On real robot is the best, in simulation as well, or at least to mention more realistic scenarios where your approach may be relevant.\n\n2) What are the resources that were used in order to train the system? Compute, time, etc.\n\n3) What is the samples complexity on the robot to complete the tasks? \n\n4) Was the transfer from simulation was in zero shot, one shot, few shot, finetunning?\n\n5) Did the simulation was realistic, or generalization over the gap between simulation and real was possible?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality of the paper: \nThe paper is of interesting topic as we know that general manipulation (say in a single cell scenario or home robots) may require robots to be able to change their tools on the fly. With that respect I find the paper addresses the right problem which will be of interest in the years to come.Nevertheless, still the examples shown in this paper are of a toy-problem nature and I would expect (knowing that it is very challenging task to make it work) to see more realistic tasks. Scooping of reaching is TMHO quite a synthetic problems and they do not address the main problems of home robots or industrial robots.\n\nClarity:\nThe paper is written clearly. \n\nOriginality:\nThe algorithm to address the dual-nature tasks (design and control) if fairly straight forward. As the authors show in their experiments, they achieve SOTA performance. I believe that these results are authentic. I find the usage of GNN clever. Therefore, all in all this work has some original aspects.\n\nSignificance:\nAs mentioned in the quality section, I tend to asses that this work is significant, although in terms of realism we are not in realistic scenario yet.\n\nStrengths:\nThe work is \"closes the loop\" from presenting the task, designing relevant solution, showing the validity in simulation, and implementing the solution on a real Franka Emika Panda robot. The design phase relied on RL is nice and valid approach TMHO.\n\nWeaknesses:\nThe tasks are partially convincing and I find hybrid approaches of automation + AI more relevant. Also, I'm not sure that RL for control is the best way to handle the control phase. Grasping, reaching, with good perception may be achieved with low sample complexity where RL is not the right tool for control in grasping today (although there is abundant works in this direction by Sergey Levine and collaborators.).\n ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lA4wbUJsy6", "reviewer": "Reviewer_RS5q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes a learning method for designing and using tools for robot manipulation. The proposed method could jointly optimize tool design policy and tool use policy for solving tasks that are constrained by the robot's own morphologies. They also show their framework is sample efficient and could be transferred in multi-goal settings in the real world as well.\n\n", "summary_of_recommendation": "In summary, this paper proposes a framework for --tool-designing and using with RL methods. Though more details and explanations are needed to be added to show the validity of the method, I am leaning toward accepting this work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\nThe paper addresses an interesting problem -- robotic tool using and design. And the paper tries to solve it by jointly optimizing the design policy and control policy. The presentation of the method is clear and straightforward. Figures clearly express the algorithm step by step. There are sufficient quantitative results, baseline comparison, and real-world experiments.\n\nWeaknesses and suggestions:\n\n1) The parametrized design morphology for tools is relatively simple and it simplifies the problems as well. Is there a more general way to describe the tool's morphology during design? (like using some 3D mesh)\n2) In line 222, \"It does so by producing specialized tools to solve each task instance, while the other methods optimize for a single design across all task variations. \" The 'single design across all task variations' is confusing, why do baselines only optimize single design across all?\n3) In figure5, though the goals changed, I do not understand why the final tool design pattern looks like this. If those goals occur uniformly during training, the output tool pattern should be similar to a parallel gripper, please provide more insights here.\n4) Since your tool design is mainly based on geometry and morphology, the lack of considering the tool's stability/ applied force should also be mentioned in the limitation part.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "fKqLG3PdQr", "reviewer": "Reviewer_1xsZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a novel task that unlocks the ability of a embodied agent to design and use tools. The authors propose a two-stage training pipeline consisting of a designer network and a controller network. They train two networks jointly using PPO and implement the network using graph neural network. In the experiment session, the authors test 5 tasks in both simulation and real-world Franka Panda arm setup. To further study the performance of the proposed framework, the authors investigate the sample efficiency and the ability to generalize to unseen tasks by zero-shot generalization and finetuning.", "summary_of_recommendation": "This work proposes a novel task that allows the embodied agent to expand its ability by designing task-dependent tool. The idea and methods are described clearly. The limitations are well addressed. Thus, I would recommend a strong accept for this work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. If the degree of freedom provided to the neural network differs from task to task, this seems like an over-simplification of the proposed task. Or is it too difficult currently to use a more general parameterization such as generating a 2D mask to represent the shape of a 2D tool?\n2. How is the performance of Scoop (3D)? I cannot find the experiment results in terms of Scoop (3D) task.\n3. How to decide the position to pick the tool after they are fabricated in the real world?\n4. The author mentions using a two-stage strategy to avoid collision with the overhang. Does this change the output of the control policy?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work provides a novel task, which is a possible way of overcoming the morphology limitations of robot. It brings significant insight of how a embodied agent can learn to expand its physical ability to manipulate the world. The method and experiments are clearly described but some parts of the experiment result are missing.\n1. Strength:\n- A novel task that demonstrates how the embodied agent can learn to build tools to expand its morphology ability\n- The generalization is well demonstrated in the experiments.\n- The proposed method is straightforward and intuitive.\n2. Weakness:\n- Different parameterization configurations limit the ability of how the algorithm can generalize to new types of tasks.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "69ND5zoSzhr", "reviewer": "Reviewer_2UAy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The article discusses the potential of using objects from the environment as tools to accomplish tasks beyond the capabilities of humans and animals. It suggests that robots can also benefit from tool use by combining deep learning techniques to optimize both their physical design and control. \n\nThis paper proposes a \"designer policy\" approach, where a policy conditioned on task information generates tool designs to solve the task, while a separate controller policy performs manipulation using these tools.\n\nThe authors present a reinforcement learning framework that demonstrates improved sample efficiency compared to previous methods in scenarios with multiple goals or variants. It also showcases the ability to handle new tasks through zero-shot interpolation or fine-tuning, enabling trade-offs between design and control complexity.", "summary_of_recommendation": "I think the idea is interesting, but the approach used to tackle the problem seems to be standalone (does not seem to be tackling with robotics in mind). There are also questions on the approach used to evaluate the proposed method (needs more experiments and details)\n\n-----------------------------\n\n**Post rebuttal**\n\nThe author had addressed some of my concerns on the contribution and the lack of experimental details / highlights of this work's contribution. I will raise my rating.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Irrelevant to robotics", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "Please address questions raised as a result of the review given.\n\nNamely:\n- Experimental details on model size, architecture, and comparison to SOTA RL models with one unified model.\n- Visualisation of the evolution and questions on the tool design\n- Questions on the controlling aspect of the robot", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a framework for learning tool design and use in robotics, specifically for manipulation tasks. The framework consists of a two-phase Markov Decision Process (MDP) representing the design and control phases of the agent's environment. A designer and controller policies are jointly trained using interactions with the environment in both phases.\n\nIn the design phase, the designer policy selects the parameters for the tool that will be used throughout the episode. The environment state consists of task observations, such as object positions and velocities, as well as the robot's end-effector position, if applicable. In the control phase, the controller policy generates actions that actuate the previously designed tool. The control state includes the task observations and the design action.\n\nThis is a very interesting area, and the setting is also intriguing, where it asks the question of the best tool design for a given task.\nHowever, the whole setup feels like it is more targeting designing experiments (for \"inventing\" the best tools). This is an area that is being tackled in drug development [1] and as a sequential optimization problem [2]. It is an interesting problem. However, it can be an entirely decoupled problem with nothing to do with the robot arm. The paper uses two different policies \\pi_D and \\pi_C to model the \"designer\" and \"controller\" separately, and it was also found to be the better architecture than the proposed model with shared-arch (i.e., a unified one) (figure 4). The authors argue that:\n\n> The ablation shows the importance of learning separate designer and controller policies. (lines 225-226)\n\nHowever, what is the size of the network parameters? Does the \"shared arch\" essentially consists of half the trainable parameters count than the \"designer\"-\"controller\" counterpart? Is the \"worse performance\" contributed by having less trainable parameters? It would also be informative to have the corresponding training/inference time for each comparing model to assess the proposed model's actual performance correctly.\n\nThe fact that the solving of the whole robot control problem is decoupled into \"designer\" and \"controller\" models suggested that the problem is just being solved separately (and just so happened to be set to optimize in the same iteration step). A possible way for the author to argue their approach is to use some of the more advanced RL algorithms, like TD3, SAC, PPO, Rainbow DQN, etc., as a comparison as opposed to other ad-hoc models. Instead of using a two-stage \"designer\" / \"controller\" optimization step, other models can simply be optimizing all parameters jointly (both to control the arm and the design parameters of the tool).\n\n\nThe paper selects a design space for tools composed of rigid links, sufficient for manipulation tasks, and easily deployed in the real world. The policies are trained using proximal policy optimization (PPO), and a graph neural network (GNN) policy and value function architecture is adopted. Randomized goals are sampled from the environment during training in goal-conditioned environments. However, it is very hard to understand the result of the product from the \"designer.\" A better way to illustrate the designer's strength is to directly compare the final tools that each model created for some given tasks (to see why certain tool designs are superior to others). Concerning the videos, perhaps a video to see the evolution of the designed tool over the optimization epoch would also be highly useful to see how the tool evolves over the training horizon.\n\n\nI also have high doubts about the method used to evaluate the control of the robot. In lines 177-178, it was mentioned that 3D end-effector positions were used to control the arm. Based on the methodology in the paper and the video shown in the supplementary, the pose of the arm never changes. Despite the one short-coming of controlling the robot in world-space instead of configuration space (which is already limiting the expressiveness of controlling the robot), shouldn't the robot at least be controlled with the full SE(3) of the end-effector? \n\nIn addition, I need clarification on why the 2D scenario uses velocity control while the 3D scenario only uses position control. (Both 2D & 3D scenarios have \"Scoop\" action, which seems like they should be using a similary method to allow for a direct comparison?)\n\nResource constraints are also considered, and a trade-off between design material cost and control energy consumption is introduced through an auxiliary reward. This reward adjusts the task reward at each environment step based on the utilization of tool components and control velocity.\n\nLastly, based on the tool design figure given in Figure 2 (which consists of [ l_{1,...,3}, \\theta_{1, 2} ] ), I cannot see how the \"designer\" can possibly come up with the design shown in Figure 9 (bottom row). None of the parameters control the thickness of the \"stick\", no? Unless each task has its own specific default thickness across the \"sticks\"? However, if that is the case, that would imply that the author injects bias towards each task (instead of being a unified auto-tool-design framework across tasks).\n\n\n[1] Réda, C., Kaufmann, E., & Delahaye-Duriez, A. (2020). Machine learning applications in drug development. Computational and structural biotechnology journal, 18, 241-252.\n[2] Blau, T., Bonilla, E. V., Chades, I., & Dezfouli, A. (2022, June). Optimizing sequential experimental design with deep reinforcement learning. In International Conference on Machine Learning (pp. 2107-2128). PMLR.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to design the tool for each specific manipulation task by having separate designer and control policies where the designer policy designs the tool and the control policy performs the task using the tool. The approach is evaluated in both simulation and with a real Franka Panda robot arm. The experiments show sample efficiency and transfer to new tasks.
Decision: Accept (Poster)