Designing and Using Goal-Conditioned Tools
Authors: Ziang Liu
Stephen Tian
Michelle Guo
Karen Liu
Jiajun Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Design
Manipulation
RL
Tool use

Abstract: When limited by their own morphologies, humans and some species of animals have the remarkable ability to use objects from the environment towards accomplishing otherwise impossible tasks. Embodied agents might similarly unlock a range of additional capabilities through tool use. Recent techniques for jointly optimizing morphology and control via deep learning output effective solutions for tasks such as designing locomotion agents. But while designing a single-goal morphology makes sense for locomotion, manipulation involves a wide variety of strategies depending on the task goals at hand. An agent must be capable of rapidly prototyping specialized tools for different goals. Therefore, we propose the idea of learning a designer policy, rather than a single design. A designer policy is conditioned on task goals, and outputs a design for a tool that helps solve the task. A design-agnostic controller policy can then perform manipulation using these tools. In this work, we introduce a reinforcement learning framework for learning these policies. Through simulated  manipulation tasks, we show that this framework is more sample efficient than black-box optimization methods in multi-goal settings. It can also perform zero-shot interpolation or finetuning to tackle previously unseen goals. Finally, we demonstrate that our framework allows tradeoffs between the complexity of design and control policies when required by practical constraints.

Under review as a conference paper at ICLR 2023
DESIGNING AND USING GOAL-CONDITIONED TOOLS
Anonymous authors Paper under double-blind review
ABSTRACT
When limited by their own morphologies, humans and some species of animals have the remarkable ability to use objects from the environment towards accom- plishing otherwise impossible tasks. Embodied agents might similarly unlock a range of additional capabilities through tool use. Recent techniques for jointly optimizing morphology and control via deep learning output effective solutions for tasks such as designing locomotion agents. But while designing a single-goa, morphology makes sense for locomotion, manipulation involves a wide variety of strategies depending on the task goals at hand. An agent must be capable of rapidly prototyping specialized tools for different goals. Therefore, we propose the idea of learning designer policy, rather than single design. A designer pol- icy is conditioned on task goals, and outputs a design for a tool that helps solve the task. A design-agnostic controller policy then perform manipulation us- ing these tools. In this work, we introduce a reinforcement learning framework for learning these policies. Through simulated manipulation tasks, show that this framework is more sample efficient than black-box optimization methods in multi-goal settings. It can also perform zero-shot interpolation or finetuning to tackle previously unseen goals. Finally, we demonstrate that our framework al- lows tradeoffs between the complexity of design and control policies when re- quired by practical constraints. Additional task visualizations can be found at this link: httpp:: //tool desi gii-gn--enn icl 2023. gi thub. io/
INTRODUCTION
Humans and animals are able to make use of tools to solve manipulation tasks when they are con- strained by their own morphologies. For example, when an item has been lost below the sofa, one might quickly deduce that long stick will help them retrieve it. Chimpanzees have been observed using tools to access termites as food and hold water (Goodall 1964). and cockatoos are able to create stick-like tools by cutting shapes from wood (Auersperg et al. 2016). To flexibly and re- sourcefully accomplish a range of tasks comparable to humans, embodied agents should also be able to leverage tools. However, while any object in human robot's environment is potential tool, these objects often need to be correctly selected or combined to form a useful aid for the task goal at hand. For this reason, we investigate only how agents can perform control using tools, but also how they can design appropriate tools when presented with a particular task goal, such as target position or object location.
For an embodied agent to design and use tools in realistic environments with minimal supervision, it must be able efficiently learn design and control policies with reward signals specified based only on task completion. Furthermore, it should form specialized tools based on the task goal at hand, as Finally, should be able to work with the materials has available, rather than
shown in Figure [ attempting to create potentially unrealizable designs.
Without detailed supervision, how can an agent acquire effective policies for both tool design and control? The combined space of potential designs and control policies grows exponentially even for simple tasks, and the majority of candidate tools and trajectory executions may not make any progress towards task completion. As result, zeroth-order optimization techniques like evolution- ary strategies and naive reinforcement learning approaches require many samples from the environ- ment to find solutions. Prior works have studied joint learning of agent morphologies and control policies for locomotion tasks (Pathak et al. 2019 Luck et al. 2019 Hejja et al.) [202]] Gupta et al.) 2021), and methods leveraging graph neural networks (GNN)s have shown promising performance

---

Under review as a conference paper at ICLR 2023
improvements using just task rewards supervision (Yuan et al. 2022). However, these approaches optimize designs for a generic goal, such as maintaining balance or forward speed. Designing a single-gor -oaa morphology is suited for locomotion, but manipulation requires range of strategies depending on the given task. An agent must be capable of rapidly prototyping specialized tools for different manipulation goals.
In this work, we tackle the challenges of learning design and control solely from task progress rewards by lever- aging recent work in joint morphology and control opti- mization for locomotion agents, which performs RL us- ing multi-stage Markov decision process (MDP) com- bined with graph neural network (GNN) policies and value functions to achieve improved sample efficiency and performance in the joint learning setting. This, com- bined with simple chain-link tool design parameteriza- tion, allows us to perform efficient learning of designer and controller policies together in nigh-dimensional combined space. We train these policies on multiple goal settings for each task that they can produce designs best suited to each goal and manipulate tools with varying geometries. Lastly, we investigate how tunable parame- ters can control the trade-off's between the complexity of design and manipulation under resource constraints.
An agent may need design and different tools to fetch high-up book (orange) or push it into the hookshelf (green). Therefore, it should rapidly proto- type tools for the tasks at hand.
Figure I:
Our main contribution is learning framework for embodied agents to design and use rigid tools for manipulation tasks. We leverage multi-stage reinforcement learning pipeline to learn specifici tools in addition to manipulation policies that can perform control with range of tools. We demon- strate that this approach can jointly learn these policies in sample-efficient manner variety of sparse reward manipulation tasks, outperforming zeroth-order stochastic optimization approaches. By introducing a tradcoff parameter between the complexity of design and control components, our approach allows to adjust the learned components to fit resource and environmental constraints, such as available materials or energy costs. To the best of our knowledge, this work is the first that studies learning goal-dependee tool design and control without any prior knowledge about the task.
2 RELATED WORK
Computational approaches to agent design. Many works have studied the problem of optimiz- ing the design of robotic agents and end-effectors via model-based optimization (Kawaharazuka
cet al. 2020: Allen et al. 2022). generative modeling (Wu et al. 2019 Ha et al. 2020), evolution- ary strategies (Hejna et al. 2021), stochastic optimization (Exarchos et al. (202)), or reinforcement learning (Li et al. 2021). These methods provide feedback to the design procedure by having the agent execute predefined trajectories or perform motion planning. In contrast, we aim to jointly learn control policies along with designing tool structures. In settings where the desired design is known but must be assembled from subcomponents, geometry (Nair et al. (2020) and reinforcement learning (Ghasemipour et al. 2022) have been used to compose objects into tools.
Learning robotic tool use. Several approaches have been proposed for empowering robots to learn use tools. Affordance learning is one common paradigm (Fang et al.) (2018) Brawer al. 2020 Xu et al. 2021). Noguchi et al. (202T) integrate tool and gripper action spaces in a
T Transporter-style
framework. Learned or simulated dynamics models (Allen et al., 2019 Xie et al.) 2019 Girdhar et al. 2020 Lin et al. 2022) have also been used for model-based optimization of tool-aided control. These methods assume that a helpful tool already present in the scene, whereas we focus on optimizing tool design in conjunction with learning manipulation, which is more likely scenario for a generalistt robot operating for example a houschold.
Joint optimization of morphology and control. One approach for jointly solving tool design and manipulation problems is formulating and solving nonlinear programs, which have been shown to be especially effective at longer horizon sequential manipulation tasks (Toussaint et al. 2018 2021 In this work, we aim to apply our framework to arbitrary environments, and so we select purely learning-basee based approach at the cost of increasing the complexity of the search space.
2

---

Under review as a conference paper at ICLR 2023
Designer Policy
A
TD
Tool design
State, goal observation S0, 9
Design phase
Control Policy T
Actions
Observation, Reward
Environment
Control phase:
Solving task using learned designer and controller policies. During the design phase, the designer policy outputs the parameters for tool that will help solve the given task. In the control phase, the controller policy then outputs motor commands given the tool structure and task specification.
Figure 2:
Reinforeement-leamiing based approaches have also been applied to jointly learning morphology and control. These include policy gradient methods with either separate (Schaff et al., (019) or weight-sharing (Ha 2018 design and control policies, or considering the agent design compu- tational graph (Chen et al. 2020). Evolutionary or Bayesian optimization algorithms have also been combined with control policies learned through RL to solve variety of locomotion and manipula- tion tasks (Bhatia et al. 202T). Luck et al. (2019) use an actor-critic RI. formulation with graph neural network (GNN) value function for improved sample efficiency. Pathak et al. (2019) learn to design modular agents by adding morphology-modifying actions to MDP. Yuan et al. (2022) provide a generalized formulation with multi-stage MDP and GNN policy and value networks. These methods have demonstrated promising performance locomotion tasks. In this work, focus on manipulation, which often involves sparser rewards and where adding additional actuated joints to designs is infeasible. Furthermore, rather than optimizing for single task objective, our work addresses the challenge of goal-conditioned design and control.
3
OUR FRAMEWORK
In this section, introduce problem formulation and describe one instantiation of our frame- work that we study in the remainder of this paper.
3.1 PROBLEM SETTING
Our framework tackles learning tool design and use for agents to solve manipulation problems, without any supervision about the quality of designs except for task progress. Because would like flexibility in the formulation of the design space in an arbitrary control state space, we represent the agent's environment as two-phase Markov decision process (MDP), including a design phase and control phase. Then, by jointly learning policies for the two phases, we can use the collected environment interactions for training both policies.
At the start of every episode, the environment begins in the design phase, visualized in the top of Figure 2 During the design phase, the action a E Ap specifies the parameterization of the tool that will be used for the rest of the episode. In this phase, each state Sp consists of vector of task observations, for example positions and velocities of objects the scene. Because the utility of a given tool varies depending the task hand, rewards provided during this phase.
3

---

After single transition, the MDP switches to the control phase, illustrated in the lower half of Figure 2 During the execution phase, the actions a E Ag represent motor commands applied to the previously designed tool, and the agent receives rewards based on task progress (e.g., the distance of an object being manipulated to the target position). The control phase state space Sp includes the task observations just like in the design phase, but also the tool parameterization from the design Under review as a conference paper at ICLR 2023
phase. The execution phase continues until the task is solved or time limit is reached, and the episode then ends.
Given this two-phase MDP, the goal is to learn optimal design and control policies, (a]s) and The (a|s) respectively, that maximize the expected discounted return over the entire episode. This is a challenging problem setting because the agent only receives signal about the quality of the tool designs through the final task performance.
In this work, we design this formulation for jointly learning design and control for manipulation. A similar formulation has been explored in prior work for locomotion (Yuan et al.) [2022); however, manipulation and locomotion problems have a key difference: while in locomotion, a morphology can be optimized for a single objective like forward velocity and eventually constructed the real world, tool use for manipulation requires varied design and control strategies based on the task at hand. Therefore, rather than finding a single tool design, we would like to learn a designer policy that can create tools depending on the given task, well as a controller policy that can perform manipulation using different tools. We formulate this as conditioning on a supplied goal g from a goal space 9, such as the desired final location of object to be manipulated. The objective is then to find the optimal goal-conditioned designer and controller policies TD (als, g). ( g) that maximize the expected discounted return of a goal-dependent reward function R(ss a, g.....
3.2 INSTANTIATING OUR FRAMEWORK
Next, we provide a concrete implementation of our framework catered towards solving a series of 2D manipulation tasks. Specifically, we select a tool design space, policy learning procedure, and auxiliary reward function.
Tool design space. The design space can significantly impact the difficulty of the joint design and control optimization problem. When the set of possible designs is large but many of them are unhelpful for any task, the reward signal for optimization sparse. Thus, we would like to select a design parameterization that is low-dimensional, but can also enable many manipulation tasks. Furthermore, we prefer designs that are easy to deploy in the real world.
For this implementation, we parameterize the tool using 3-link rigid chain, but we note that our framework is not limited to this choice of design space. We find that while this parameterization is simple. it is sufficient to help solve a variety of 2D manipulation tasks. Because of its topology, it can also be easily deployed in the real world for example on soft robots (Exarchos et al. 2022) or through rapid fabrication techniques like 3D printing. Specifically, each tool is parameterized by vector [4,, 12, 13. 01, 02] E IR5, where represents each link length and 0 represents the relative angle between the links. The width of each link is fixed.
2022) to learn the designer and Policy learning. We follow a similar procedure as Yuan et al.
controller policies. Specifically, we interactively collect experience in the environment using the design and control policies, where each trajectory spans the design and execution phase. We then train the policies jointly using proximal policy optimization (PPO) (Schulman et al. (017)) a popular off-the-shelf policy gradient method. As prior work (Yuan et al. 2022) has shown that graph neural network (GNN) architectures can be helpful for accelerating learning when an input morphology can be converted to a graph representation, we use GNNs as part of our policy architectures for the design, control, and value function networks. The input graph structure consists of node for each joint in the tool design and an edge for each pair of connected joints. For our chain link parameterization, this is always the three-node path graph Ps. Each input node feature includes the environment observation, and additionally for the control policy and value function, the design parameters (e.g. relative angle and length) of the corresponding node in the tool design. As in prior work, GraphCony (Morris et al. 2019) to perform graph convolution, before flattening features from all nodes into a single vector and passing this through a multi-layer perceptron (MLP). Additional architectural and training details are provided in Appendix A
4

---

Under review as a conference paper at ICLR 2023
(a) Pushing task.
(b) Ball catching task.
(c) Scooping task.
Figure 3: Visualizations of the 2D simulated manipulation environments for testing framework.
During training, in order for the designer and controller to learn to achieve a variety of goals, we sup- ply the policies with randomized goals sampled from the environment for each interaction episode. The training objective is to maximize the expected return across sampled goals.
Auxiliary reward. When developing embodied agents that can flexibly create and use tools in the real world, resource constraints are an additional consideration. Most co-optimization procedures assume that material such actuated joints or body links can be arbitrarily added the agent morphology. However, as an example, when an agent solves manipulation tasks in household environment, may not have access to additional motors or large quantities of building materials. One possible solution is for the agent to instead design tool that requires less material to create, and execute more complex manipulation trajectory instead. On the other hand, when constraints allow, constructing larger tool can reduce the amount of energy that the agent consumes through motor control, especially a task must be completed many times.
We enable framework to accommodate preferences in this trade-off between design material cost and control energy consumption using a parameter a that adjusts an auxiliary reward that is added to the task reward at each environment step:
Ttradcoff = K (
aa. (a dused
+ (1 (1-a)- ) Cused ]:
(1)
dmax
Cmnx
where K is a scaling hyperparameter that we set to 0.7 for all experiments, E [0, 1] controls the balance of emphasis on either the control or design component, dused and dmax represent the used and maximum possible combined length of the three links in the design respectively, and Gued and Cmax represent the control velocity at the current step and the maximum single-step control velocity allowed by the environment. By adjusting the parameter a, can encourage the policy to favor using less material for tool construction when a large, and less energy for the control policy when is small.
EXPERIMENTS
We conduct experiments using three environments created in the Box2D simulator (Catto): pushing, ball catching, and scooping. We select these manipulation tasks to showcase the advantages of using different tools to achieve different goals, when there does not exist single optimal tool for all goals. The three multi-goal tasks are shown in Figure For each task, we initialize the designed tool by matching a fixed point on the tool a fixed starting position regardless of the goal. During the control phase, we emulate a scenario in which the tool has been grasped by a robot and is manipulated using velocity control in Cartesian space. short description of each task is follows:
Pushing: Push round puck using the tool such that stops at the specified goal location.
The goal space is a subset 2D final puck locations g C R?, and the control action space .A IR² specifies the and y tool velocities. The reward at each step is the change in & distance between the puck and the goal position after taking that step, plus bonus of 3.0 for reaching the goal with velocity below 1.0.

---

Under review as a conference paper at ICLR 2023
33
233
Reward
04
Steps
Steps
Steps
(a) Pushing.
(b) Ball catching.
(c) Scooping.
Figure 4: Leariing curves for our framework and black box optimizer baselines on multi goal and single-goal settings. As baselines, we present the single-godl performance of CEM, CMA-ES, and PGPE, as well as the multi-goal training performance of PGPE. We can see that across all studied tasks, our framework achieves substantially improved performance, as well as competitive sample efficiency. Shaded areas indicate standard error multiple seeds (5 seeds multi-goal methads and 10 seeds single-goal methods).
multi-goal
Catching: Use the tool to catch three balls that fall from the sky. The agent's 's goal is to catch all three balls, which start from varying locations on the z-y plane. We use a 1- dimensional control action space that specifies the xr velocity of the tool at each step. The reward function is 3.0 for each ball caught, plus a bonus 3.0 upon capturing all balls.
Scooping: Use the tool to scoop balls out of a reservoir containing 40 total balls. Here we specify goals of either scooping single ball, or as many as possible. The control action space E RR3 specifies the velocity of the rigid tool in x and y directions, along with its angular velocity. The reward function is bonus of 6.0 for successfully completing the single ball goal, or a bonus of 3n for scooping n balls in the multi-ball goal.
T0
In our experiments, we evaluate whether the instantiation of our framework on these 2D manip- ulation tasks has the following three properties that we argue are desirable for embodied agents performing manipulation for tool use:
Can our framework jointly learn design and control in sample-efficient manner, using just rewards based on task progress?
Do our learned designer and controller policies generalize or enable finetuning to solve goals previously unseen during training?
Can our adjustable parameter a enable agents to specify preferences in the tradeoff between design and control?
4.1 EVALUATING SAMPLE EFFICIENCY
First, we investigate whether our framework demonstrates improved sample efficiency for manip- ulation problems compared to black-box optimization procedures. We compare to the following baseline methods:
CEM (single): We evaluate the cross-entropy method (Rubinstein 1999 de Boer et al, 2005), a zero-order optimization method that leverages importance sampling to guide the sampling distribution towards higher-scoring values. We evaluate this method in a single- goal setting, optimizing a separate set of parameters for each goal. We formulate the search space single vector containing the design parameterization concatenated with control actions for all steps in an episode.
CMA-ES (single): We evaluate the covariance matrix adaptation evolution strategy (CMA- ES) (Hansen & Ostermeier 1996), a popular evolutionary strategy that maintains popu- lation of sampled candidates and models the covariances between each feature of sampled vectors. We also evaluate this method in single-goal setting, optimizing over the same search space as CEM.
PGPE (single): We evaluate policy gradient parameter exploration (PGPE) (Schnke et al. 2008) in single goal setting. PGPE is a policy-gradient RL. method that the

---

Under review as a conference paper at ICLR 2023
Visualizations of tool designs outputted by single learned designer policy for the pushing task as the goal position, indicated in green. varies. We can see that the designer outputs range of tool designs that enable to more casily push the ball in the desired direction.
Figure 5:
natural gradient. We directly optimize the parameters of neural network that takes as input single goal observation and the environment state and outputs both the design and control actions. The appropriate action is selected based on the current phase when rolling out the policy in the environment.
PGPE (multi): We additionally compare against PGPE in multi-goal setting with simi- lar setup as the single goal setting but instead train goal-conditioned neural network to output design and control parameters.
We implement each baseline method using the EvoTorch library Toklu et al. 2022). hyperparameters and tuning details for our comparisons can be found Appendix A
Additional
In Figure 4 we compare the learning curves of our method and the baselines for all three tasks. The learning curves show the total reward versus the number of environment steps taken for each method. Compared to the single-goal baselines, we see that training using our framework exhibits better final performance and competitive sample efficiency. However, we emphasize that the single-goal baselines optimize one model for each goal, while our method learns one model for all goals. When compared to PGPE (multi), which learns one model for multiple goals, our method outperforms this baseline in both sample efficiency and final performance.
We present qualitative examples tool designs output for different goals on the pushing task in Figure We find that our designer policy outputs a diverse range of tools depending on the specified goal location, and the controller policy is able to perform manipulation conditioned on the tool design and goal to solve the task.
4.2 GENERALIZATION TO UNSEEN GOALS
Rather than learning single design and a policy to control it, our framework seeks to enable rapid tool prototyping for manipulation by learning goal-conditioned designer and controller policies. In simulation, these policies can experience millions of trials for range of goals. However, when deployed the real world, designer and controller policies cannot be pre-trained on all possible future manipulation goals. The agent may instead leverage what it has learned about tools that can useful for an unseen task from the goals it has seen. In this section, we test the ability of our policies to generalize to goals unseen during training. Because it has goal space that can be manipulated in a semantically meaningful way, we focus on the pushing task for these experiments. We train policies using framework region of the space, which we call the "cutout" region. (see Figure 6a). For the pushing task, because the goal space is the rectangular region defined by x [8.0,31 32.0 and y E [18.0, 34.0) specifying
subset of goals from the entire goal space by removing

---

Under review as a conference paper at ICLR 2023
Culour Ratio
Steos
(a) Visualization of the goal space and zero-shot performance with 0.25 cutout ratio.
(b) Cumulative episode reward versus cutout size for policies trained with progressively larger cutout regions.
Fine-tuning performance compared to learning from scratch across target goals.
(c)
Figure 6: Interpolation results the pushing task. In (a), we plot the success (light bluc) and failure (dark blue) goal regions. Areas within the dolted yellow borders denote unseen cutout regions (interpolation). The region within the teal border (but outside cutout regions) shows the training region. The area outside the teal border is unseen during training (extrapolation). (b) and (c) show reward curves (averaged over 3 runs). Shaded regions denote standard error. In (b), we observe that the performance for policies trained with small cutouts is close that of the setting trained on all goals. In (c) show evaluation performance of policies finetuned for a specific unseen goal compared to learning curves for single-task policies with those goals.
desired final ball location, the cutout region corresponds to two disconnected rectangular patches
in the goal space with 21 E [11.0, 07.0). 1/1 E [22.0, 30.0] and 22 E [23.0, 29.0]. 1/2 E [22.0, 0, 30.0] respectively. Then, evaluate the generalization performance of learned policies on the goals from the cutout region and outside the training region in two separate scenarios.
Zero-shot performance. In the first scenario, test the ability of the designer and controller policy to tackle previously unseen goal directly. Using policy trained on the goal space with cutout region removed as described above, we evaluate the zero-shot performance by randomly sampling unseen goals. In Figure (aa we visualize the zero-shot performance of our design and control policies on goals across the entire environment plane, finding that our policies are able to solve even goals outside the training region boundaries.
Next, we analyze how decreasing the number of possible training goals affects generalization per- formance. We train six policies using our framework where the cutout region removes fraction of the total training area equal to 16, 0.25, 0.36, 0.49, 0.64, and 0.81 respectively. We then plot the performance of these learned policies on goals seen and unseen during training.
In Figure 6b
we show the returns as the size of the cutout region for training goals varies. As reference, we additionally visualize the performance of policies trained with framework to solve a single randomly selected task. When the cutout region is very small, the performance of our learned policies on seen and unseen goals is similar. As the area of the cutout goal region increases, the performance on unseen goals drops, but is still much better than the single goal baseline and is able to solve significant portion of unseen tasks.
Finetuning performance. Sometimes, new goals cannot be solved by directly applying our pre- trained designer and controller policies. In this section, we test whether our policies can still serve as good instantiations for achieving these goals. We hypothesize that even when our policies do not solve the task directly, the designer policy may still be able propose reasonable tool that is helpful for solving the task, while the controller policy will be able to generalize to manipulate new tools. We test this by starting with policies pre-trained with our framework on the entire training goal region and finetuning them to solve goals outside that region. Specifically, we select three goals for finetuning: ((2, 13), (37, 28), (37, 13)]. For comparison, we also train three separate policies using framework to solve the single- pushing task towards each of these goals.
In Figure (6c] we show the results the finetuning experiment. We find that even for goals that are far away from the initial training region, policies are able to learn to solve the task within handful of gradient steps. Because the pre-trained policies are able to find reasonable control and especially tool initializations, the joint space of designs and control policies to explore is signifi- cantly narrowed. In comparison, learning a single-goal policy from scratch is challenging because achieving these distant goals requires many consecutive near-optimal decisions.
8

---

Under review as a conference paper at ICLR 2023
a=1.0
a-0.7
0-0.3
Control/Design Ratio
1.0
0=0.0
0.8
14 02
0.4
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Steps
(a) Control/Design ratio with different a.
(b) Tools produced by designer policies at dif- ferent levels of
Figure 7: Qualitative examples of tools generated by setting our tradeoff parameter a to different values. We can see that as increases, the tools created by the designer policy have shorter links at the left and right sides decrease material usage. With low values, large tools prevent the control policy from having move the tool
4.3 TRADING OFF DESIGN AND CONTROL COMPLEXITY
In this section, we aim to determine whether our introduced tradeoff parameter is effective at ac- tualizing preferences in the tradeoff between design material and control energy consumption. For this experiment, we focus on the catching task, because the tradeoff has an intuitive interpreta- tion in this setting: a larger tool can allow the agent catch the objects with minimal movement, while smaller tool can save on material cost but requires longer trajectory that has additional energy costs. We train four agents independently on the catching task, setting the value of a, the tradeoff reward parameter defined in Equation 1 to O, 0.3, 0.7, and 1.0 respectively.
In Figure 7a we track the relative amounts of energy expended for control and material used for design across these agents. Specifically, we plot the ratio duua/ dene where d represents the combined Cuuad/
length of all tool links and e is the per-step control velocity. dmax and Cmax indicate the maximum tool size and control velocity allowed by the environment. with which indicates that agents that are directed to prefer saving either material energy are doing so, at the cost the other. We also visualize the outputted tool designs in Figure 7b We see that for progressively larger values of the agent shortens the tool length the side of the catcher to conserve material, and instead navigates the tools further across the plane to catch the balls.
find that this ratio indeed correlates
5 CONCLUSION
We have introduced framework for agents to jointly learn design and control policies with the purpose of solving manipulation tasks. Because the best type of tool and control strategy can vary widely depending on the manipulation goal, we propose to learn designer and controller policies to generate useful tools based on the task at hand and perform manipulation with them. By leveraging reinforcement learning methods that use GNNs to compute features, we demonstrate using simu- lated 2D manipulation tasks that our framework can be instantiated to learn designer and controller policies for a variety of commanded goals. We find that when training on only subset of possi- ble goals, the learned policies can perform zero-shot generalization or rapid finetuning using many fewer samples compared to learning from scratch. We finally demonstrate that, through the addi- tion of a tunable tradeoff parameter, the learning procedure can be adjusted to better suit practical requirements such as resource constraints.
Reproducibility. We described our method and experimental setup detail in Sections and The code will be released upon publication to facilitate future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kelsey R. Allen, Kevin A. Smith, and J. Tenenbaum. Rapid trial-and-error learning with simula- tion supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 2019.
Kelsey R. Allen, Tatiana LopezaGuevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simu- lators. arXiv preprint arXiv: Arxiv-2202.000 2202.00728 2022.
Alice M.I. Auersperg. Stefan Borasinski, Isabelle Laumer, and Alex Kacelnik. Goffin's cockatoos make the same tool type from different materials. Biology Letters, 12, 2016.
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym: A large-scale benchmark for evolving soft robots. Advances Neural Information Processing Systems, 34, 2021.
Jake Brawer, Meiying Qin, and Brian Scassellati. A causal approach to tool affordance learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8394- 8399, 2020.
Erin Catto. Box2d. URL htt.ttttt////xx..coom
Tianjian Chen, Zhanpeng He, and M. Ciocarlie. Hardware as policy: Mechanical and computational co-optimization using deep reinforcement learning. Corl, 2020.
Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19-77, February 2005.
loannis Exarchos, Karen Wang. Brian H. Do. Fabio Stroppa, Margaret M. Coad, Allison M. Oka- mura, and C. Karen Liu. Task-specific design optimization and fabrication for inflated-bentt soft robots with growable discrete joints. In 2022 International Conference on Robotics and Automa- fion, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pp. 7145-7151, IEEE, 2022.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. In Robotics: Science and Systems XIV. Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018
Seyed Kamyar Seyed Ghasemipour, Satoshi Kataoka, Byron David, Daniel Freeman, Shixi- ang Shane Gu, and Igor Mordatch. Blocks assemble! learning to assemble with large-scale struc- tured reinforcement learning. In International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 7435-7469. PMLR, 2022.
Rohit Girdhar, Laura Gustafson, Aaron Adcock, and Laurens van der Maaten. Forward prediction for physical reasoning, arXiv preprint arXiv: Arxiv-2006.10738 10734, 2020.
Jane Goodall. Tool-using and aimed throwing in a community of free-living chimpanzees. Nature, 201, 1964.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 2021.
David Ha. Reinforcement learning for improving agent design. 2018.
Huy Ha, Shubham Agrawal, and Shuran Song. Fit2form: 3d generative model for robot gripper form design. In 4th Conference Robot Learning, CoRL. 2020, 16-18 November 2020, Virtual Event Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp- 176-187. PMLR, 2020.
N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strate- gies: the covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation, pp. 312-317, 1996. doi: 10. 199.6/999999699423333
10

---

Under review as a conference paper at ICLR 2023
Donald Hejna, III. Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. In Inter- national Conference on Learning Representations, 2021.
Kento Kawaharazuka, Toru Ogawa, and Cota Nabeshima. Tool shape optimization through back- propagation of neural network. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8387-8393, 2020.
Yunfei Li, Tao Kong, Lei Li, Yifeng Li, and Wu. Learning to design and construct bridge without blueprint. In IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 Oct. 1. 2021, pp. 2398-24482 2405. IEEE, 2021.
Xingyu Lin. Zhiao Huang. Yunzhu Li, Joshua B. Tenenbaum, David Held, and Chuang Gan. Diff- skill: Skill abstraction from differentiable physics for deformable object manipulations with tools. 2022.
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co- co-adaptation of morphology and behaviour with deep reinforcement learning. In 3rd Annual Conference Robot Learning, CoRI. 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 854-869 2019.
Christopher Morrs,, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In The Thirty-Thiie AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty- Inno- vative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 2019, 2019.
Lakshmi Nair, Nithin Shrivatsav, and Sonia Chernova. Tool macgyvering: A novel framework for combining tool substitution and construction. arXiv preprint arXiv: Arxiv-2008. 10638, 2020.
Yuki Noguchi, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Tool as embodiment for recursive manipulation. arXiv preprint arXiv: Arxiv-21 2021.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. arXiv preprint arXiv: Arxiv-1902.05546, 2019.
Reuven Rubinstein.
The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127-190, 1999.
Charles B. Schaff, David Yunis, Ayan Chakrabarti, and Matthew R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. In International Conference on Roboticy and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pp. 9798-9805. IEEE, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.063477 2017.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß. Alex Graves, Jan Peters, and Jürgen" Schmidhuber. Policy gradients with parameter-based exploration for control. In Artificial Neural Networks /CANN 2008, 2008
Nihat Engin Toklu, Timothy Atkinson, Vojtech Micka, and Rupesh Kumar Srivastava. Evotorch. https: http//tttttttutt com nna seeiieeeeeeeeeeeeeeeee www.s/eeooooereee 2022.
Mare Toussaint, Kelsey R Allen, Kevin A Smith, and Josh B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Proc. of Robotics: Science and Systems (R:SS), 2018.
Mare Toussaint, Jung- Su Ha, and Ozgur S. Oguz. Co-optimizing robot, environment, and tool design via joint manipulation planning. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2021.

---

Under review as a conference paper at ICLR 2023
Yizhe Wu, Sudhanshu Kasewa, Oliver Groth, Sasha Salter, Li Sun, Oiwi Parker Jones, and Ingmar Posner. Imagine that! leveraging emergent affordances for 3d tool synthesis. arXiv preprint arXiv: 13561, 2019.
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical un- derstanding: Using novel objects as tools with visual foresight. In Robotics: Science and Systems XV. University of Freiburg, Freiburg Breisgau, Germany, June 22 22-26, 2019, 2019.
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martínn Yuke Zhu, Silvio Savarese, and Li Fei-Fei.
Deep affordance foresight: Planning through what can be done in the future. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6206-6213, IEEE, 2021.
Ye Yuan, Yuda Song. Zhengyi Luo, Wen Sun, and Kris M. Kitani. Transform2act: Learning a maunffrm-and-comme policy for efficient agent design. In International Conference Learning Representations, 2022.
12

---

Under review as a conference paper at ICLR 2023
A TRAINING HYPERPARAMETERS & ARCHITECTURE DETAILS
A.1 OUR FRAMEWORK
Here we provide detailed hyperparameters for our framework. Unless otherwise specified, we use the neural network architectures for the design policy, control policy, and value function from Yuan
al. (202)).
Hyperparameters
Values (20, 10)
Tool Position Init.
Control Steps Per Action
Max Episode Steps
150
Slack Reward
-0.001
Tool Length Ratio
(-0.5, 0.5)
Tool Length Init.
(2.0, 2.0, 2.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-1.0, 1.0)
Tool Angle Scale
90.0
Control Log Std.
-1.0
Design Log Std.
-2.3
Fix Design & Control Std.
True
Policy Learning Rate
2c-5
Entropy /
0.01
Value Learning Rate
le-4
KL Divergence Threshold
0.005
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table I: Hyperparameters used for our framework on the pushing task.
Hyperparameters
Values (20, 10)
Tool Position Init.
Control Steps Per Action
Max Episode Steps
150
Slack Reward
-0,001
Tool Length Ratio
(-0.5,
Tool Length Init.
(2.0, 1.0, 1.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-1.0, 1.0)
Tool Angle Scale
60.0
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design & Control Std.
True
Policy Learning Rate
2e-5
Entropy /
0.01
Value Learning Rate
lc-4
KL Divergence Threshold
0.002
Batch Size
50000
Minibatch Size
2000
PPO Steps Per Batch
10
Table 2: Hyperparameters used for our framework on the ball catching task.
A.2 BASELINES
For each baseline, we perform hyperparameter sweeps for a fair comparison with our framework. The tested hyperparameter configurations for each baseline are listed in Table
13

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Values
Tool Position Init.
(15, 10)
Control Steps Per Action
Max Episode Steps
30
Slack Reward
-0,001
Tool Length Ratio
(-0.7, 0.2)
Tool Length Init.
(6.0, 3.0, 3.0)
Tool Angle Init.
(0.0, 0.0, 0.0)
Tool Angle Ratio
(-0.1, 0.7)
Tool Angle Scale
90.0
Control Log Std.
0.0
Design Log Std.
0.0
Fix Design & Control Std.
True
Policy Learning Rate
2e-5
Entropy |
0.01
Value Learning Rate
3e-4
KL Divergence Threshold
0.1
Batch Size
10000
Minibatch Size
400
PPO Steps Per Batch
10
Table 3: Hyperparameters used for our framework on the scooping task.
Method
Hyperparameters
Values
Population Size
10, 24, 100, 1000
Initial Stdev
0.1. 1.0, 10.0
Center Learning Rate
0.1,
CMA-ES
Covariance Learning Rate
0.01, 0.1, 1.0
Rank ş Learning Rate
0.01, 0.1, 1.0
Rank One Learning Rate
0.01, 0.1, 1.0
Population Size
10, 24, 100, 1000
CEM
Initial Stdev
0.1, 1.0, 10.0
Parenthood Ratio
0.01, 0.1, 0.4, 0.8
Population Size
10, 24, 100, 300
Center Learning Rate
0.001, 0.01, 0.1, 1.0
Stdev Learning Rate
0.001, 0.01. 0.1, 1.0
PGPE
Initial Radius
0.01, 0.1, 1.0, 2.0, 5.0
Table We tune over these values for hyperparameters baseline methods. Bolded values indicate the best performing settings, which use in our comparisons.
14


Reviews:
[{"review_id": "h-Iv-d42Qf", "reviewer": "Reviewer_xbGt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a tool to design policies for solving tasks for different goals. It provides a general framework, that the downstream policy can then use in a RL framework. The paper provides proof of concept demonstrations for designing policies able to solve different manipulation tasks, conditional on the type of goal being used. ", "summary_of_the_review": "My understanding of this paper is very limited and I am not sure if my review should be counted as fair. It is very much possible that the core contribution of this work is beyond the scope of my understanding. The work seems useful, especially if the tools can be used by RL practitioners off the shelf. \n", "strength_and_weaknesses": "1. The paper is a bit difficult to understand in its current form, and it is not clear how these policies are actually designed. It may however be due to my lack of understanding or familiarity with contributions of this type. \n2. The experimental demonstrations considered are well explained - but the exact framework in the way this paper is written, is difficult to understand. \n3. The paper seems to propose a useful contribution, especially based on the claim that the proposed framework and tool can be used for tackling out of distribution goals too. The overall pipeline is a bit difficult to understand, and it would have been useful if a schematic of the overall pipeline could be provided. \n4. I do think this type of work is useful for the community, especially for steps towards making RL useful in the real world.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dvu93W_DOQE", "reviewer": "Reviewer_kbaD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new task of joint tool and controller design. Here policies are required to first produce a tool, and then to use this to solve some goal oriented task. In this work tool production policies essentially predict the parameters of a 3-link chain (angles and lengths), which is then used by the control policy. Results show that this two stage policy optimisation process is more effective than a range of evolutionary and parameter search strategies. The idea is interesting and the motivation very exciting, but the final results slightly underwhelming.", "summary_of_the_review": "Overall, I like the problem set out in the motivation, and think this is interesting. Unfortunately, the solution proposed in this work seems too constrained to have much practical value, and it is unclear that the proposed approach and findings will scale to more complex tool design settings.", "strength_and_weaknesses": "Strengths:\n\nThe paper clearly motivates the need to equip agents with ability to design and use tools (constrained by available materials), and I agree this is an interesting manipulation task. The argument that unlike locomotion, where it suffices to learn and optimise the policy of a single morphology, manipulation with tools requires the ability to design tools is a good one, and I think this will inspire an interesting line of work. \n\nWeaknesses:\n\nUnfortunately, tool construction is neglected by this work, which rapidly simplifies tool design to a problem of regressing the fixed parameters of a highly constrained model (3 link) and then learning a general policy to solve talks requiring relatively simple motion trajectories (using much the same approach as in the locomotion setting, which contradicts the argument that this is somehow different for the tool use case). \n\nThe work only considers very toy problems (Box2d envs), and glosses over the challenge of actually assembling a tool prior to use. I understand that abstractions like this are important to progress algorithmically, but this abstraction and policy choice very conveniently fits existing approaches [Yuan 22] and the fixed 3-link tool removes some of the really interesting aspects of what it means to design a tool with interesting affordances from available materials (eg. the macgyvering approach of Nair et al.). It is also unclear to what extent the proposed approach will scale.\n\nUltimately, this means the experimental results amounts to a comparison between the design, control approach and evolutionary and policy gradient parameter estimation techniques in a very constrained environment, which while interesting, does not really line up with the ultimate motivation of this work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aX-emFDQPr", "reviewer": "Reviewer_BpS2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Goal: creating specialised tools for different manipulation goals.\n\nProposed Solution: A designer policy that is conditioned on the task goal + design agnostic controller policy that can perform manipulation using the designed tools. This paradigm has already been proposed in recent work (Yuan et al.) the authors have cited. The authors extend this to the goal-conditioned setting to accommodate multiple goals within a single manipulation task. (i.e. the designer can immediately design task-appropriate tools, control policy can directly use the designed tool for the task)\n\nDesigner Policy:\nState space consists of observations (positions of velocities of objects in the scene)\nAction space consists of the parameterisation of the tool (designed tool)\nRewards: NO rewards provided\nOne transition is designing one tool in which the MDP switches to the control phase.\nIn this paper, the tool is parameterised by a 5-dimensional vector representing a 3-link rigid chain (length of each link and the relative angles between links)\n\nControl Policy:\nState space consists of the observations + the parameterisation of the tool from the design phase.\nAction space consists of the motor commands applied to the designed manipulation tool.\nReward: receives task reward", "summary_of_the_review": "Extending existing previous work to the goal-conditioned setting seems like a good idea. While the methods seem sound, the experiments and baselines selected and presented and unconvincing and slightly weak for the above reasons listed.\n\n", "strength_and_weaknesses": "Strengths:\n- The novelty of this work lies in extending a previous framework of morphology design and control to the goal-conditioned setting.\n- Proven building blocks which handle this problem well such as GNNs are used.\n- The auxiliary reward used for a tradeoff between material cost and control seems like something practical.\n\n\nWeaknesses:\n- In the introduction, it is not clear what a task/goal is and sometimes this is used interchangeably which can be confusing. The authors work here corresponds to a single task where multiple goals are possible. However, when reading the introduction, I had the impression that the authors were proposing a framework to deal with multiple tasks with one policy. For example, “design appropriate tools when presented with a task”.  But the experiments clearly show it is for dealing with multiple goals (i.e. number of balls) for the same task (i.e. scooping).\n- In section 3.2, it is slightly weird to say the proposed framework is “not limited to the selected design space choice”, given that the authors themselves explain why a low-dimensional parameterisation is used during the design stage to avoid sparse rewards during optimisation.\n- The novelty of this work lies in extending a previous framework to the goal-conditioned setting. However, I find not many details on how this affected the need to change anything in the methods from prior work to accommodate for this. It seems as if the goals were just additionally supplied to the policy with randomised goals as in normal GC-RL. Would appreciate a bit more explanation if any additional things were required, and if not, why not? Would also appreciate some ablations on what was critical to this method working.\n- I find it odd that the pushing task is used in the experiments where the goal space is final puck locations, given that one of the authors' main arguments when motivating the paper is contrasting with locomotion tasks from prior work which usually only involve a single goal. But a locomotion task can also involve moving the various 2-D locations and not just going forward.\n- Could the authors provide some explanation on why only black-box optimisation baselines are used? Given that these methods are generally sampling-based methods, they are usually slightly more sample inefficient so comparing a PPO (policy gradient based - authors method) approach to such baselines would not be completely fair. If the authors think this is fair or there is a limitation on why other baselines can’t be used like goal-conditioned PPO, I would appreciate seeing them explained in the paper. Why couldn’t the previous work which the authors build on, be used as a single goal baseline (since the authors are also comparing to other single baselines)? Likewise, why aren’t more “multi” baselines presented?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject