Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: In this work, the authors present learning a robot that learns both to (1) design a tool and afterward (2) to use the tool in order to solve a manipulation task in hand. The manipulation tasks are pushing an object, catching a ball, scooping balls, fetching cubes, and lift cups. In training, they divide the training steps into two phases: "design" and "control". In the design phase the agent optimizes the tool at hand. In the control phase the agent learns how to use the tool. The agent maintain two separate policies for the the two phases. Some points were considered by the authors in order to enable efficient learning such as low dimensional design space but still enabling wide range of tasks, relevant algorithm (PPO) with relevant representation (GNN), and finally, good auxiliary reward. The algorithm was tested both in simulation and on real robot. Comparison to other tool design algorithms was done as well. The results showed that validity of the algorithm, even on unseen tasks.
review: Quality of the paper: 
The paper is of interesting topic as we know that general manipulation (say in a single cell scenario or home robots) may require robots to be able to change their tools on the fly. With that respect I find the paper addresses the right problem which will be of interest in the years to come.Nevertheless, still the examples shown in this paper are of a toy-problem nature and I would expect (knowing that it is very challenging task to make it work) to see more realistic tasks. Scooping of reaching is TMHO quite a synthetic problems and they do not address the main problems of home robots or industrial robots.

Clarity:
The paper is written clearly. 

Originality:
The algorithm to address the dual-nature tasks (design and control) if fairly straight forward. As the authors show in their experiments, they achieve SOTA performance. I believe that these results are authentic. I find the usage of GNN clever. Therefore, all in all this work has some original aspects.

Significance:
As mentioned in the quality section, I tend to asses that this work is significant, although in terms of realism we are not in realistic scenario yet.

Strengths:
The work is "closes the loop" from presenting the task, designing relevant solution, showing the validity in simulation, and implementing the solution on a real Franka Emika Panda robot. The design phase relied on RL is nice and valid approach TMHO.

Weaknesses:
The tasks are partially convincing and I find hybrid approaches of automation + AI more relevant. Also, I'm not sure that RL for control is the best way to handle the control phase. Grasping, reaching, with good perception may be achieved with low sample complexity where RL is not the right tool for control in grasping today (although there is abundant works in this direction by Sergey Levine and collaborators.).
 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I really thank the authors for the paper and appreciate their work. The decision is not trivial in this case. The work is fair and the problem is interesting. But I'm not sure the work addresses (with the current experiments, as detailed above) the bar of CoRL.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper describes a learning method for designing and using tools for robot manipulation. The proposed method could jointly optimize tool design policy and tool use policy for solving tasks that are constrained by the robot's own morphologies. They also show their framework is sample efficient and could be transferred in multi-goal settings in the real world as well.


review: Strengths:

The paper addresses an interesting problem -- robotic tool using and design. And the paper tries to solve it by jointly optimizing the design policy and control policy. The presentation of the method is clear and straightforward. Figures clearly express the algorithm step by step. There are sufficient quantitative results, baseline comparison, and real-world experiments.

Weaknesses and suggestions:

1) The parametrized design morphology for tools is relatively simple and it simplifies the problems as well. Is there a more general way to describe the tool's morphology during design? (like using some 3D mesh)
2) In line 222, "It does so by producing specialized tools to solve each task instance, while the other methods optimize for a single design across all task variations. " The 'single design across all task variations' is confusing, why do baselines only optimize single design across all?
3) In figure5, though the goals changed, I do not understand why the final tool design pattern looks like this. If those goals occur uniformly during training, the output tool pattern should be similar to a parallel gripper, please provide more insights here.
4) Since your tool design is mainly based on geometry and morphology, the lack of considering the tool's stability/ applied force should also be mentioned in the limitation part.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In summary, this paper proposes a framework for --tool-designing and using with RL methods. Though more details and explanations are needed to be added to show the validity of the method, I am leaning toward accepting this work.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The article discusses the potential of using objects from the environment as tools to accomplish tasks beyond the capabilities of humans and animals. It suggests that robots can also benefit from tool use by combining deep learning techniques to optimize both their physical design and control. 

This paper proposes a "designer policy" approach, where a policy conditioned on task information generates tool designs to solve the task, while a separate controller policy performs manipulation using these tools.

The authors present a reinforcement learning framework that demonstrates improved sample efficiency compared to previous methods in scenarios with multiple goals or variants. It also showcases the ability to handle new tasks through zero-shot interpolation or fine-tuning, enabling trade-offs between design and control complexity.
review: The paper presents a framework for learning tool design and use in robotics, specifically for manipulation tasks. The framework consists of a two-phase Markov Decision Process (MDP) representing the design and control phases of the agent's environment. A designer and controller policies are jointly trained using interactions with the environment in both phases.

In the design phase, the designer policy selects the parameters for the tool that will be used throughout the episode. The environment state consists of task observations, such as object positions and velocities, as well as the robot's end-effector position, if applicable. In the control phase, the controller policy generates actions that actuate the previously designed tool. The control state includes the task observations and the design action.

This is a very interesting area, and the setting is also intriguing, where it asks the question of the best tool design for a given task.
However, the whole setup feels like it is more targeting designing experiments (for "inventing" the best tools). This is an area that is being tackled in drug development [1] and as a sequential optimization problem [2]. It is an interesting problem. However, it can be an entirely decoupled problem with nothing to do with the robot arm. The paper uses two different policies \pi_D and \pi_C to model the "designer" and "controller" separately, and it was also found to be the better architecture than the proposed model with shared-arch (i.e., a unified one) (figure 4). The authors argue that:

> The ablation shows the importance of learning separate designer and controller policies. (lines 225-226)

However, what is the size of the network parameters? Does the "shared arch" essentially consists of half the trainable parameters count than the "designer"-"controller" counterpart? Is the "worse performance" contributed by having less trainable parameters? It would also be informative to have the corresponding training/inference time for each comparing model to assess the proposed model's actual performance correctly.

The fact that the solving of the whole robot control problem is decoupled into "designer" and "controller" models suggested that the problem is just being solved separately (and just so happened to be set to optimize in the same iteration step). A possible way for the author to argue their approach is to use some of the more advanced RL algorithms, like TD3, SAC, PPO, Rainbow DQN, etc., as a comparison as opposed to other ad-hoc models. Instead of using a two-stage "designer" / "controller" optimization step, other models can simply be optimizing all parameters jointly (both to control the arm and the design parameters of the tool).


The paper selects a design space for tools composed of rigid links, sufficient for manipulation tasks, and easily deployed in the real world. The policies are trained using proximal policy optimization (PPO), and a graph neural network (GNN) policy and value function architecture is adopted. Randomized goals are sampled from the environment during training in goal-conditioned environments. However, it is very hard to understand the result of the product from the "designer." A better way to illustrate the designer's strength is to directly compare the final tools that each model created for some given tasks (to see why certain tool designs are superior to others). Concerning the videos, perhaps a video to see the evolution of the designed tool over the optimization epoch would also be highly useful to see how the tool evolves over the training horizon.


I also have high doubts about the method used to evaluate the control of the robot. In lines 177-178, it was mentioned that 3D end-effector positions were used to control the arm. Based on the methodology in the paper and the video shown in the supplementary, the pose of the arm never changes. Despite the one short-coming of controlling the robot in world-space instead of configuration space (which is already limiting the expressiveness of controlling the robot), shouldn't the robot at least be controlled with the full SE(3) of the end-effector? 

In addition, I need clarification on why the 2D scenario uses velocity control while the 3D scenario only uses position control. (Both 2D & 3D scenarios have "Scoop" action, which seems like they should be using a similary method to allow for a direct comparison?)

Resource constraints are also considered, and a trade-off between design material cost and control energy consumption is introduced through an auxiliary reward. This reward adjusts the task reward at each environment step based on the utilization of tool components and control velocity.

Lastly, based on the tool design figure given in Figure 2 (which consists of [ l_{1,...,3}, \theta_{1, 2} ] ), I cannot see how the "designer" can possibly come up with the design shown in Figure 9 (bottom row). None of the parameters control the thickness of the "stick", no? Unless each task has its own specific default thickness across the "sticks"? However, if that is the case, that would imply that the author injects bias towards each task (instead of being a unified auto-tool-design framework across tasks).


[1] Réda, C., Kaufmann, E., & Delahaye-Duriez, A. (2020). Machine learning applications in drug development. Computational and structural biotechnology journal, 18, 241-252.
[2] Blau, T., Bonilla, E. V., Chades, I., & Dezfouli, A. (2022, June). Optimizing sequential experimental design with deep reinforcement learning. In International Conference on Machine Learning (pp. 2107-2128). PMLR.
robotics_focus: Irrelevant to robotics
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I think the idea is interesting, but the approach used to tackle the problem seems to be standalone (does not seem to be tackling with robotics in mind). There are also questions on the approach used to evaluate the proposed method (needs more experiments and details)

-----------------------------

**Post rebuttal**

The author had addressed some of my concerns on the contribution and the lack of experimental details / highlights of this work's contribution. I will raise my rating.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This work presents a novel task that unlocks the ability of a embodied agent to design and use tools. The authors propose a two-stage training pipeline consisting of a designer network and a controller network. They train two networks jointly using PPO and implement the network using graph neural network. In the experiment session, the authors test 5 tasks in both simulation and real-world Franka Panda arm setup. To further study the performance of the proposed framework, the authors investigate the sample efficiency and the ability to generalize to unseen tasks by zero-shot generalization and finetuning.
review: This work provides a novel task, which is a possible way of overcoming the morphology limitations of robot. It brings significant insight of how a embodied agent can learn to expand its physical ability to manipulate the world. The method and experiments are clearly described but some parts of the experiment result are missing.
1. Strength:
- A novel task that demonstrates how the embodied agent can learn to build tools to expand its morphology ability
- The generalization is well demonstrated in the experiments.
- The proposed method is straightforward and intuitive.
2. Weakness:
- Different parameterization configurations limit the ability of how the algorithm can generalize to new types of tasks.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This work proposes a novel task that allows the embodied agent to expand its ability by designing task-dependent tool. The idea and methods are described clearly. The limitations are well addressed. Thus, I would recommend a strong accept for this work.
review_title: null

Meta-review: The paper proposes to design the tool for each specific manipulation task by having separate designer and control policies where the designer policy designs the tool and the control policy performs the task using the tool. The approach is evaluated in both simulation and with a real Franka Panda robot arm. The experiments show sample efficiency and transfer to new tasks.
Decision: Accept (Poster)