Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Goal: creating specialised tools for different manipulation goals.

Proposed Solution: A designer policy that is conditioned on the task goal + design agnostic controller policy that can perform manipulation using the designed tools. This paradigm has already been proposed in recent work (Yuan et al.) the authors have cited. The authors extend this to the goal-conditioned setting to accommodate multiple goals within a single manipulation task. (i.e. the designer can immediately design task-appropriate tools, control policy can directly use the designed tool for the task)

Designer Policy:
State space consists of observations (positions of velocities of objects in the scene)
Action space consists of the parameterisation of the tool (designed tool)
Rewards: NO rewards provided
One transition is designing one tool in which the MDP switches to the control phase.
In this paper, the tool is parameterised by a 5-dimensional vector representing a 3-link rigid chain (length of each link and the relative angles between links)

Control Policy:
State space consists of the observations + the parameterisation of the tool from the design phase.
Action space consists of the motor commands applied to the designed manipulation tool.
Reward: receives task reward
strength_and_weaknesses: Strengths:
- The novelty of this work lies in extending a previous framework of morphology design and control to the goal-conditioned setting.
- Proven building blocks which handle this problem well such as GNNs are used.
- The auxiliary reward used for a tradeoff between material cost and control seems like something practical.


Weaknesses:
- In the introduction, it is not clear what a task/goal is and sometimes this is used interchangeably which can be confusing. The authors work here corresponds to a single task where multiple goals are possible. However, when reading the introduction, I had the impression that the authors were proposing a framework to deal with multiple tasks with one policy. For example, “design appropriate tools when presented with a task”.  But the experiments clearly show it is for dealing with multiple goals (i.e. number of balls) for the same task (i.e. scooping).
- In section 3.2, it is slightly weird to say the proposed framework is “not limited to the selected design space choice”, given that the authors themselves explain why a low-dimensional parameterisation is used during the design stage to avoid sparse rewards during optimisation.
- The novelty of this work lies in extending a previous framework to the goal-conditioned setting. However, I find not many details on how this affected the need to change anything in the methods from prior work to accommodate for this. It seems as if the goals were just additionally supplied to the policy with randomised goals as in normal GC-RL. Would appreciate a bit more explanation if any additional things were required, and if not, why not? Would also appreciate some ablations on what was critical to this method working.
- I find it odd that the pushing task is used in the experiments where the goal space is final puck locations, given that one of the authors' main arguments when motivating the paper is contrasting with locomotion tasks from prior work which usually only involve a single goal. But a locomotion task can also involve moving the various 2-D locations and not just going forward.
- Could the authors provide some explanation on why only black-box optimisation baselines are used? Given that these methods are generally sampling-based methods, they are usually slightly more sample inefficient so comparing a PPO (policy gradient based - authors method) approach to such baselines would not be completely fair. If the authors think this is fair or there is a limitation on why other baselines can’t be used like goal-conditioned PPO, I would appreciate seeing them explained in the paper. Why couldn’t the previous work which the authors build on, be used as a single goal baseline (since the authors are also comparing to other single baselines)? Likewise, why aren’t more “multi” baselines presented?
clarity,_quality,_novelty_and_reproducibility: The work is presented clearly - but the introduction could do with some clarification following the comments above. As mentioned above, the novelty is in extending existing work to the goal-conditioned setting. However, there is a lack of focus and explanations in the methods regarding this.
summary_of_the_review: Extending existing previous work to the goal-conditioned setting seems like a good idea. While the methods seem sound, the experiments and baselines selected and presented and unconvincing and slightly weak for the above reasons listed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a tool to design policies for solving tasks for different goals. It provides a general framework, that the downstream policy can then use in a RL framework. The paper provides proof of concept demonstrations for designing policies able to solve different manipulation tasks, conditional on the type of goal being used. 
strength_and_weaknesses: 1. The paper is a bit difficult to understand in its current form, and it is not clear how these policies are actually designed. It may however be due to my lack of understanding or familiarity with contributions of this type. 
2. The experimental demonstrations considered are well explained - but the exact framework in the way this paper is written, is difficult to understand. 
3. The paper seems to propose a useful contribution, especially based on the claim that the proposed framework and tool can be used for tackling out of distribution goals too. The overall pipeline is a bit difficult to understand, and it would have been useful if a schematic of the overall pipeline could be provided. 
4. I do think this type of work is useful for the community, especially for steps towards making RL useful in the real world.
clarity,_quality,_novelty_and_reproducibility: 	- I think the paper proposes novel contributions, but my understanding may be very limited. 

I do have concerns on the reproducibility of this work though, since it is not clear how these tools can be used off the shelf by existing practioners. It might be useful if the paper was written in such a way to show how the tools can be used - although I am not sure whether this contribution is ideal for a ICLR submission or not. 
summary_of_the_review: My understanding of this paper is very limited and I am not sure if my review should be counted as fair. It is very much possible that the core contribution of this work is beyond the scope of my understanding. The work seems useful, especially if the tools can be used by RL practitioners off the shelf.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new task of joint tool and controller design. Here policies are required to first produce a tool, and then to use this to solve some goal oriented task. In this work tool production policies essentially predict the parameters of a 3-link chain (angles and lengths), which is then used by the control policy. Results show that this two stage policy optimisation process is more effective than a range of evolutionary and parameter search strategies. The idea is interesting and the motivation very exciting, but the final results slightly underwhelming.
strength_and_weaknesses: Strengths:

The paper clearly motivates the need to equip agents with ability to design and use tools (constrained by available materials), and I agree this is an interesting manipulation task. The argument that unlike locomotion, where it suffices to learn and optimise the policy of a single morphology, manipulation with tools requires the ability to design tools is a good one, and I think this will inspire an interesting line of work. 

Weaknesses:

Unfortunately, tool construction is neglected by this work, which rapidly simplifies tool design to a problem of regressing the fixed parameters of a highly constrained model (3 link) and then learning a general policy to solve talks requiring relatively simple motion trajectories (using much the same approach as in the locomotion setting, which contradicts the argument that this is somehow different for the tool use case). 

The work only considers very toy problems (Box2d envs), and glosses over the challenge of actually assembling a tool prior to use. I understand that abstractions like this are important to progress algorithmically, but this abstraction and policy choice very conveniently fits existing approaches [Yuan 22] and the fixed 3-link tool removes some of the really interesting aspects of what it means to design a tool with interesting affordances from available materials (eg. the macgyvering approach of Nair et al.). It is also unclear to what extent the proposed approach will scale.

Ultimately, this means the experimental results amounts to a comparison between the design, control approach and evolutionary and policy gradient parameter estimation techniques in a very constrained environment, which while interesting, does not really line up with the ultimate motivation of this work.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the approach seems simple enough to reproduce. I think the proposed problem is somewhat novel, but the solution is relatively basic and doesn't necessarily advance the field substantially because of the constrained nature of the tasks being solved.
summary_of_the_review: Overall, I like the problem set out in the motivation, and think this is interesting. Unfortunately, the solution proposed in this work seems too constrained to have much practical value, and it is unclear that the proposed approach and findings will scale to more complex tool design settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summary
The paper extends a method by Yuan et al. for combining morphology and controller design with design-conditioned control policies. The paper focuses on 2D manipulation tasks with 3 link rigid chains as tools. The proposed approach is compared to a number of baselines.

(b) Strengths
- The paper tackles a very interesting problem
- The paper is well written and easy to follow
- The experimental results clearly show the benefits of the method

(c) Weaknesses
- There are still doubts about the significance of the proposed extension
- As the authors say, the experiments are currently more a proof of concept than showing the full potential of the method (or conversely, it is unclear how well this will scale)
justification_for_why_not_higher_score: None of the reviewers recommends acceptance due to the weaknesses outline above.
Decision: Reject