Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduced a reinforcement learning algorithm that simultaneously optimizes the design as well as the controller of a simulated robot to perform locomotion tasks. The core idea is to train a conditioned policy that performs the task in three stages: 1) morphology design of the robot, 2) design parameter adjustment, and 3) controlling of the robot to perform the task. By integrating the design process into the policy learning framework, they are able to design novel and effective agents to complete a variety of tasks. To support the proposed algorithm, graph neural networks is heavily used to support different morphologies. They further propose a joint-specific architecture to improve flexibility of the network, which improves the performance of the algorithm.
main_review: Strengths:
- The resulting morphologies of the algorithm seem interesting and effective.
- The idea of breaking the training into multiple stages where policy first designs the agent and then controls it is novel.
- Paper is in general well written.

Weaknesses/Questions:
- Not sure how much control a user has over the design space. For example one may want the design to mimic certain animals, or be symmetrical.
- The training for the first two stages would involve a delayed sparse reward signal. Is there any intuition why PPO handles this okay in the presented case? Does the horizon of the first two stages, where no reward is given, impact the learning effectiveness?
- It’d be interested to analyze individual components of the proposed algorithm. E.g. comparing to [1] or [2] assuming a fixed morphology. This will help compare the condition policy approach to a bi-level optimization approach.

[1]. Reinforcement Learning for Improving Agent Design. Ha.
[2]. Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning. Schaff et al.

summary_of_the_review: The paper presents a concrete algorithm with good results in general. It would be helpful to provide some more details and insights in what makes the method work and what are some limitations of the current work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a transform-and-control policy to optimize the robotic agents' designs. Contributions include:
- A novel perspective on agent design: rather than formulating agent design as a bi-level optimization, this paper embeds both design generation and control into a single decision-making process such that both design and control are optimized by the same RL algorithm.
- In this formulation, the training experience from different designs is shared to improve sample efficiency.
- Joint-specialized MLP on top of the GNN policy that further finetunes the control of individual joints.
main_review: Strengths:
- Formulating design and control co-optimization as one sequential decision-making problem is novel.
- The paper's ideas all make sense (transform-and-control policy, skeleton and attribute transform, JSMLP).
- The empirical results look stronger than existing baselines.
- The paper is written to be easy to understand.

Weaknesses:
- Experiments are only conducted on four custom environments. Why not use existing environments from NGE or [1] (see references below)? Also, three random seeds are far below the standard. 
- Little analysis on the empirical results, given no theoretical justification of the algorithm. The analysis can be further enhanced from several aspects: 1) Discussing comparison with NGE (probably the strongest baseline) about similarities and differences and how those differences lead to a huge performance increase; 2) Enriching ablation studies by training with skeleton transformation disabled and attribute transformation disabled respectively.

Concerns:
- I can't entirely agree with the argument that this approach enables first-order optimization of agent design. Technically, both this approach and ES-based methods do not have access to the ground-truth gradient and estimate first-order gradients based on the gathered experiences. So, this approach is still zeroth-order optimization, and it's not appropriate to claim that the sample efficiency comes from the first-order nature of the method.
- While I agree that ES-based methods have a high-dimensional search space for design, your approach does not essentially reduce that search space. Instead, the search space for policy is much larger in your formulation, i.e., the dimension of MDP becomes much higher.
- Could you provide more intuition on the comparison against NGE? By looking at the performance curves, it outperforms NGE by a large margin even without all JSMLPs. Is it because of the new formulation of the design optimization, or the attribute transform is optimized (unlike sampling from uniform distributions in NGE), or other reasons?
- I'm skeptical about the reported performance of NGE because of several reasons: 1) In the original NGE's paper, it outperforms RGS by a large margin, while in Figure 3 of this paper, the improvement is marginal; 2) I believe NGE should be much better than RGS is because NGE also uses GNN policies that allow experience sharing across different designs; 3) If experience sharing via GNN is not effective in NGE, why this is effective in your approach as you claimed? A good way to address my concerns is to probably run NGE's original implementation besides your own implementation and report the performance.

Other suggestions:
- Section 4 can be shortened to include more analysis in Section 5.
- In many practical use cases, probably we already have a decent hand-designed agent at the beginning. If your approach can also effectively improve upon that and is better than other baseline methods, the results will be more solid.
- The result will look even stronger if you can show your method even beat the previous works on continuous design optimization [2,3] (probably with attribute transform only and no skeleton transform).

Typos:
- JSMPLs -> JSMLPs in the caption of Figure 4.

References:
- [1] Gupta, Agrim, et al. "Embodied Intelligence via Learning and Evolution." arXiv preprint arXiv:2102.02202 (2021).
- [2] Luck, Kevin Sebastian, Heni Ben Amor, and Roberto Calandra. "Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning." Conference on Robot Learning. PMLR, 2020.
- [3] Schaff, Charles, et al. "Jointly learning to construct and control agents using deep reinforcement learning." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

--------- Post Rebuttal Update ----------

The authors adequately addressed my main concerns through extra experiments and analysis. 

summary_of_the_review: This paper provides an interesting perspective on efficient agent design with reasonable technical approaches, and the results look empirically good. I recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an algorithm for simultaneous agent design and policy optimization. The choice of the body structure is treated as another action available to the agent. Therefore, the policy is parameterized by graph neural networks (GNNs), and it outputs i) the skeleton structure, ii) node attributes such as bone length, size, motor strength, and iii) motor control commands. Thanks to the parameterization via GNNs, the policy can be trained with PPO. Experiments show that the proposed method outperforms prior approaches, which mainly employ evolutionary methods for optimization, whereas the proposed method leverages more sample efficient policy gradient algorithms.
main_review: Strengths: clear idea, described well, convincing experiments

Weaknesses:
1) Formulations of some claims could be made more neutral. E.g., "improves sample efficiency tremendously" -> substantially, considerably, significantly
2) The paper refers to policy gradients (PG) as "first-order" optimization methods. This may not be exactly technically correct as these methods still only use zero order information about the objective function. The authors are encouraged to consult the recent literature on the analysis of PG methods, e.g., [1].
3) Table 1 in Appendix E shows that N_s=5 skeleton transforms and N_z=1 attribute transform were selected using hyperparameter search. What qualitative results does one obtain if there are more than 1 attribute transform? Why allowing more skeleton transforms performs worse? In principle, it should provide more flexibility. Adding a discussion explaining the effects of these numbers to the main body of the paper would be illuminating.
4) Table 1 in Appendix E shows values -2.3 and 0.0 are selected for diagonal values of covariance matrices Sigma^z and Sigma^e. This seems to be a mistake, because diagonal values should be greater than zero.

[1] Agarwal, A., Henaff, M., Kakade, S., & Sun, W. (2020). Pc-pg: Policy cover directed exploration for provable policy gradient learning. arXiv preprint arXiv:2007.08459.


summary_of_the_review: Strong paper. The proposed approach is novel and can be impactful in other discrete-continuous optimization domains. Experiments validate the advantages of the proposed method. The paper is clear and easy to follow.

====

I am satisfied with the response of the authors and I maintain my score "8: accept, good paper".
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper poses the problem of morphology design for robots as RL training of one joint GNN policy. Their policy first generates the robot's morphology and then evaluates the design with a common behavior policy that conditions on it. The authors also introduce a technique called JSMPL to allow asymmetric morphologies. Experiments in the Mujoco simulator demonstrate a large improvement over evolutionary methods in terms of sample efficiency and final performance (although the latter is less clear, as neither method has clearly converged). Ablation studies show that the GNN architecture is essential, but are less clear about the impact of JSMPL.

main_review: **POSITIVES**

To the reviewers knowledge (which is admittedly limited), this is the first attempt to learn the morphology of a robot this straight forward with RL. The method is easy to understand and improvements over evolutionary baselines appear very significant (even for 3 seeds).

**NEGATIVES**

The reviewer has 4 major complaints:

(1) The reward of the tested environments is not scale invariant. When the agent is allowed to construct a robot with twice the dimensions and torque, the robot will move (approximately) twice as fast and therefore get twice the reward. The reviewer assumes that there is a build in penalty for using large torques, but this is a delicate balancing act that the authors need to at least discuss. An additional figure that shows how the robots' size changes during training would reveal a systematic growth of the robot and put this reviewers mind at ease if that is not the case.

(2) The results are based on 3 random seeds. While this is (sadly) all to common in RL, it severely limits the conclusions one can draw with any certainty from the results. The performance gap to the baselines appears large enough to be significant, and the same holds for the "ours w/o GNNs" ablation, but the ablations without JSMLP have barely non-overlapping standard deviations, which is an insufficient measure of statistical significance for 3 seeds anyway. To make the presented claims about the ablations, the authors need at least (!) twice the number of seeds (if not more).

(3) The JSMLP method is complicated and poorly explained. It is also not clear to the reviewer why asymmetry is inherently favorable for robotics design. Even if one would like to include the possibility, the presented JSMLP method seems very ad-hoc and might have some unintended side effects. For example, the decision to remove one node can change the node index (e.g. from 31 to 21 in Figure 6) without changing anything about the node. The authors are encouraged to look into Relational GCN as an alternative to represent asymmetric morphologies.
 
(4) It is unclear whether GNN actually generalize well (in particular out-of-distribution) over different robot morphologies. Kurin et al. (2021) have shown some very convincing counter-examples and suggest to use attention instead of GNN layers. The authors must discuss this alternative and are encouraged to evaluate their experiments with attention layers to see whether this yields any improvement. 

**DETAILED COMMENTS**

- $r_t$ must be defined
- clarify that in the definition of J, H is a variable (due to episodic environment)
- when you introduce the design D, doesn't it also affect the state and action space?
- you do not explicitly state in the main paper which variant of GNN you use, e.g. which aggregation function
- (eq.7) looks as if $a^e$ and $a^d$ can be executed simultaneously instead of alternatively. Better select one $a$ (over the union of the action spaces).
- it is not clear why you allow the removal of joints, as it could lead to cycles
- it would be good to have a "stop changing the morphology" action instead of always running for $N_s$ and $N_z$ actions
- the "Policy Update" line in Algorithm 1 must be in the first while loop, as PPO first collects the data set M and then uses it to update the model (currently it is called after every episode)
- you should mention before page 7 that your graph has a root node
- it would be nice to run in Figure 3 PPO on some standard morphology in each environment, for example HalfCheetah, Ant, Swimmer and Hopper, to see that your algorithm is a "super-human" designer. In particular Ant would make for an interesting comparison

**REFERENCES**

Kurin, V.; Igl, M.; Rocktaeschel, T.; Boehmer, W., and Whiteson, S. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=N3zUDGN5lO.
summary_of_the_review: The paper introduces the (to the reviewers knowledge) novel concept of learning the morphology with RL instead of evolution. The experiments need more seeds, but the advantage of the presented method appears significant. The method to allow asymmetric morphologies appears less significant and one might find a more elegant way to ensure this property (if it is at all needed). 

The reviewer would be willing to increase the score if the authors could substantiate point (1) and promise to discuss (or convincingly argue against) the other points of criticism.

**POST-REBUTTAL**

While the response does not answer 100% of this reviewers questions, the resulting paper is nonetheless interesting enough, and the results clear enough, to merit publication.  The reviewer is therefore raising the score to 8.
review_title: null

Meta-review: The paper considers the problem of learning both the physical design (morphology and parameters) of a robot together with the corresponding control policy to optimize performance at a target task. Unlike several contemporary methods that formulate this as two separate, but coupled, optimization problems, the paper unifies these decisions into a single decision-making framework. More specifically, a conditional policy learns to first change an agent's physical design (i.e., the morphology/skeletal structure and its associated parameters), and then to control the design. The policy is formulated as a graph neural network, enabling a single policy to simultaneously control robots with different morphologies (and, in turn, different action spaces). Experimental results demonstrate that the approach outperforms recent baselines on a variety of simulated control tasks.

The paper considers an interesting and challenging problem, that of jointly optimizing an agent's physical design and its control policy, an area of research that has received renewed attention of-late. As the reviewers note, the idea of treating design and control in the context of a single decision-making process is novel. The approach is principled and the experimental results largely justify the significance of the contributions. The reviewers agree that the approach is described clearly and that the paper is well written. The reviewers initially raised a few concerns regarding the experimental evaluation, including the desire for more in-depth evaluations and the need for more random seeds. They also questioned some of the claims made in the initial submission. The authors provided a detailed response to each of these points and made changes to the paper to resolve most of the concerns.

In summary, the paper proposes a novel approach to an interesting problem with convincing results.
Decision: Accept (Oral)