Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper describes a method for optimizing a robot design given a task. 

The robot topology is defined as a graph which, in turn, is generated according to a grammar that guarantees only valid robot topologies. 

The grammar is also used to generate a synthetic Dataset of 500k robot topologies, used to train a VAE mapping the graph structure of the robots to a continuous latent space. An additional constraint is imposed that similar points in the latent space produce robots with a similar “shape”.

Bayesian Optimization is used to search over the latent space to generate the robot with the highest performance.
Generated robots are evaluated in Sim against existing baselines.

The controller is computed using a variation of MPC which is the most computationally expensive part of the algorithm.

An ablation study is performed to show the effectiveness of the auxiliary parameter (the Property Predictor) in conjunction with the VAE to locate similar design in the latent space.
strengths_and_weaknesses: ## Strengths:

* The paper is very well written, It’s clear, builds on solid foundations yet performs quite well over other benchmarks. 

* The idea of training a continuous latent space on a variety of robots and being able to use it to optimize for other tasks or other domains altogether (manipulation and so forth) seems particularly powerful and promising.

## Weakness:

* No hardware experimentation
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is very well written, It’s clear, builds on solid foundations yet provides a meaningful increase over the performance metric. The Authors supply the Code to run the experiments and allow the community to keep building upon this work.
issues: The paper lacks an “on robot” experiment; albeit not strictly required, a validation on simple robots made with cheap servo motors and 3D printed limbs could provide stronger evidence on the quality of the algorithm.

Other than the above mentioned issue the Reviewer could not find any meaningful shortcoming to the proposed work.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a robot co-design algorithm that co-optimizes the shape and control of terrestrial robots. This paper is based upon the RoboGrammar paper [5] and uses grammar to constrain the robot topology search space to be compact. The key idea of this paper is to train an autoencoder and decoder for the robot structure, which maps the discrete robot topology into a continuous latent space representation. This encoder-decoder framework enables to optimize in the continuous latent space with bayesian optimization instead of optimizing in the discrete and constrained robot topology space. The authors demonstrate the advantages of the proposed method for the tasks defined in [5] and properly design the ablation study to validate each technical component of the method.
strengths_and_weaknesses: The idea of mapping a discrete topology space into a continuous latent space to simplify the robot design optimization problem is interesting. The method to decode the continuous latent space into the robot structure with a depth-first approach and using teacher-force style training is inspiring. The most appealing part of the algorithm to me is the reusability of the trained autoencoder, which has the potential to apply the pre-trained autoencoder to other tasks/problems.

However, there are a few concerns and questions I have for the authors:
1. To train the autoencoder, the authors collect a large dataset of designs by applying random rules in the RoboGrammar. However, randomly picking the rules to apply will result in a super unbalanced dataset, where the number of simple designs with short rule sequences will dominate the whole dataset. The autoencoder trained with such an unbalanced dataset will probably lead to a biased encoder that only generates simple designs. Does the author do any data balance technique to address this problem? 

2. The robot decoder part (section 3.3) is the most important part of the whole pipeline since it requires a clever way to convert a continuous latent space to valid discrete structure space. More details about the decoder will be necessary for a better understanding and reproducibility of the approach. Specifically, I have the following questions regarding the decoder part:
     1. How does the recorder message passing work? Since the partial graph keeps changing during the decoding phase, do the messages stored at each node get cleared and propagate from scratch every round?

     2. One of the advantages of using grammar is to constrain the robot designs to be valid and manufacturable designs. However from the paper, during the decoding phase, it seems that the algorithm just iteratively adds a node with a certain type to the current partial graph without any verification on whether the design is actually spanned by the grammar. If that is the case, the produced designs can be invalid for real manufacturing thus the applicability of the proposed method to a real robot system is questionable.

     3. In RoboGrammar, the grammar rules are constructed to provide the symmetry of the robot structure. The designs in the associated video also have such symmetry property. From the paper, I didn’t find such symmetry constraints in the decoder part. Could you clarify more on this part?

     4. While it is appealing to convert a discrete parameter space (i.e. robot topology space) into a continuous space (i.e. latent space), it is counter-intuitive. While there are gradients in the continuous space, the gradients are not defined in the discrete space. As an example, if we slightly perturb the latent representation, how will the robot structure produced by the decoder change? 

     5. It is good to visualize the learned latent space. You can take two latent vectors and get intermediate latent vectors by interpolating between these two vectors, and visualize what the corresponding robot structures after decoding look like.

     6. How is the ground truth label is constructed? For a node, if it has multiple other nodes connected to it (e.g. the root node), then there can be multiple choices for the label of the first node starting from this node, which means the ground truth label q is not unique.

     7. The training takes place in a teacher-force style, it will produce aggregation errors during testing. How do the authors resolve this problem?

3. The wall terrain task looks different from the one in RoboGrammar [5] (the wall is narrower in this work), any reasons for that?

4. The comparison to GHS looks questionable. In [5], GHS can achieve much higher rewards after training for 2000 iterations (e.g. over 6 in Flat Terrain Task), which means both the GHS and GLSO have not reached the best design yet after 500 iterations. It would be helpful to train the GLSO and GHS for a long enough time to compare their convergence speed and final performance.

5. How long does the algorithm run? The major advantage of training an autoencoder is that it can generalize to different new tasks without re-training the encoder and saves time. So it would be more supportive if the robot optimization time can be reported.

6. Missing references:
    1. Evolutionary Algorithms:
        > Embodied intelligence via learning and evolution, Gupta et al. 2021
   
        > Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots, Bhatia et al. 2021
       
        > Evolving virtual creatures, Sims 1994
     
    2. RoboGrammar: citation [10] should be included in section 2.2.

    3. Latent Space Optimization: 
        > Emergent hand morphology and control from optimizing robust grasps of diverse objects, Pan et al. 2020
      


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Given the novelty of the method and the potentially promising results and applications, I would suggest weak acceptance for now and will change my score accordingly based on whether the authors address my concerns properly.
issues: All the issues have been mentioned in the above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper describes an approach to improving the computational efficiency of optimization over discrete, grammar-constrained robot design spaces. Fundamental to this approach is using a variational autoencoder to learn a continuous, low-dimensional representation of the design space that can be optimized using Bayesian optimization. Experimental results show advantages over existing baselines along with the value of training the VAE with an additional objective that encourages the learned representations of similar designs to be nearby in the latent space.

The idea of using a learned latent representation to improve the efficiency of optimization is interesting and offers potential advantages over standard methods that involve combinatorial search. As such, this work may prove useful to others in the robot learning community who are interested in design-control optimization. However, the advantages of this approach in terms of computational efficiency are not fully clear.
strengths_and_weaknesses: The paper considers the problem of learning the physical design of (legged) robots optimized according to a given (locomotion) task. Building off of recent work, the paper adopts a graph-based representation of the design space that is governed by a grammar. While the grammar has been shown enable heuristic search strategies, the *discrete* design space remains combinatorially large.

The key contribution of this paper is the proposal to learn a low-dimensional, *continuous* design representation that affords efficient optimization using standard optimization methods (i.e., Bayesian optimization). This representation is learned using a graph neural network-based variational autoencoder (VAE) that seeks to reconstruct a collection of input graphs that model designs randomly generated according to the grammar. In addition to the decoder, the network includes a "property predictor" head that takes as input the learned latent representation and outputs the 2D coordinates of a design's ground contact points. The VAE is trained using a loss that combines the standard VAE objective with the MSE of the predicted coordinates. Having learned the latent representation, the authors use Bayesian optimization to reason over candidate designs and use MPC to control each design.

The method is evaluated in simulation on a series of locomotion tasks and compared to several baselines, including the graph heuristic search proposed in the RoboGrammar paper that this work builds on. The paper then ablates the contributions of the VAE prediction objective as well as the grammar.


STRENGTHS

+ The paper considers an important problem that has attracted a fair bit of attention in the robot learning community of-late.
+ The idea of learning a low-dimensional continuous representation of the otherwise discrete design space as a means of improving efficiency is interesting.
+ The paper is well written and easy to follow.

WEAKNESSES

- The core contribution of the paper is the use of a VAE to learn a continuous design representation as a means of improving optimization efficiency. In its current form, however, the paper lacks a concrete demonstration of these gains in efficiency over baseline methods.
- It is not clear what logic was used to choose the maximum steps for training (500). Comparing the results presented here to those in the RoboGrammar paper [5 (in this paper], Guided Heuristic Search (GHS) reaches ~5.5 reward at ~1200 iterations for Frozen Lake, ~6.15 reward at 2000 iterations (and ~6.0 at ~1000 iterations) for Flat Terrain, ~5.25 reward by 2000 iterations for Ridged Terrain, and ~5.6 reward by 2000 iterations (with a rapid increase around 500 iterations). This may make one wonder whether the choice of 500 was cherry picked. What does the reward for GLSO look like farther into training?
- Given the motivation of improving computational efficiency, it would be helpful to see a visualization of performance as a function of computation time (ignoring the fact that methods like the random baseline could be parallelized).
- The means by which contact points are used to define the additional prediction objective is ad hoc and seems brittle (e.g., it relies on a default configuration of each design). It's also not clear how it would generalize to other domains, while the paper claims that the overall method generalizes across domains.
- The paper omits a discussion of related work in design-control optimization [1,2,3]. This includes existing work that uses a VAE to learn a low-dimensional representation that facilitates the optimization of the design and control of soft robots (i.e., by making the full pipeline differentiable) [1]  and another that employs a variational objective to learn a latent representation suited to control and estimation [2]. Also relevant is work that uses graph neural networks to reason over graph-based representations of robot design spaces [3, 4, 5, 6].
- The limitations discussion is weak. The inability to handle acyclic graphs is listed as a limitation without a discussion of the advantages of being able to handle cycles. A more obvious limitation, which the text only partially alludes to, is the challenge of generalizing this approach to produce physically realizable designs that can be fabricated and controlled in the real world.

QUESTIONS/COMMENTS:

* How was the computational cost of the random baseline balanced with the computation afforded the other methods (e.g., since the random baseline can be parallelized)?
* Lines 47–53: I find this discussion a bit contradictory. The paper states that structurally similar robots may have very different performance and uses this as motivation for using contact point prediction as an auxiliary objective. Wouldn't the same structurally similar designs also have similar contact points?
* Can the authors elaborate on the "computationally expensive training procedures" required of end-to-end methods.
* The VAE is trained on a random set of designs generated according to the grammar. Instead, it would be interesting if this optimization could be focused on higher-performing designs.

MINOR:

* Line 29 (and elsewhere): "e.g." should be followed by a comma
* Line 56: Erroneous space before footnote.
* Footnote 1* (and elsewhere): Numbered citations shouldn't be used as words.
* Line 73: "... when the design space are ..." --> "... when the design **spaces** are ..."
* Line 84: "... in a A* manner, ..." --> "... in **an** A* manner, ..."
* Line 153: $\sigma$ is overloaded.
* Line 186: "KL" should be capitalized.
* Line 209: "... uses Gaussian Process" --> "... uses Gaussian **Processes**"


REFERENCES

[1] Andrew Spielberg, Allan Zhao, Tao Du, Yuanming Hu, Daniela Rus, and Wojciech Matusik, "Learning-In-The-Loop Optimization: End-To-End Control And Co-Design of Soft Robots Through Learned Deep Latent Representations," NeurIPS 2019

[2] Andrew Spielberg, Alexander Amini, Lillian Chin, Wojciech Matusik, and Daniela Rus, "Co-learning of task and sensor placement for soft robotics." IEEE Robotics and Automation Letters 6, no. 2 (2021): 1208-1215.

[3] Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris Kitani. "Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design," arXiv preprint arXiv:2110.03659 (2021).

[4] Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros, "Learning to control self-assembling morphologies: a study of generalization via modularity," NeurIPS 2019.

[5] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler, "Nervenet: Learning structured policy with graph neural networks." ICLR 2018.

[6] Wenlong, Huang Igor Mordatch, and Deepak Pathak, "One policy to control them all: Shared modular policies for agent-agnostic control." ICML 2020.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper describes an approach to improving the computational efficiency of optimization over discrete, grammar-constrained robot design spaces. Fundamental to this approach is using a variational autoencoder to learn a continuous, low-dimensional representation of the design space that can be optimized using Bayesian optimization. Experimental results show advantages over existing baselines along with the value of training the VAE with an additional objective that encourages the learned representations of similar designs to be nearby in the latent space.

The idea of using a learned latent representation to improve the efficiency of optimization is interesting and offers potential advantages over standard methods that involve combinatorial search. As such, this work may prove useful to others in the robot learning community who are interested in design-control optimization. However, the advantages of this approach in terms of computational efficiency are not fully clear.

UPDATE AFTER AUTHOR RESPONSE/DISCUSSION:

I appreciate the authors' response, which addresses some of my initial questions and concerns.

I remain unconvinced by the justification for the choice to stop at 500 iterations as well as the argument against parallelization. I appreciate that running GLSO for a greater number of iterations and for multiple seeds would take more time, but we aren't talking about a lot of time here, particularly if GLSO was run on a machine more powerful than a. desktop, particularly given that the baseline results already exist. Real-world fabrication is not a focus here---the algorithm is not designed with the specific goal of enabling sim-to-real transfer and there is little discussion about how the challenge of realizing designs that are effective in the real-world would be addressed. This is not a criticism of the work, since it is consistent with the large majority of recent approaches to co-optimization, but calls into question the arguments for stopping at 500 iterations and for not considering parallelization.

As a result, I am keeping my score.
issues: See issues discussed above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Design optimization of kinematic robots (e.g. walking robots) is challenging because the design space is highly combinatorial; it is effectively a search over graphs representing kinematic chains. This combinatorial structure has (in prior work) precluded the use of efficient optimization methods over continuous spaces. To solve this challenge, the authors propose to use a graph variational auto-encoder to learn a continuous latent representation of the graph grammar defining a walking robot, then apply Gaussian-process Bayesian optimization to optimize in this continuous latent space. They additionally use a "performance predictor" to regularize the learned latent space to group physically similar robots together. The paper is well-written and includes sufficient experimental evidence to justify the authors' design choices and show an improvement over the state of the art.
strengths_and_weaknesses: Strengths:
- The paper is well-written.
- The challenge of optimizing kinematic robots is clear, and the authors' approach is well-motivated and novel.
- The paper contains sufficient experimental evidence to show that the proposed method outperforms the state of the art, and the ablation studies justify the design choices made.

Weaknesses:
- There is little theoretical justification for the effect of performance prediction on the structure of the latent space. I am curious about questions like, "how does the choice of predicted metric impact the structure of the latent space?" and "is the contact configuration the best metric for this task?". The paper provides sufficient empirical results to justify their heuristic, but I am still curious.
- As the authors note in the paper, the walking robots they study are not terribly realistic and might be difficult to construct and deploy in hardware.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper was fun to read and contains a very interesting core idea (using a learned latent space to enforce a continuous structure on a combinatorial problem) that I hope to see future works build upon.
issues: The biggest issue with this paper is that the details of the encoder and decoder architecture are not very clear, particularly to a reader unfamiliar with graph neural networks. It would be helpful to either reference a clear explanation of this message passing process (perhaps a survey or tutorial) or include a more thorough explanation in the supplementary materials.
review_title: null

Meta-review: metareview: Concluding comments and observations:
The authors present their work clearly and report improvements over baseline evolutionary design approaches.

Pros:
- The work tackles a very challenging robot design optimization problem featuring both discrete and continuous optimization aspects which are tackled in an interesting manner in the proposed architecture with promising numerical results.
-  I personally think the work has merit also as a promising direction for future work to build upon and may generate discussion among the CoRL community.

Cons:
- No real world evaluation of robot designs. All results based on simulation alone.
- Success/Quality of a robot design is considered in terms of success of learning MPC controller for certain tasks only.
- the question of weather evaluation up to only 500 designs/iteration steps is sufficient requires further investigation

The reviewers' opinion was somewhat divided between between 1x strong accept, 2x weak accept and 1x weak accept. While there are certainly aspects that could be further improved, I lean slightly towards accept given the potential of the proposed methodology and its future improvements to robot design.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)