Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new curriculum-based method for co-designing morphology and control of voxel-based soft robots. This curriculum-based method expands the design space from a small size to the target size using reinforcement learning with a predefined curriculum. To address incompatible state-action spaces, local observations of robot voxels are modeled as a sequence and self-attention is used to control the voxels.
strength_and_weaknesses: Strength: 

+ The paper addresses an important problem on co-design of morphology and control with the potential to greatly impact voxel-based soft robots.

+ The paper identifies several interesting challenges for soft robot co-design, such as the high dimensionality of the joint design and control space and the difficulty in generalizability. Methods are designed to address these specific challenges.

+ The paper is well organized and well written.

Weakness:

- The goal of the paper is the co-design of morphology and control. However, the approach assumes that "The morphology of the robot is unchangeable during the interaction with the environment". While the control policy uses the learned morphology, but how does the control inform or help morphology design?

- In the related work section, the paper argues that methods such as Transform2Act by Yuan et al (2022) have "the limitation of aggregating multi-hop information". In the experiment section, the paper states that CuCo-NCU (CuCo without the curriculum component) is similar to Transform2Act. How is this multi-hop information aggregated by CuCo. Other than the curriculum, what are the key differences between CuCo and Transform2Act?

- While simulators such as Evolution Gym and MuJoCo are helpful to evaluate theories, out of curiosity, how are these simulators, the proposed method, and the experimental results applicable to real soft robots? For example, what are the existing prototypes of real voxel-based soft robots that can use the proposed method for co-design?

clarity,_quality,_novelty_and_reproducibility: This paper introduces a new solution for co-design of morphology and control for soft robots, and originally addresses several key challenges. The paper is well written, and the ideas are clearly justified. The experimental results could be reproduced using the Evolution Gym simulator.
summary_of_the_review: Although several technical details need to be clarified, this paper proposes a new method and addresses several specific challenges for morphology and control of simulated voxel-based soft robots.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the effectiveness of using a predefined curriculum over the size of a Voxel-Based Soft Robot (VSR) in co-training design and control policies simultaneously using RL (PPO). Their results in training a target 7x7 VSR configuration provides strong evidence that such curricula over VSR body size is effective for producing more performant morphologies and associated control policies on 4 environments in simulation. The architectural and training procedure used in this paper is largely taken from Yuan et al, 2022 (Transform2Act).
strength_and_weaknesses: ### Strengths
- The paper is overall well-written with clear figures and diagrams.
- The experimental setting and baselines are clearly communicated and seem appropriate for addressing their research question about whether body-size curricula is useful for co-designing VSR control and morphology.
- The experiments comprehensively ablate each component of the proposed method: the use of a curriculum, the use of self-attention, and whether the morphology or design policy are inherited across curriculum stages.

### Weaknesses
- The study can benefit from an ablation of the choice of using an NCA architecture for the design policy.
- Several key details are left unclear in the current paper, as detailed in the Clarity section.
- A primary weakness with this work is that the results are only validated in simulation. The goal of VSR is to transfer to real robots, and the so the study would be much more impactful if the authors can demonstrate the effectiveness of the learned morphologies and controllers in a sim2real setting.

Lastly, there is important related work around automatic curriculum learning that the authors should cite, especially around unsupervised environment design, which describes the same problem setting that the authors investigate (where the environment is the combination of the task and morphology):
- Matiisen, Tambet, et al. "Teacher–student curriculum learning." IEEE transactions on neural networks and learning systems 31.9 (2019): 3732-3740.
- Portelas, Rémy, et al. "Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments." Conference on Robot Learning. PMLR, 2020.
- Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.
- Dennis, Michael, et al. "Emergent complexity and zero-shot transfer via unsupervised environment design." Advances in neural information processing systems 33 (2020): 13049-13061.
- Jiang, Minqi, et al. "Replay-guided adversarial environment design." Advances in Neural Information Processing Systems 34 (2021): 1884-1897.
- Parker-Holder, Jack, et al. "Evolving Curricula with Regret-Based Environment Design." International Conference on Machine Learning. PMLR, 2022.



clarity,_quality,_novelty_and_reproducibility: ### Clarity
A few points in the paper are unclear:
- The paper uses the term "open-loop controller" for describing prior works. However, how exactly this term is used in this setting and how the proposed method differs from an "open-loop controller" is not clear.
- It is not clear whether the morphology is inherited by inheriting the exact previous design or the design policy weights. This seems like a significant distinction since it seems the design policy can be stochastic.
- Related to the previous point, it is not clear whether the control and design policies are deterministic or stochastic.
- Finally, the paper seems to base their architecture and training procedure on Transformr2Act. The authors could make it clearer which aspects of this paper are novel contributions and which are prior work, on which they build.

### Quality
Overall, this paper is well written with clear figures.

### Novelty
This paper validates the effectiveness of pre-defined curriculum over robot size in VSR design. This is a useful, novel contribution that can inform applications and future research in this area, e.g. applying curricula over other parameters of the design space or applying automatic curriculum learning methods to VSR. 

### Reproducibility
The algorithm, architecture, and hyperparameters are clearly detailed in this paper, making this work, in principle, reproducible.
summary_of_the_review: Overall, this paper provides a clean investigation of the impact of body-size curricula in co-training morphology design and control policies for VSR. The experiment setting is sound, including several VSR environments, key ablations, and sensible baselines. The main finding—that body-size curricula lead to improved morphologies and control policies in simulation—seems valuable in informing future work in this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel curriculum-based co-design method for evolving soft voxel-based robots. The main contributions are:
- A novel curriculum mechanism that effectively evolves robots from simple to large design spaces.
- A transformer-based control policy representation, enabling adaptation to arbitrary dimensional observations due to morphology changes.
- A novel design policy based on Neural Cellular Automata (NCA), making end2end training of both design and control policy possible.
- Impressive visual behavior and empirical performance compared to baseline methods with abundant ablation studies.
strength_and_weaknesses: Strengths:
- A novel and sound technical approach. Each component of the approach is intuitive and effective (curriculum, transformer-based generalizable control policy, NCA-based design policy).
- Good empirical performance with abundant ablation studies (Sec 5.4), which explains the necessity of all the algorithmic components.

Weaknesses:
- This paper only performs evaluations on 4 environments/tasks from Evolution Gym [Bhatia et al. 2021], which is not sufficient enough in my opinion to show the generalizability of the proposed approach. I would suggest at least evaluate on 6-10 environments from Evolution Gym, to convince people that the results are not cherry-picked.
- Limitations or failure cases are not discussed well in the paper, and I think this is crucial. Is there any case in which CuCo fails to beat the baselines? 

Questions:
- By looking at the results, CuCo tends to produce regular shapes for the robot design compared to GA. It does not appear to evolve robots with some thin limb structures. However, my intuition tells me that for some complex tasks (such as climbing/throwing/lifting in environments), thin structures like limbs or fingers are the key to success. Could you please evaluate your method on those environments in Evolution Gym as well and report the performance?
- Could you show the performance/convergence curves for the baseline methods as well?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and sufficiently novel, with reasonable technical approaches. For reproducibility, it's not clear to me - depending on if the authors are willing to release the source code.
summary_of_the_review: This paper proposes a novel method with sufficiently new insights into soft robot co-design. While the approach is theoretically sound, on the practical side, I still have several concerns (evaluations on more environments, discussions on limitations and failure cases).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new co-designing curriculum for optimizing the morphology of a soft robot along with its control. The reviewers unanimously agree that the ideas presented here are quite interesting and impactful for agent design. There are several other technical ideas such as transforming the observations of the robot as a sequence compatible with self-attention. The only concern here is the relevance of voxel-based soft-robot design in simulation for the ICLR community.
justification_for_why_not_higher_score: Scope of this paper is limited to soft robot design and control. This may not be very relevant to the broad ICLR audience.
justification_for_why_not_lower_score: Strong technical ideas and strong experimental results.
Decision: Accept: poster