Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper describes a framework (PreCo) for the joint optimization (co-design) of the voxel-based design and corresponding control policy for soft robots. Integral to the approach is its use of a pre-training phase to jointly optimize design and control on a set of training tasks before fine-tuning on a different target task. The policy is shared across designs and both the design and control are trained via model-free RL. Experiments demonstrate that PreCo outperforms curriculum-learning and meta-learning baselines in terms of zero-shot generalization to the target task as well as few-shot (with the exception of one domain).
review: STRENGTHS
+ The paper is well written and easy to read.
+ The topic of the paper is highly relevant---the problem of joint optimization of robot design and control has garnered renewed interest in the robot learning community within the last few years.


WEAKNESSES
- The significance of the paper's contributions is unclear. It is not the first to propose using a universal controller/policy as part of joint optimization (e.g., Reference [33] in the paper and References [1-4] below) as a means of improving the efficiency and (in some cases) performance of the resulting design. Also relevant, many of these papers similarly motivate the idea of learning a single policy that can control different designs as a form of multi-task RL (e.g., [42] in the paper and [5, 6] below). Instead, the primary novelty lies in optimizing over a set of different tasks as a pre-training phase in an effort to facilitate "transfer" to the target task. The benefits are fairly clear in the context of zero-shot and few-shot transfer, but the results don't demonstrate that such an approach is better than training from scratch on the target task. However, it's not obvious whether there is anything fundamental to the algorithm that enables multi-task pre-training as opposed to training a transformer-based framework that has a universal policy on a set of different tasks. Indeed, there is no discussion of how one might go about deciding on these pre-training tasks when there is a target task in mind. It seems that they were chosen randomly.
- Related, Table 2 only includes one domain from Table 1. It is important to compare the performance of the resulting designs as a result of fine-tuning to those that are trained from scratch with the same architecture as well as a baseline (e.g., see the results in the "Evolution Gym" paper).
- The paper is missing a few references to relevant work in soft robot co-design [1, 2, 5, 7]. Reference [5] seems particularly relevant given the focus on soft robots and the use of a universal controller, though it seems that they only train on a single task.
- The paper doesn't provide sufficient details about the design space
- The paper provides no discussion of how one might extend these results to physical hardware, which is likely difficult given the significant sim-to-real gap, particularly for voxel-based designs and differentiable simulators [3].
- Certain claims in the paper are not supported. This includes the statement that the universal policy "significantly facilitates exploration of the design space". The results show that it improves performance including in a zero-shot setting, but there is no results that show improved exploration.
- Lines 148-150: The statement regarding deficiencies in optimization as performed by existing methods that alternate between updating the design and the control are not substantiated. 
- While the appendix provides the hyper-parameter settings, it would be helpful if there was more discussion of the training procedure in the main paper, including the way in which tasks are chosen during pre-training (i.e., how often and from what distribution (uniform?)), how the method alternates between updating the design distribution and rolling out the current policy, etc.
- Personally, I find the frequent parallels to "brain" and "body", "genome", and "pleiotrophy" to be a distraction since the relationship between biological evolution and what the paper describes is purely superficial.
- Minor: Section 1 (first paragraph): After stating the importance of co-design, the motivating examples that are provided demonstrate the importance of optimizing the design, but not control.
- Minor: Lines 240-241: Any claims about why the model learned a particular design are purely speculative


GRAMMAR:
* Line 187: "Does our method, PreCo, ~~can~~ effectively perform"


REFERENCES

[1] R. Deimel, P. Irmisch, V. Wall, and O. Brock. Automated co-design of soft hand morphology and control strategy for grasping. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.

[2] S. Kriegman, S. Walker, D. Shah, M. Levin, R. Kramer- Bottiglio, and J. Bongard. Automated shapeshifting for function recovery. In damaged robots. in Proceedings of Robotics: Science and Systems (RSS), 2019.

[3] S. Kriegman, A. M. Nasab, D. Shah, H. Steele, G. Branin, M. Levin, J. Bongard, and R. Kramer-Bottiglio. Scalable sim-to-real transfer of soft robot designs. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft), 2020.

[4] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

[5] C. Schaff, A. Sedal, and M. R. Walter. Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer. In Proceedings of Robotics: Science and Systems (RSS) 2022

[6] C. Schaff, and M. R. Walter. N-LIMB: Neural Limb Optimization for Efficient Morphological Design. arXiv:2207.11773 2022

[7] A. Spielberg, A. Amini, L. Chin, W. Matusik, and D. Rus. Co-learning of task and sensor placement for soft robotics. IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1208–1215, 2021
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The problem of co-optimization is interesting and relevant to robot learning and robotics, more generally, The core contribution is the use of a pre-training phase in which the framework performs joint optimization on a set of tasks as a means of facilitating zero- and few-shot transfer to a target task. However, it is not obvious whether there is anything fundamental to the architecture that enables this transfer vs. perhaps being the first to try a transformer-based universal control policy (which is not new) with a series of tasks. Without key insights into how these tasks should be chosen, the significance of the contributions is unclear.

UPDATE AFTER AUTHOR RESOPNSE/DISCUSSION

The authors' response addresses the primary concerns that I had with the paper and as a result, I have increased my score.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The co-design methods in the literature mainly consider discovering an optimal robot morphology and controller for a specific task. Yet, biological morphologies can perform multiple tasks. Drawing inspiration from pleiotropy in biology, this paper proposes a learning framework for designing a modular soft robot capable of performing a wide variety of tasks. The authors achieve fast adaptation to new co-design tasks by leveraging past co-design knowledge and demonstrate the effectiveness of brain-body pre-training. The paper presents simulation results to evaluate the performance of the proposed approach. 
review: The paper is well organized and the writing is clear. The presented methods are sound. The context of this paper is interesting because traditionally co-design methods focus on optimizing for "one robot one task", unlike biological system, while this paper presents a new learning approach aiming to address multiple co-design tasks simultaneously. The authors used modular soft robots to build their case allowing them to construct robot bodies by combining various types of deformable cubes and adjusting their volume to generate control signals. Aiming to design a single modular soft robot that achieves a wide variety of tasks, the paper first train the robot by adapting multi-task reinforcement learning. Then, the robot is further trained on unseen tasks. The authors implemented a new co-design policy, namely pleiotropic co-design policy, allowing them to integrate design and control into a unified decision process. 

The authors illustrate that their proposed method could be more effective in learning and executing multiple tasks against various baseline models in simulation environment. However, the morphologies that their approach generated for locomotion purpose on various terrain profiles seem unconventional and not relevant to legged or wheeled locomotion morphologies. It would be interesting to see that their method converge to an efficient morphology for locomotion. Maybe varying the reward function towards legged or wheeled appendages might serve such a purpose.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Motivated by the idea of facilitating effective body-environment interaction, the paper focus on co-designing both the physical bodies and the control systems that operate on the robot body. Their implementation of "pleiotropic co-design policy" is novel and can generate effective results against similar algorithms.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper studies the problem of automatically synthesizing designs and controllers for soft robots that are capable of generalizing (either zero-shot or with some fine tuning) to new tasks. Specifically, they study the co-design problem for 2D soft robots in environments of varying difficulties. The main innovation in the paper appears to be the use of a single shared backbone network (a transformer) for both the design and control policy networks (rather than separate representations as in prior works). The authors demonstrate better zero-shot and fine-tuned transfer to previously unseen tasks on a range of problems.
review: ### Strengths

- The authors clearly motivate their use of a single backbone network for design and control, grounding it in a biological analogy.
- The proposed method yields impressive empirical results.

### Weaknesses

- The authors identify a number of challenging aspects of the co-design problem, but it is not clear how their approach solves all of these challenges. For example:
	- Challenge: since relatively few design actions occur early in each rollout (compared to far more control actions), there is relatively less reward information on which to train the design policy. The authors claim that a shared backbone helps with this, and their ablation `PreCo-Sep` supports this.
	- Challenge: different design morphologies result in action spaces of different dimension. The authors claim that a transformer architecture helps with this, but they don't explain how. Please add more detail here.
- You say that the genetic algorithm is used to provide an estimate of the upper bound performance for each task, but the GA doesn't seem to learn well on any task. Why is that?
- The limitations section is not very comprehensive. In addition to the two limitations discussed (poor performance on the bridge environment and the risk of destructive mutations/forgetting), I can think of several others:
	- How would you generalize to robots in three dimensions? Would this problem suffer from a curse of dimensionality (e.g. 3D convolutions tend to be substantially harder than 2D convolutions; would a transformer-based architecture share this problem)?
	- How would you fabricate a robot with one of these designs? How could you enforce manufacturability constraints? Are there existing robot architectures or manufacturing methods that are compatible with the types of designs that your method produces?

### Minor comments

- "Transformer" is commonly written lower-case as "transformer" unless at the start of a sentence.
- When discussing your results, please don't use terms like "remarkable" or "astonishing". It's up to the reader whether they are astonished or not.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents a simple idea (a shared design/control policy backbone plus pre-training) that yields good empirical results in simulation. I think the core idea is worth sharing, but I recommend that the authors invest a considerable amount of time in revising and making sure that their explanation is clear. Please take care to disentangle causes and effects in your explanations; there are a number of distinct challenges in co-design, and you propose a number of distinct solutions, so it is important that you help the reader understand which solutions help address which challenges (both through explanation and empirical ablation).

**Update post-rebuttal** I have no further questions and am content to leave my score at weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a novel approach dubbed PreCo, which incorporates brain-body pre-training within the co-design framework for modular soft robots, aimed at bolstering their generalization and adaptability capabilities. The methodology hinges on the encoding of co-design principles into models achieved through pre-training a versatile co-design policy on a wide-ranging set of tasks. This pre-trained co-designer is subsequently employed to generate preliminary designs and control policies, which are then fine-tuned specific to a co-design task. The results reveal not only zero-shot generalization to unseen co-design tasks but also expedited adaptation with substantially fewer iterations. The work provides noteworthy contributions to co-design in robotics, more specifically within the field of modular soft robots.


review: The introduction of the PreCo concept is novel, with practical implications for bolstering generalization within the co-design procedure for modular soft robots. The paper offers incisive empirical evidence concerning the efficacy of brain-body pre-training, zero-shot generalization, and fine-tuning. The structure of the pleiotropic co-design policy is robust and eliminates the requirement for a robot population, leading to enhanced flexibility and sample efficiency. 

Also, a comprehensive and well-organized review of pertinent literature is presented. The experiments conducted on the modular soft robot system are thorough and meticulous, providing a robust foundation for the proposed methodology.

However, the experiments do not encompass a diverse range of tasks, and further investigation into the choice of co-design tasks could serve as an intriguing area for future research. The manuscript could benefit from a comparison or discussion with established baselines in the field, thereby providing a more informed understanding of the degree of improvement proffered by the proposed methodology. Details for replication of the study are limited, and a more exhaustive evaluation and ablation studies of the proposed method are required.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The manuscript provides substantial contributions to the domain of co-design in robotics, particularly regarding modular soft robots, by presenting an innovative and pragmatic methodology, coupled with astute empirical findings and a comprehensive review of relevant literature. However, the study does fall short in certain aspects such as limited implementation details and hardware demo and evaluation studies, which necessitate further clarification.
review_title: null

Meta-review: The paper concerns a novel idea, and provides some working solutions to a long-held issue with robotic design.

The authors have been responsive during the review phase.

It is important to put this work in context of the large body of work on co-optimization that dates back several decades and that has received renewed attention within the last several years.

Furthermore, the biological framing for this work is unwarranted as any biological comparisons are superficial at best, and misleading at worst.
Decision: Accept (Oral)