PreCo: Enhancing Generalization in Co-Design of Modular Soft Robots via Brain-Body Pre-Training
Authors: Yuxing Wang
Shuang Wu
Tiantian Zhang
Yongzhe Chang
Haobo Fu
QIANG FU
Xueqian Wang
Venue: CoRL 2023 Oral (2023)
Keywords: Robot Co-design
Pre-training
Reinforcement Learning
Modular Soft Robots

Abstract: Brain-body co-design, which involves the collaborative design of control strategies and morphologies, has emerged as a promising approach to enhance a robot's adaptability to its environment. However, the conventional co-design process often starts from scratch, lacking the utilization of prior knowledge. This can result in time-consuming and costly endeavors. In this paper, we present PreCo, a novel methodology that efficiently integrates brain-body pre-training into the co-design process of modular soft robots. PreCo is based on the insight of embedding co-design principles into models, achieved by pre-training a universal co-design policy on a diverse set of tasks. This pre-trained co-designer is utilized to generate initial designs and control policies, which are then fine-tuned for specific co-design tasks. Through experiments on a modular soft robot system, our method demonstrates zero-shot generalization to unseen co-design tasks, facilitating few-shot adaptation while significantly reducing the number of policy iterations required.

PreCo: Enhancing Generalization in Co-Design of Modular Soft Robots via Brain-Body Pre-Training
Yuxing Wang', Shuang Wu?, Tiantian Zhang', Yongzhe Chang", Haobo Fu?,
Qiang Fu', Xueqian Wang" 1"
TTsinghua University Tencent AI Lab
Abstract: Brain-body co-design, which involves the collaborative design of con- trol strategies and morphologies, has emerged as promising approach to en- hance robot's adaptability to its environment. However, the conventional co- design process often starts from scratch, lacking the utilization of prior knowl- edge. This can result in time-consuming and costly endeavors. In this paper. we present PreCo, a novel methodology that efficiently integrates train-body pre- training into the co-design process of modular soft robots. PreCo based on the insight of embedding co-design principles into models, achieved by pre-training a universal co-design policy on a diverse set of tasks. This pre-trained co-designer is utilized to generate initial designs and control policies, which then fine-tuned for specific co-design tasks. Through experiments on modular soft robot sys- tem, our method demonstrates zero-shot generalization to unseen co-design tasks, facilitating few-shot adaptation while significantly reducing the number of policy iterations required. Our video is available here.
Keywords: Co-design, Pre-training, Modular Soft Robots
1
Introduction
Nature does not treat the development of the brain and body as separate processes, indicating that cognitive processes are intricately connected to the body and the external environment in which organisms operate [1, 2]. This theory holds significant implications for the robotics community. To enable effective interaction with the environment, it is essential to prioritize the co-design of both physical bodies and control systems of robots. In this work, we consider the co-design of Modular Soft Robots (MSRs), which are promising category of flexible robotic systems that offer designers the ability to construct robot bodies by combining various types of deformable cubes, and the control signals can be generated by adjusting cubes' volume (Figure 1). Currently, the majority of related co-design studies [3, 4, 5] for MSRs mainly focus on "one robot one task", where the primary objective is to discover the optimal robot morphology and controller for a specific task. However, this approach seems diverge from biological morphologies, such as the human body, which inherently possesses the ability to perform multiple tasks.
As matter of fact, even improving a robot morphology for a single task can be highly challenging due to: (1) the presence of a severe combinatorial explosion within the robot design space and (2) the existence of incompatible state-action spaces that necessitate training a separate control policy for each morphology. Consequently, past studies [6. 7, 8, 9, 10] often address these challenges by considering the evolution of body and control as separate processes, directly conducted within the large. nigh-dimensional design space. In other words, these methods typically learn from scratch, neglecting prior co-design knowledge, resulting in costly and inefficient endeavors. But how leverage this knowledge to enhance the co-design process for new applications?
"Correspondence to: Yongzhe Chang and Xueqian Wang (chaggyongzhe, wang.xq) xq] @es..ssinhua.ccu..n
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

soft bridge
Diverse Co- D Design Tasks
Unseen Co- doo-ddsigg Tasks
Jump across stairs
Jump over gaps
Pre- Pre-trained Co-designer
Brain-Bady Pre Training
Brain. Bodny Fine-Tuning
Universal Design-and-
Climb stairs
Control Pollcy
Descend stairs
Generalizable Prototype
Traverse terrain
More terrain
Empty voxe!
Rigid vowel
actuator
actuator
Workflow of PreCo. the heart of PreCo lies universal co-design policy, which un- dergoes pre-training using end-to-end deep reinforcement learning diverse set of co-design tasks. The resulting pre-trained co-designer utilized to generate initial designs and control poli- cies, which are further fine-tuned for unseen tasks.
Figure
In this paper, we propose PreCo, methodology that entails pre-training universal co-design policy to grasp the interdependencies between the robot morphology, control and tasks. Following this foundational step, the policy is utilized to generate initial designs and control strategies, which are then fine-tuned for unseen co-design tasks, thereby reducing the learning burden. Precisely, our approach implies that both the morphologies and control strategies of a robot stem from the same set of parameters. Any mutation the parameter simultaneously affects the two, and they are trained using deep reinforcement learning. In contrast conventional co-design methods that segregate only the optimization processes of the brain and body but also the parameters that generate them, we draw inspiration from pleiotropy in biology, which refers to single gene expressing multiple phenotypic traits [11, 12, 13J. Thus, our method eliminates the need for a robot population and provides empirical evidence of its capacity to enhance the efficiency of the learning process.
Our study offers the following key contributions: Firstly, introduce brain-body pre-training. Subsequently, we present PreCo, novel approach that learns universal design-and-contro) policy capable of handling multiple challenging co-design tasks for modular soft robots. Secondly, using the pre-trained co-design policy, we showcase that properly integrating prior knowledge makes co- design on new tasks easier in several ways: enabling improved sample efficiency, zero-shot general- ization and effective few-shot adaptation, providing the benefit over training from scratch. Thirdly, through empirical analysis, we demonstrate that the shared policy structure of PreCo exhibits greater robustness terms of mitigating premature convergence, resulting in improved exploration and flexibility. Furthermore, work provides the first experimental comparison among meta-learning. curriculum learning and pre-training methods in addressing co-design problems.
2 Related Work
Robot Co-Design The process developing both the physical body and the cognitive capabilities nature is intricately intertwined [14, 15]. To replicate this fundamental principle, robot designers are tasked with concurrently optimizing the morphology and control strategy of robots. In the field of Evolutionary Robotics (ER), researchers have extensively investigated the application of Evolu- tionary Algorithms (EAs) for co-designing robotic systems [6, 8, 10, 16). A prominent focus in this field lies in the representation of robot morphologies. Various approaches, such as generative encod- ing schemes [17, 18), have been explored to facilitate the discovery of novel and efficient designs. Techniques like neural networks [19, 20J, Neural Cellular Automata (NCA) [21, 22, 23] and Compo- sitional Pattern- Prodeunnr Networks (CPPNs) [4, 24] have been employed generate diverse and complex robot morphologies, enabling the exploration of broad design space. In addition, EAs also be integrated with Reinforcement Learning (RL), allowing robots evolve and improve their behaviors through interactions with the environment [3, 25]. These approaches, however, adopt

---

separate parameters for generating the robot morphology and control, which are optimized in a bi- level fashion. Consequently, they rely on population of design prototypes to facilitate exploration, leading to challenges in terms of sample efficiency and computational requirements. In contrast to these population-
approaches, we propose an alternative methodology that utilizes a universal policy representation, enabling the robot morphology and control strategy to be derived from the same set of parameters and jointly optimized. Through empirical experiments, demonstrate that this shared representation facilitates the exploration of the design space, leading to enhanced sample efficiency and increased flexibility.
In addition to EA methods, when we have access to certain aspects of the system's physical dynam- ies, model-based differentiable simulator can be employed to jointly optimize the design parame- ters and control using Back-Propagation Through Time (BPTT) [26, 27, 28, 29, 30, 31J. In contrast, work specifically addresses the model-free setting, where system modeling is not required. To achieve this, Policy Gradient (PG) methods can be used to approximate the gradient of design param- eters or evaluate the fitness of robot morphology [5, 32, 33, 34, 35J. Building upon this technology, PreCo is introduced as a novel approach that distinguishes itself from previous works by tackling a more challenging objective of addressing multiple co-design tasks simultaneously.
Many endeavors have been conducted bring robot co-design to real-world settings, including co-designing soft hands [36], voxel-based soft robots [37, 38], soft robotic fishes [39, 40] and soft legged robots [41]. To enable effective sim-to-real transfer, numerical mathematic techniques like Finite Element Method (FEM) [42] or Material Point Method (MPM) [43] together with a high- quality simulator are required to model and simulate soft-body physics. Besides, factors like material imperfections, air resistance, friction and many others come into play, iteratively refining the design and control algorithms based on real-world feedback is also needed.
Multi-task Reinforcement Learning In this study, we approach the challenge of brain-bodt pre- training by adopting a Multi-task Reinforcement Learning (MTRL) framework, which has garnered considerable interest in the field of embodied intelligence [44, 45, 46, 47, 48]. MTRL involves the training of an agent to perform multiple tasks concurrently, aiming to leverage shared knowledge across tasks for improved learning efficiency and generalization. However, previous research based on the transformer structure primarily focuses on training universal controller for multiple robot bodies [9, 49, 50, 51, 52, 53]. PreCo takes further step by exploring how the intrinsic brain-body connections can be utilized to improve efficiency and generalization when facing new applications. In essence, our method aims to avoid learning from scratch, sharing a similar spirit with curriculum learning [5, 19, 54] and meta-lcarning [25, 55, 56, 57] but differs in its framework.
3 Preliminaries
Reinforcement Learning our study, we approach the problem of brain-body pre-training for a set of К co-design tasks by formulating it as MTRL problem. In the domain of RL, the problem is typically formulated Here, S represents the state space and A represents the action space. The transition function P S x S x A + [0, 1] determines the probability of transitioning from one state to another given specific action. The reward function r(s,a) : S x A + IR assigns numeric value to the state- action pairs, indicating the desirability of taking particular action in a given state, and the discount factor Y E (0, 1] specifies the degree to which rewards are discounted over time. Our goal is to find policy parameters which can maximize the average expected reward across all co-design tasks: K
Markov Decision Process (MDP), defined by 5-tuple (S,A, P.r, r)
* E Eeo 7 (st, dt), here the policy is represented by deep neural network parameterized as (a st), which maps from states to distributions over actions.
We employ Proximal Policy Optimization (PPO) [58], a popular RL. algorithm that is widely used in a variety of robot tasks. The algorithm utilizes a surrogate objective function that approximates the policy gradient, and the objective function of PPO is:
To (a(()))))
((0) = Eg
A. - 3.D (ToLLa ( [-(((s))]]
(I)
Ti (ae St)
3

---

Design
Control
d'
Pollcy
of
Modular observation
Add Norm
Uneor Prosciiio
Position embeddingg
Forward
Feed
Task related "obervation
Add Norm
Multi-Hedd Attansion
Design Action Decoder
Concatenate
Design s!
Figure 2: Architecture of the co-design policy. Our policy is designed with shared structure that influences both morphology and control. It receives unified design-and-comen observations and generates corresponding actions. This policy operates under the framework of reinforcement learning, where the design and control processes are unified as single MDP (gray box).
where A, is the advantage estimation, E, represents the empirical average over a batch of generated samples. By iteratively collecting experiences and optimizing ((). the policy re(a)(s)) is updated the direction of maximizing the cumulative reward.
Transformer Transformer [59] is popular neural network architecture that has revolutionized the domain of natural language processing and computer vision [60, 61, 62, 63, 64), and has become a fundamental component in many cutting-edge models. At its core, transformer employs a pow- erful self-attention mechanism to capture the dependencies and relationships among elements sequence. It allows the model to dynamically allocate attention to different parts the input se- quence based on their relevance. each self-attention layer, attention weights are computed for each element by considering interactions with all other elements in the sequence. These weights play crucial role in aggregating information from the entire sequence, enabling the transformer to generate comprehensive and informative representations for each element.
The transformer architecture well-suited for modular robot systems because agnostic to in- compatible state-action spaces. In our work, we model the co-design of modular soft robots as geuuence-to-squeene task. Under this framework, the local observations from voxels are or- ganized in sequences. By leveraging the self-attentio mechanism, the co-design policy can focus more crucial parts of the state space and capture the internal dependencies between voxels, al- lowing the policy to dynamically adjust its focus depending on the input context, which caters to the need for dynamically accommodating changes in morphologies.
Brain-Body Pre-Training
Our motivation for employing brain-body pre-training is that given the assumption of the existence of underlying structural similarity between the pre-training tasks and the target tasks, properly inte- grating prior co-design knowledge into a universal co-design policy makes robot co-design easier. For instance, if a target task requires the robot to master a complex skill, such as traversing across extremely uneven terrain, this skill can be broken down into some foundational abilities like walk- ing, ascending stairs and surmounting minor obstacles. During pre-training, the universal co-design policy aims to extract basic brain-bode body links from these tasks and merge them. When facing spe- cific target tasks, it is anticipated to leverage the prior knowledge, thereby alleviating the co-design

---

challenge. In the remaining section, we describe details about our universal co-design policy, which is optimized end-to-end through reinforcement learning within a unified state-action space.
Universal Co-Design Policy We start by reviewing co-design methods [5, 35] that utilize RL to approximate the gradient of design and control parameters. At the start of each episode, a dedi- cated design policy takes finite number of actions in order to develop robot morphology, and no reward is assigned to the design policy during this period. Subsequently, the resulting robot is consumed by control policy to collect the environmental rewards, which also provides learning signals for the design actions. Using the RL method, two policies are optimized jointly to maximize the performance for the given task.
However, notable issue of this approach is the presence of imbalanced sample distribution between design and control. While this imbalance might not be immediately evident during the initial stages of learning, where both design and control steps are short, it becomes more pronounced as training advances (the execution steps become much longer). When employing randomly sampled experiences for training, the design policy, given the separate policy representation, tends to receive fewer updates compared to the control policy. Consequently, can quickly become optimized only for limited region around the local morphological optimum, which hinders the effective exploration of the design space, as shown in Section 5.2.
To address this concern, our co-design policy (actor network) is designed to facilitate more infor- mation sharing (Figure 2). While directly representing the intricate interplay between morphology and controller is challenging, we employ shared parameters to implicitly capture their relationships. This approach guarantees that both the "brain" and the "body" of a robot are derived from the same set of parameters and developed together.
Unified State-Action Space Our study focuses on co-designing flexible MSRs comprising various types of blocks, also known as voxels. Each voxel in the design space is represented by a discrete value that corresponds to its material or type of actuator (e.g.. empty voxel=0, soft voxel=1, rigid voxel=2, horizontal actuator=3 and vertical actuator=4). In practice, we employ one-hot encoding to represent these values. The co-design policy is uniformly denoted as n (0(((s)) and integrated into the aforementioned design-and-control MDP. Here, 8, {sy s{} represents the concatenation of the design observation sy and control observation sf at time step in each episode. During the design stage, of of will be set to zero and during the control stage, se of 8; will be consistent with the state of the last design step and unchanged. Precisely, with N denoting the size of the design space (e.g., N- = 25 for a 5 x 5 design space), we define sf = { ,8 ,..., si for voxel i is a vector comprising its type and the types of its Moore neighborhood, as shown in Appendix A. sf the control observation, which comprises local observations from all voxels, denoted as sf { s.). Si {s²" ,88 ..., s.., represents modular observations, where s includes the relative position of each voxel's four corners with respect to the center of mass of the robot. se represents task-related observations, such the terrain information.
de
dx J. where
We utilize two feed-forward neural networks to decode shared information from the transformer- based encoder. The output layer dimension of the design action decoder matches the total number of material types (5 in our work), while the output layer dimension of the control action decoder set to As we model the co-design of modular soft robots as a sequence-to-sequene task, both the design and control actions have length of N. During training, voxels are determined by sampling from categorical distribution, which is formulated based on the output logits. During evaluation, the action corresponding to the highest logit value is selected. Additionally, the control action decoder generates the mean value μ.. By combining it with a constant standard deviation Σ, control signals can be sampled from this Gaussian distribution and then clipped within the range of (0.6, 1.6). which corresponds to gradual contractions/expansions of the actuators. We use action masks inform the policy whether an element in the output sequence an actuator.
Learning Process Based on the unified MDP and standard RI. practices, we use distributed trajec- tory sampling with multiple CPU threads to collect training data. Given that we have K pre-training tasks (or environments in RL terminology), each task is allocated to its respective CPU thread.
5

---

Therefore, K' also signifies the number of threads we deploy. During each RL interaction, the state fed to our policy is represented as [s( [gkasks baask :.., s suckk) ), which can he viewed as a uniform sampling process. Note that the time step ł here may vary among tasks. We train our co-design pol- icy using PPO [58], which is based on the popular actor-critic architecture. The critic network shares the same architecture as the actor network (Figure 2), and it computes the value function, indicating probable policy distribution. Its output is N continuous-valin corresponds to the estimated value of voxel. Here, we represent the overall morphology value by averaging the values across all voxels. With the policy gradient technique, the co-design policy is updated to optimize the predicted morphology value. After completing the pre-training phase, the resulting pre-tree te-irrined co-designer is utilized to generate initial designs and control policies, which are subsequently fine-tuned using PPO again for unseen tasks.
vector where each element
Experiments
In this section, we experimentally evaluate our proposed approach to answer the following ques- tions: (1) Does method, PreCo, effectively perform brain-body pre-training and discover robot morphologies capable of executing multiple tasks? (2) How well does our method demonstrate zero- shot generalization and brain- body fine-tuning capabilities when faced with unseen co-design tasks? (3) What is the impact of the unified policy representations on the performance of PreCo?
5.1 Environments and Implementation
Based on the Evolution Gym platform [3], we establish modular robot state-action space? that supports brain-body pre-training described in Section 4. Our focus lies a fixed design space of size 5 x which includes 9 locomotion co-design tasks with open-ended environments: Walker-v0 (easy), HhatformJumper-e (hard), UpStepper-v0 (medium), Obstacle Traverser-v0 (medium), SridgeWalker-ye (easy), Gap.umper-vv (hard), DownStepper-vd (easy),
(hard) and Hurdler-v0 (hard), as shown in Figure 1. The difficulty levels are determined based on the performance of evolution-based co-design algorithms from the platform. For more information regarding the environmental details, please refer Appendix
We work the assumption that there are structural similarities between the pre-training and target co-design tasks. Guided by this understanding, we select the first four tasks for pre- pre-training. The tar- get tasks in our study essentially encompass different adaptations of the pre-training tasks, including the following types: (1) More challenging scenarios, such as the transition from ObstacleTraversee v0 to Traverssrra where the terrain becomes increasingly uneven; (2) Transfers com- parable difficulty, as seen when moving from Walker-vO Bridge Walkeraller--v (with a shift to softer terrain) from PlatformJumper-v0 to GapJumper-vO, wherein the gap between steps expands but the height of these steps reduces; (3) "Reverse" scenarios, exemplified by the transition from UpStepper-v0 to DownStepper-vO, where the direction of the steps is inverted.
We compare PreCo against the following baselines that are also suitable for multiple co-design tasks: (1) PreCo-Sep. which is based on PreCo but utilizes separate transformer- baased design and control policies. This baseline allows to investigate the effectiveness of the unified policy representation; (2) CuCo [5], a curriculum-based co-design method that consists of separate NCA-based design policy and transformer-based control policy. We set the curriculum of CuCo to be 3 x 3 5 x 5; (3) MeCo, which adopts the same network architecture as PreCo but is trained using Reptile [65], popular meta-learning method. We use a 3-layer transformer encoder and run all experiments with the same number of policy iterations. More implementation details can be found in Appendix C.
5.2 Results
Brain-Body Pre-Training We show the learning curves and converged robot morphologies of all methods in Figure 3. For each method, the learning curve is reported over 7 different runs. The
hatps://github.com/ooos Wang- hatea///otttnaaaaaaeaettttttaaon

---

Training Tasks
Training Tasks
Awerage Performnce
Performancs
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Iterations (X10")
keeations
(x10")
PreCo
CuCo
PreCo-Sep
MeCo
Figure 3: Learning curves and converged morphologies of brain-body pre-training. In the left figure, demonstrate the mean and standard deviation of average task performance against the number of policy iterations for all methods. The right figure displays the individual learning curves of PreCo. The bottom figure shows representative converged morphologies from each method.
Unseen Co-Design Tasks
8
Perfommnce
BridgeWalkor-4
DownSteppar-v0
Gap.luppr-40
Figure 4: Evaluation of zero-shot generalization. The height of each bar represents the average zero-shot performance method, and error bars indicate the corresponding standard deviation.
figure clearly demonstrates that proposed method, PreCo, outperforms the baselines in terms of learning speed and final performance. A comparison of the robot morphologies designed by each method reveals intriguing distinctions. Both PreCo and PreCo-Sep are capable of discovering robot bodies with rigid legs, which are essential for maintaining balance in various locomotion tasks. PreCo, in particular, exhibits the ability to utilize empty voxels, resulting robots with serrated and hollow structures that potentially contribute to enhanced performance. On the other hand, CuCo produces robots with several repeated body segments. Although it benefits from eurriculum learn- ing, evident from its learning curve, it converges to local morphological optimum with lower performance. As for MeCo, the "meta-bodies" it discovered seem to prefer horizontal actuators, which may not be efficient in certain jumping tasks. However, our primary interest lies in evaluating its performance on the unseen co-design tasks.
Zero-Shot Generalization One of our objectives in this work train a co-design policy capable of generating single modular soft robot that generalize unseen co-design tasks. Figure 4 illustrates that PreCo consistently achieves higher zero-shot performance across environments
7

---

Table 1: Final performance across environments and baselines, each result is reported over differ- ent runs, Methods on the left side of the divider are fine-tuned from their corresponding pre-trained models, while those on the right side are trained from scratch on target tasks.
Environment
PreCo-FT
PreCo-Sep-FT
MeCo-FT
CuCo-FT
PreCo-Scratch
GA
BridgeWalker-vO
3.69 0.63
3.11 1 0.91
4.75 0.62
4.37 1 0.61
4.81 1 0.21
5.31 1: 0.25
Hurdler-vO
4.10 0.52
1.73 2.19
1.88 1.30
1.66 1 1.24
2.76 1 1.39
1.51 1: 0.25
DownStepper-vo
9.01 0.02
8.90 0.01
4.58 4.74
8.65 0.33
8.98
0.01
8.91 =: 0.01
GapJumper-vO
5.78 1.25
2.58 +: 0.19
2.83 f 0.50
3.58
1: 0.87
3.48 + 0.66
3.35
1 0.15
DostacleTraverser...
4.88 0.12
3.88 1 0.69
3.03 0.50
2.00 1 0.77
3.38 1 0.69
2,98 1 0.63
when compared to the baseline methods. Figure 7 in Appendix D shows that the robot designed by PreCo demonstrates the ability to employ the skill of somersaulting for traversing challenging terrains, relying on its environmental comprehension. Furthermore, it exhibits an understanding of the necessity to lean back to preserve stability while descending stairs.
Brain-Body Fine-Tuning Besides the evaluation of zero-shot generalization, also consider more general setting that allows the co-design policy to fine-tune its parameters to adapt to target tasks. We aim to investigate whether brain-body fine-tuning is better than training from scratch. Keeping this in mind, we introduce PreCo-Scratch and GA [3] as additional baselines, all of which are trained from scratch on target tasks. Here, we limit the number of brain-body fine-tuning it- erations to 300 and policy iterations of learning from scratch 2000. Table presents all results across unseen environments. It is evident that PreCo outperforms the baseline algorithms in most environments. For morphological results, Figure in Appendix D demonstrates that PreCo exhibits intelligent behavior by retaining the beneficial serrated structure for effective stair climbing and obstacle traversal, while also making adaptive modifications to suit the new environment.
The Shared Policy Representation To go a step further, in Appendix E, we provide a performance comparison between PreCo and PreCo-Sep when trained from scratch across 10 co-design tasks (Table 3). Figure 9 and Figure 10 illustrate their learning processes in complex task, Climber-vO, which requires the policy to have a good exploration ability grow irregular structures. Clearly, PreCo exhibits the ability to explore beyond the local morphological optimum, allowing it to develop thin limbs that aid in climbing. In summary, the shared policy representation creates additional opportunities for exploring the design space. This is because, as the parameters of the "control policy" undergo adjustments, the "design policy" concurrently updated.
Limitations and Conclusion
We have introduced PreCo, a co-design method that utilizes brain-body pre-training to generate modular soft robots capable of performing multiple tasks. Through the adoption of shared policy representations, which capture the inherent brain-body connections across various co-design tasks, have observed its favorable zero-shot generalization and few-shot adaptation capabilities in ad- dressing previously unseen co-design tasks.
There are a number of areas for improvement. As shown in Table 1, our method does not perform very well in Walker This might be because the selection of training tasks does not cover the variation of soft terrain, potentially leading ambiguities in the co-design policy. Exploring the selection of pre-training co-design tasks could be interesting for future research. Addition- ally, although our policy representation appears to facilitate the learning process, it is worth noting that destructive mutations of the network parameters can still occur. Further investigation into the mapping of this policy would be valuable. In our paper, we tested our method using a simulator with relatively fundamental modules as a proof of concept to show its effectiveness, developing a general pipeline for translating learned models to physical will definitely be our next step. We also provide detailed discussion of this sim-to-real issue in Appendix and envision our method serving foundation for subsequent research which tends make the co-design of modular robots more practical both in the simulation and the real world.
8

---

Acknowledgments
We sincerely thank the anonymous reviewers for their helpful comments in revising the paper. This work supported by the National Key R&D Program of China (202YFFB4701400/44014444)
References
[1] H. Lipson, V. SunSpiral, J. Bongard, and N. Cheney. On the difficulty of co-optimizing morphology and control in evolved virtual creatures. In IEEE Symposium Artificial Life, 2016.
[2] R. Pfeifer, F. lida, and M. Lungarella. Cognition from the bottom up: on biological inspiration, body morphology, and soft materials. Trends in Cognitive Sciences, 18:404 413, 2014.
[3] J. Bhatia, H. Jackson, Y. Tian, J. Xu, and W. Matusik. Evolution gym: A large-scale benchmark for evolving soft robots. In NeurlPS, 2021.
[4] N. Cheney, C. Bongard, V. SunSpiral, and H. Lipson. Scalable co-optimization of morphol- ogy and control in embodied machines. Journal of The Royal Society Interface, 15, 2018.
[5] Y. Wang, S. Wu, H. Fu, Q. Fu, TT Zhang, Y. Chang, and Wang. Curriculum- based co- design of morphology and control voxel-based soft robots. In The Eleventh International Conference Learning Representations, 2023.
[6] K. Sims.
Evolving 3d morphology and behavior by competition. Artificial Life, 1:353-372,
1994.
[7] N. Cheney, R. MacCurdy, J. Clune, and H. Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. In GECCO '13, 2013.
[8] E. Medvet, A. Bartoli, F. Pigozzi, and M. Rochelli. Biodiversity in evolved voxel-based soft
robots. Proceedings the Genetic and Evolutionary Computation Conference, 2021.
[9] T. Wang, Zhou, S. Fidler, and J. Ba. Neural graph evolution: Towards efficient automatic robot design. ArXiv, abs/1906.05370, 2019.
[10] T. F. Nygaard, D. Howard, and K. Glette. Real world morphological evolution is feasible. Pro- ceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, 2020.
[11] G. Williams. Pleiotropy, natural selection, and the evolution of senescence. Evolution, 11, 1957.
[12] N. Solovieff, C. Cotsapas, P. H. Lee, S. M. Purcell, and J. W. Smoller. Pleiotropy in complex traits: challenges and strategies. Nature Reviews Genetics, 14:483-499 2013.
[13] D. Marzougui, M. Biondina, and F. Wyffels. A comparative analysis genome pleiotropy for evolved soft robots. Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2022.
[14] M. Farina. Embodied cognition: dimensions, domains and applications. Adaptive Behavior, 29(1):73-88, 2021.
[15] A. Cangelosi and M. Asada. Cognitive robotics. MIT Press, 2022.
[16] J. Nordmoen, F. Veenstra, K. O. Ellefsen, and K. Glette. Quality and diversity in evolutionary modular robotics. 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 2109-2116, 2020.
[17] F. Veenstra, A. Faiña, S. Risi, and K. Støy. Evolution and morphogenesis of simulated modular robots: A comparison between direct and generative encoding. In EvoApplications, 2017.

---

[18] E. Samuelsen, K. Glette, and J. Tørresen. A hox gene inspired generative approach to evolving robot morphology. In Annual Conference on Genetic and Evolutionary Computation, 2013.
[19] R. Wang, Lehman, J. Clune, and O. Stanley. Poet: open-ended cocvolution of environ- ments and their optimized solutions. Proceedings of the Genetic and Evolutionary Computa- tion Conference, 2019.
[20] K. Walker and H. Hauser. Evolution of morphology through sculpting in voxel based robot. In ALIFE, 2021.
[21] A. Mordvintsev, E. Randazzo, E. Niklasson, and M. Levin. Growing neural cellular automata. Distill, 2020.
[22] Sudhakaran, E. Najarro, and S. Risi. Goal-guided neural cellular automata: Learning to control self-organising systems. ArXiv, 0alss22055068000 2022.
[23] R. B. Palm, M. G. Duque, S. Sudhakaran, and S. Risi. Variational neural cellular automata. ArXiv, 155/2201.12660 2022.
[24] F. H. K. dos Santos Tanaka and C. C. Aranha. Co-evolving morphology and control of soft robots using single genome. 2022 IEEE Symposium Series on Computational Intelligence (SSC1), pages 1235-1242, 2022.
[25] Á. Belmonte-Bacza, J. Lee, G. Valsecchi, and M. Hutter. Meta reinforcement learning for optimal design of legged robots. IEEE Roboties and Automation Letters, 112134-12141, 2022.
[26] T.-H. Wang. P. Ma, A. E. Spielberg. 7. Xian, H. Zhang, J. B. Tenenbaum, D. Rus, and C. Gan. Softzoo: A soft robot co-design benchmark for locomotion in diverse environments. arXiv preprint arXiv:2303.09.. 09555, 2023.
[27] P. Ma, T. Du, J. Z. Zhang, K. Wu, A. Spielberg, R. K. Katzschmann, and W. Matusik. Diffaqua: differentiable computational design pipeline for soft underwater swimmers with shape inter- polation. ACM Trans. Graph., 40:132:1-132:1 2021.
[28] M. Bächer, E. Knoop, and C. Schumacher. Design and control of soft robots using differen- tiable simulation. Current Robotics Reports, 2(2):21 1-221, 2021.
[29] J. Z. Zhang, Y. Zhang, P. Ma, E. Nava, T. Du, P. Arm, W. Matusik, and R. K. Katzschmann. Sim2real for soft robotic fish via differentiable simulation. arXiv preprint arXiv:2109.1488 2021.
[30] Y. Hu, J. Liu, A. Spielberg, B. Tenenbaum, W. T. Freeman, Wu, D. Rus, and W. Ma- tusik. Chainqueen: real-time differentiable physical simulator for soft robotics. 2019 International conference on robotics and automation (ICRA), pages 6265-6271. IEEE, 2019.
[31] T. Du, J. Hughes, S. Wah, W. Matusik, and D. Rus. Underwater soft robot modeling and control with differentiable simulation. IEEE Robotics and Automation Letters, 6(3):9994-501,, 2021.
[32] K. S. Luck, H. B. Amor, and R. Calandra. Data-efficient co-adaptation of morphology and hehaviour with deep reinforcement learning. In CoRI., 2019.
[33] C. B. Schaff, D. Yunis, A. Chakrabarti, and M. R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pages 9798-9805, 2019.
[34] D. R. Ha. Reinforcement learning for improving agent design. Artificial Life, 25:352-365, 2019.
10

---

[35] Y. Yuan, Y. Song, Z. Luo, W. Sun, and K. M. Kitani. Transform2act: Learning a transform- and-control policy for efficient agent design. ArXiv, abs/2110.036544 2022.
[36] R. Deimel, P. Irmisch, V. Wall, and O. Brock. Automated co-design of soft hand mor- phology and control strategy for grasping- 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1213-1218, 2017. URL https://api.
semanticscholar arg/CorpusID:300706
[37] S. Kriegman, S. J. Walker, D. Shah, M. Levin, R. Kramer-Bottiglio, and J. C. Bongard. Automated shapeshifting for function recovery in damaged robots. Robotics: Science and Systems XV, 2019. www.11667//s.22111x......
[38] S. Kriegman, A. M. Nasab, D. S. Shah, H. Steele, G. Branin, M. Levin, C. Bongard, and R. Kramer-Bottiglio. Scalable sim-to-real transfer of soft robot designs. 2020 3rd IEEE Inter- national Conference on Soft Robotics (RoboSoft), pages 359-366, 2020.
[39] S.-D. Gravert, M. Y. Michelis, S. Rogler, D. Tscholl, T. Buchner, and R. K. Katzschmann. Planar modeling and sim-to-real of tethered multimaterial soft swimmer driven by peano- hasels. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9417-9423. IEEE, 2022.
[40] E. Nava, J. Z. Zhang, M. Y. Michelis, T. Du, P. Ma, B. F. Grewe, W. Matusik, and R. K. Katzschmann. Fast aquatic swimmer optimization with differentiable projective dynamics and neural network hydrodynamic models. In International Conference on Machine Learning, pages 16413-16427. PMLR, 2022.
[41] C. Schaff, A. Sedal, and M. J. Walter. Soft robots learn crawl: Jointly optimizing design and control with sim-to-real transfer. Robotics: Science and Systems XVIII, 2022. doi: 1el:10.15677 rss. 2ss.2022.xvii.062.
[42] M. Dubied, M. Y. Michelis, A. Spielberg, and R. K. Katzschmann. Sim-to-real for soft robots using differentiable fem: Recipes for meshing, damping, and actuation. IEEE Robotics and Automation Letters, 7(2):5015-5022, 2022.
[43] A. Spielberg. A. Amini, L. Chin, W. Matusik, and D. Rus. Co-learning of task and sensor placement for soft robotics. IEEE Robotics and Automation Letters, 6(2): 122:112081 2021.
[44] Y. W. Teh, V. Bapst, W. M. Czamecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. M. O. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. In NIPS, 2017.
[45] I. Espeholt, H. Soyer, R. Munos, Simonyan, V. Mnih, T. Ward, Doron, V. Firoiu, T. Harley, Dunning, S. Legg, and K. Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ArXiv, abs/1802.01561. 2018.
[46] M. Hessel, H. Soyer, L. Espeholt, W. M. Czarnecki, S. Schmitt, and H. V. Hasselt. Multi-task deep reinforcement learning with popart. In AAAI Conference on Artificial Intelligence, 2018.
[47] T. Yu, S. Kumar, A. Gupta. S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. ArXiv, abs/2001.06782. 2020.
[48] 7. Xu, K. Wu, Che, Tang, and Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. ArXiv, abs/2010.07494, 2020.
[49] A. Sanzhez- N. O. Heess, J. T. Springenberg. J. Merel, M. A. Riedmiller, R. Had- sell, and P. W. Battaglia. Graph networks as learnalle physics engines for inference and control. ArXiv, abs/1806.01244 2018.
[50] W. Huang, I. Mordatch, and D. Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In ICML, 2020.

---

[51] A. Gupta, L. J. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers. ArXiv, abs/2203. 11122033.1111,. 11931, 2022.
[52] B. Trabucco, M. Phielipp, and G. Berseth. Anymorph: Learning transferable polices by infer- ring agent morphology. In ICML, 2022.
[53] G. Cheng, Dong, W. Cai, and C. Sun. Multi-task reinforcement learning with attention- based mixture of experts. IEEE Robotics and Automation Letters, 8:3811-3818 2023.
[54] R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. Li, J. Clune, and K. O. Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In ICML, 2020.
[55] T. Anne, J. Wilkinson, and Z. Li. Meta-reinforcement learning for adaptive motor control in changing robot dynamics and environments. ArXiv, abs/2101.07599, 2021.
[56] Yu, D. Quillen, Z. He, Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2019.
[57] K. Rakelly, A. Zhou, Quillen, C. Finn, and S. Levine. Efficient off-policy meta- reinforcement learning via probabilistic context variables. ArXiv, abs/1903.08255 2019.
[58] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. ArXiv, abs/1707.06347. 2017.
[59] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. ArXiv, abs/1706.03766 2017.
[60] K. He, X. Chen, Xie, Y. Li, P. Doll'ar ar, and R. B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pages 15979-15988, 2021.
[61] Z. Liu, Lin, Y. Cao, H. Hu, Y. Wei, Zhang, S. Lin, and B. Guo. Swin transformer: Hier- archical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992-10002, 2021.
[62] C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910. 10683, 2019.
[63] Z. Dai, Z. Yang, Y. Yang. J. G. Carbonell, Q V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. ArXiv,
2019.
[64] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. ArXiv, abs/2004.051500 2020.
[65] A. Nichol and J. Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint 1rXiv:180.002999 2(3):4, 2018
[66] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. Vetoto, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang. Bai, and S. Chintala. Pytorch: An imperative style, performmree deep learning library. In NeurIPS, 2019.
[67] A. Antoniou, H. Edwards, and A. Storkey. How to train your maml. In Seventh International Conference on Learning Representations, 2019.
[68] C. Zhang, P. Zhu, Y. Lin, Z. Jiao, and J. Zou. Modular soft robotics: Modular units, connection mechanisms, and applications. Advanced Intelligent Systems, 1900166,, 2020.
12

---

[69] M. Tebyani, A. Spaeth, N. Cramer, and M. Teodorescu. A geometric kinematic model for flexible voxel-based robots. Soft Robotics, 10(3):11-5226, 2023.
[70] Legrand, S. Terryn, E. Roels, and Vanderborght. Recontigurable, aulti-material, voxel- based soft robots. IEEE Robotics and Automation Letters, 8(3):1255-13) 1262, 2023.
[71] N. Kellaris, V. Gopaluni Venkata, G. M. Smith, S. K. Mitchell, and C. Keplinger. Peano-hasel actuators: Musele-mimetic, electrohydraulic transducers that linearly contract on activation. Science Robotics, 3(14):ear32776, 2018
[72] S. Kriegman, S. Walker, D. S. Shah, M. Levin, R. Kramer-Bottiglio, and J. C. Bongard. Auto- mated shapeshifting for function recovery in damaged robots. ArXiv, abs/1905500924, 2019.
A Parameterization of the Design Space
Sliding window
Padding
Cropping
Flatten
0,0.0.33,,2222
0,0.0.00..33..222222202
4.0.0.00..000..0.0
x
x
25
Moore Neighborhood
25 x x
Empty voxel=0
voxel=1
Rigid voxel=2
Horizontal actuator=3
Vertical
Figure 5: Parameterization of the design space. Initially, the design space is surrounded by empty voxels. Each voxel denoted by a discrete value, reflecting its material characteristic. Then, sliding window is used get each voxel's 's local state, which is composed of its type and the types of its Moore neighbors. Finally, the design state is formulated as an ordered sequence.
B Environment Details
Our experiments are based on the simulation platform from [3, 5]. In this section, we provide additional details of the used environments (Figure 6).
Position. pº is 2-dim vector that represents the position the center mass of an object the simulation at time ph and p are æ and y components of this vector, respectively. pa calculated by averaging the positions all the point-masses that make up object time t.
Velocity, v" is 2-dim vector that represents the velocity of the center of mass of an object in the
simulation at time vo and oy are x and y components of this vector, respectively. t calculated by averaging the velocities of all the point-masses that make up object at time
Orientation. @ 1-dim vector that represents the orientation of object in the simulation time t. Let p. be the position of point mass of object o. 0" is computed by averaging over all i the angle between the vector - p" at time t and time O. This average is weighted average weighted by Ip, - - at time 0.
Other observations. (d) a vector length (2d+1) that describes elevation information around the robot below its center of mass. More specifically, for some integer S d, the corresponding entry in vector (d) will be the highest point of the terrain which is less than P between range of [z, x + 1] voxels from ph in the =-direction.
13

---

BridgeWalker-v0
Walker-v0
DownStepper-v0
PlatformJumper-v0
UpStepper-v0
Gaplumper-v0
Hurdler-vO
Obstacle Traverser-v
Figure 6: Visualization of all environments used in our work.
B.1 Walker-v0
In this task, the robot is rewarded by walking as far as possible on flat terrain. The spec-fie observation is robott and the reward R is:
R Apre rabot
(2)
which rewards the robot for moving in the positive x-direction. The robot receives reward of 1 for reaching the end of the terrain. The episode duration reaches a 500 time steps.
B.2 BridgWalkers
In this task, the robot is rewarded by walking as far as possible on a soft rope-bridge. The task- specific observation is robot (robott. 3, and the reward is:
R == Ap robot
(3)
which rewards the robot for moving in the positive r-direction. The robot receives reward of 1 for reaching the end of the terrain. The episode duration reaches a 500 time steps.
B.3 Upstepper-v0
In this task, the robot climbs up stairs of varying lengths. The task-specific observation is formed by concatenating vectors robot grobot h robot (5)). and the reward is:
R Ap robot
(4)
which rewards the robot for moving in the positive x-direction. The robot also receives one-time reward of 2 for reaching the end of the terrain. The episode duration reaches 600 time steps.
B.4 Downstepper-v0
In this task, the robot climbs down stairs of varying lengths. The task-specific observation is formed by concatenating vectors robot grabot krobol (5)). and the reward R is:
R Ap obot
(5)
14

---

which rewards the robot for moving in the positive r-direction. The robot also receives one-time reward for reaching the end of the terrain, and a one-time penalty of -3 for rotating more than 90 degrees from its originally orientation in either direction. The episode duration reaches a 500 time steps.
B.5 ObstacleTraverseer-y
In this task, the robot walks across terrain that gets increasingly more bumpy. The task task-specific observation is formed by concatenating vectors {vrobot grobot ,,, hrobot (5)). and the reward R is:
R = =
(6)
which rewards the robot for moving in the positive x-direction. The robot also receives a one-time reward 2 for reaching the end of the terrain, and a one-time penalty of -3 for rotating more than 90 degrees from its originally orientation in either direction. The episode duration reaches 1000 time steps.
B.6 ObstacleTraversse.ae
In this task, the robot walks through very bumpy terrain. The task-specifer observation is formed by concatenating vectors {urobot grobot Ir (5)). , and the reward is:
R == Ap robot
(7)
which rewards the robot for moving in the positive x-direction. The robot also receives one-time reward for reaching the end of the terrain. The episode duration reaches 1000 time steps.
B.7 Hurdler-v0
In this task, the robot walks across terrain with tall obstacles. The task-specific observation is formed by concatenating vectors {urobot grobot Arobot (5)}. and the reward R is:
R
Apr robot
8
which rewards the robot for moving in the positive x-direction. The robot also receives a one-time penalty of -3 for rotating more than 90 degrees from its originally orientation either direction. The episode duration reaches 1000 time steps.
B.8
GapJumper-v0
In this task, the robot traverses a series of spaced-out floating platforms all at the same height. The task-specific observation is formed by concatenating vectors {orobot (robot (robbt(5)), and the reward is:
Ap- robot
which rewards the robot for moving in the positive x-direction. The robot also receives a one-time penalty of -3 for falling off the platforms. The episode duration reaches 1000 time steps.
B.9 PlatformJumppe-.y
In this task, the robot traverses series of floating platforms different heights. The tar- get design space is 5 x 5. The task-specific observation is formed by concatenating vectors robot grobot }5)), and the reward R is:
R Ap robot
(10)
which rewards the robot for moving the positive d-rirectonn, The robot also receives one-time penalty of -3 for rotating more than 90 degrees from its originally orientation in either direction or for falling off the platforms (after which the environment resets). The episode duration reaches 1000 time steps.
15

---

C Implementation Details
C.1 Hyperparameters and Training Procedure
We use PyTorch [66] to implement all the models used in our work. We take the official imple- mentation of transformer from Pytorch which uses TransfrmerEnccderrlyye module, and add learnalle position embedding. All hyperparameters of PreCo are listed Table 2.
Our co-design policy can be trained in an end-to-end RL manner because we unify the design and control processes a finite number of design actions to develop robot morphology, and no reward is assigned to the policy during this period. Subsequently, the resulting robot is controlled by this policy to collect the environmental rewards, which also provides learning signals for the design actions. Once desired number of trajectories collected using distributed trajectory sampling (described in Section 4), the policy is updated using PPO. We also ensure that the baselines and our method use the same number of policy iterations (simulation steps) for optimization.
single MDP. That is, at the start of each RI episode, the policy first takes
Table 2: Hyperparameters of PreCo.
Hyperparameter
Value
GAE
True
GAE À
0.95
Learning rate
2.5 10
Linear learning rate decay
True
Clip parameter
0.1
Value loss coefficient
0.5
Entropy coefficient
0.01
Time steps per rollout
5120
PPO
Optimizer
Adam
Evaluation interval
10
Discount factor
0.99
Clipped value function
True
Observation normalization
True
Observation clipping
10, 10
Reward normalization
True
Reward clipping
10, 10
Policy epochs
8
Neighborhood
Moore
Design steps
1
Number of layers
3
Number attention heads
Transformer
Embedding dimension
128
Feedforward dimension
256
Non linearity function Dropout
ReLU
0.0
C.2 Details of the Baseline Algorithms
For baseline algorithms, we use the official implementation of GA from Evolution Gym [3] and employ a population of 12 agents. It's worth noting that the inner loop of control optimization is also driven by PPO, while the outer loop of morphology optimization is implemented using the evolutionary algorithm. Additionally,
the official implementation of CuCo from [5] and Reptile from [65]. In the remaining section, demonstrate details about these baselines.
16

---

and adhere the original GA GA directly encodes the robot's morphology as vector where each element is tailored to the voxel's material property in order. uses elitism selection and simple mutation strategy to evolve the population of robot designs. The selection keeps the top 2% of the robots from the current population as survivors and discards rest, and the mutation can randomly change each voxel of the robot with a certain probability (mutation rate). our study, the survivor rate starts at 60% and decreases linearly 0%, and the mutation rate is set to 10%.
CuCo CuCo is curriculum- baasedduu---eed co-design method that consists of separate NCA- design control policy. This curriculum-base method expands the design space from a small size to the target size using reinforcement learning with predefined curriculum. In our study, we set the curriculum of CuCo be 3 x 3 - hyperparameter settings of CuCo presented in [5].
policy and transformer-base
MeCo MeCo utilizes the same network architecture as PreCo but is trained with the Reptile [65], a popular meta-learning method. Reptile is designed to identify model parameters that serve as an optimal starting point for adaptation across various tasks. When encountering a novel task, the model expected to need fewer updates or episodes to achieve proficient performance. In contrast another meta-learingg method, MAML[67], which necessitates second-order gradients (gradients of gradients) during its meta-update step, Reptile simply averages the updates. This characteristic makes Reptile more computationally efficient and easier implement. In our study, set the meta-learning rate to 0.25, and the update iteration for each training task is configured to be 20.
C.3 Computational Cost
We use distributed trajectory sampling with multiple CPU threads to collect training data (described in Section 4). For pre- vre-training experiments in the paper, takes around 2 days to train our model on a standard server with 40 CPU cores and an NVIDIA RTX 3090 GPU.
D Visualization Results
In this section, we provide some visualization results of brain-body pre-training and train-body fine-tuning, as shown in Figure 7 and Figure respectively.
More uneven terrain
Zero-Shot Generalization
Descend stairs
Cross terrain
Climb stairs
Training Tasks
Figure 7: Visualization PreCo's 's zero-shot behavior. The figure shows screenshots at consecutive time intervals of the designed robot's behavior. Compared with training tasks, We find that PreCo shows favorable generalization properties and intriguing behavior when facing new environments (beginning from 00 28 in this video).
E
Ablation of the Shared Policy Representation
The performance results of PreCo and PreCo-Sep in Figure 3 and Table suggest that a shared policy representation facilitates zero-shot generalization and few-shot adaptation, surpassing methods that employ separated representations. Furthermore, we present comparison of the performance of
17

---

Brain-Body
Pre- Pre-Training
Brain-body
Fine-Tuning
Figure 8: Visualization of brain-body fine-tuning. The pre-trained co-design policy shows the ability to swiftly adjust both the morphology and control strategy to adapt to new co-design tasks (beginning from 00 42 this video).
Table 3: Performance results across 10 co-design tasks, All methods trained from scratch.
Climber-vl
2.5
Environment
PreCo-Seratch
PreCo-Sep-Scratch
Walker-vO
10.47 0.01
10.46 = 0.01
Climher-v0
2.23 0.87
0.43 0.02
1.5
Hurdler-vO
2.76 1.39
2.07 1.33
UpStepper-v0
7.23 + 1.46
4.06 + 0.28
DownStepper-v))
8.98 + 0.01
7.46 0.71
GapJumper-vO
3.51
Bridge Walker-O
4.81 + 0.21
5.46 + 1.01
PlaformJumper--00
6.12 + 0.82
3.97 + 0.33
0.00
0.25
0.50
0.75
1.00
1.25
1.30
1.75
2.00
Iooration (x100)
Dostacle/raavereerree
6.03 +: 2.34
5.08 0.19
DostacleTraverser-.
3.38 0.69
2.78 1.06
Figure 9: Learning curves in Climber-vO.
H
PreCo-Scratch
Lc
Iteration: 25
Iteration:
Iteration:
Iteration: 250
Iteration: Iteration: 1000
L
PreCo-Sepe Sap--e---ccchh Scratch
Figure 10: Morphological results of the two methods obtained during their learning processes.
these two methods when trained from scratch across 10 co-design tasks, the results shown in Table 3. Figure 9 and Figure 10 illustrate their learning processes in complex task, Climber-vO.
F Discussion of the Sim-to-Real Issue
In our paper, tested method using simulator with relatively fundamental modules proof of concept to show its effectiveness. In this section, We discuss the sim-to-real issue of our work.
From the perspective of "Sim", the EvolutionGym platform [3] used our work employs several simplifications to reach a trade-off between the simulation quality and velocity. Thus, for more realizable sim-to-real transfer, an improved version of its physies engine needed to model soft- body physics in 3D space. We believe that using the Finite Element Method (FEM) numerical simulation would be of the feasible ways to narrow this sim-to-real gap because the voxel-based design provides naturally organized mesh, and its resolution and element type could be relatively
18

---

Fine-lunig on Obstacle 1
Training Co-Design Tasks
Oedecalheeeedd nittrnkurpen
3
Average Performance
Performanas
2
0.01
0.05
0.10
0.15
0,20
0.26
0.30
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Iterations (x10²)
Iterations (x10P)
Figure 11: Additional experiments. Left: visualization of the tin-ttnnig process. curve of brain-body pre-training on more diverse co-design tasks.
Right: learning
easily determined. Moreover, FEM has shown promise for real-world soft robotic models when combined with highly stable implicit Euler integration [42].
In our study, focus the model-free co-design problem where system modeling is not required, and we reinforcement learning to approximate the gradient of design and control. This endeavor requires large amount of training data. However, when considering the real-world problem, it possible to transfer to the model-based setting. By leveraging the differentiable properties of certain FEM models [40], the universal parameterized co-design policy can be updated using Back- Propagation Through Time (BPTT), resulting in more efficient brain-body pre-training. When we move to the 3D modular robot design space, we would like to adapt the parameterization method depicted in Figure 5 to its 3D version. By using "3D convolutional kernel", can easily create input design observation sequences for the mansformer-basee policy. Although transformer has suf- ficient capacity to handle long sequences, more expansive design space (e.g., thousands of voxels) would necessitate more complex transformer model. This, in turn, could demand significantly higher computational power.
From the perspective of "Real", one the foremost considerations is the material selection, which should be predicated upon the required flexibility, durability, and functionality [68, 69]. With this criterion, the Diels-Alder (DA) polymer [70] or silicone voxels [38], in conjunction with multi- material cubic blocks produced through 3D printing, may serve as ideal components for constructing the body of a MSR. To establish the local observation space, each voxel could be equipped with an array of sensors, such as touch, pressure and velocity sensors. Alternatively, soft sensors, crafted from conductive elastomers that alter resistance upon deformation, could offer valuable feedback to the control system. Furthermore, Seano-HASEL actuators [71] or pneumatic actuators [72] might be suitable for volumetric actuation (probably limited expansion for efficient simulation), and closed- loop control could be achieved by utilizing Neural Networks (NNs). In the real-world setting, factors like material imperfections, air resistance, friction and many others come into play, we also need to iteratively refine the design and control algorithms based on real-world feedback.
We acknowledge that each point discussed above presents its challenges but is well worth in-depth investigation, and we aspire for our work to serve as a catalyst for future research into the co-design of modular soft robots.
G Additoona Experiments
G.1 How Does Fine-Tuning on the Target Task Affect Performance on the Training Tasks?
It is worth noting that when pre-trained co-design policy undergoes fine-tuning for target task, it can experience what's 's known as "catastrophic forgetting". This means that it might for-
19

---

Walker-vO 6.13
Pusher-vO
UpStepper-v0 R: 3.65
Carrler-v0 R: 2.23
PlatformJumper-v00 R: 4.03
ObstaeteTrawerser-vo R: 3.86
Figure 12: Morphological results of brain-body pre-training.
get certain information patterns it learned during the pre-training phase, potentially leading decrease in performance on the original training tasks.
We track the performance changes on pre-training tasks when the co-design policy is fine-tuned on ObstacleTrraverser-v The left side of Figure 11 illustrates a consistent decrease in performance for Walker-vO and PlatformJumper-vO due the significant disparity between the target task and the original tasks. In contrast, If the target task is similar the pre-training tasks, the co-design policy might retain of its initial knowledge. For instance, performance tasks like UpStepper-v0 and TrastrcleT remains less affected throughout the fine-tuning.
G.2 Pre-Trainee Training on More Diverse Co-design Tasks
In our paper, we select locomotion tasks for pre-training and 5 tasks for testing. As we focus on co- designing modular soft robots to perform multiple tasks, the wealth brain-body links embedded within the enormous combined search space offers sufficient diversity for effective policy learning.
To further explore the potential of PreCo, also conduct additional experiment that encom- passes more diverse co-design tasks for pre-training. In this experiment, except for the original pre-training tasks, we add Pusher-vO (the robot is encouraged to push box initialized in front of as far as possible) and Carrier-vO (the robot is encouraged to carry a box initialized above it as far as possible) to the co-design policy's learning procedure. Moreover, we add Thrower-vO (the robot is encouraged to throw a box initialized above as far as possible) to the test tasks. The right side of Figure 11 demonstrates the learning curves, and the result is averaged over different runs. The results of Figure 12, Figure 13 and Figure 14 show that PreCo still performs well on zero-shot generalization and fhw-shot adaptation.
20

---

BridgeWalker-v0 1.36
Hurdler-vO R: 1.85
DownStepper-v0 R: 3.54
Thrower-vO R: 1.52
GapJumper-v0 1.89
ObstacleTraverser--11 R: 3.63
Figure 13: Morphological results of zero-shot generalization.
BridgeWalker-v0 R: 2.58
Hurdler-v0 R: 3.55
DownStepper-v0 R: 5.27
Thrower-v0 3.13
GapJumper-v0 5.52
ObstacleTraverserrvv 5.13
Figure 14: Morphological results of brain-body fine-tuning.
21


Reviews:
[{"review_id": "nwdUstOrR7m", "reviewer": "Reviewer_rtsy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes a framework (PreCo) for the joint optimization (co-design) of the voxel-based design and corresponding control policy for soft robots. Integral to the approach is its use of a pre-training phase to jointly optimize design and control on a set of training tasks before fine-tuning on a different target task. The policy is shared across designs and both the design and control are trained via model-free RL. Experiments demonstrate that PreCo outperforms curriculum-learning and meta-learning baselines in terms of zero-shot generalization to the target task as well as few-shot (with the exception of one domain).", "summary_of_recommendation": "The problem of co-optimization is interesting and relevant to robot learning and robotics, more generally, The core contribution is the use of a pre-training phase in which the framework performs joint optimization on a set of tasks as a means of facilitating zero- and few-shot transfer to a target task. However, it is not obvious whether there is anything fundamental to the architecture that enables this transfer vs. perhaps being the first to try a transformer-based universal control policy (which is not new) with a series of tasks. Without key insights into how these tasks should be chosen, the significance of the contributions is unclear.\n\nUPDATE AFTER AUTHOR RESOPNSE/DISCUSSION\n\nThe authors' response addresses the primary concerns that I had with the paper and as a result, I have increased my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1) Can the authors clarify the nature of the time index for $s_t^d$ and $s_t^c$. In the case of the design state $s_t^d$, my understanding is that $t$ indexes the point in training, whereas in the case of the controller state $s_t^c$ it is episode time. Is that correct?\n\n2) Line 180: What is meant by \"At every time step, $K$ state information of each co-design task is uniformly sampled\"? What is $K$? What is the \"state information of each co-design task\"?\n\n3) How does fine-tuning on the target task affect performance on the training tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "STRENGTHS\n+ The paper is well written and easy to read.\n+ The topic of the paper is highly relevant---the problem of joint optimization of robot design and control has garnered renewed interest in the robot learning community within the last few years.\n\n\nWEAKNESSES\n- The significance of the paper's contributions is unclear. It is not the first to propose using a universal controller/policy as part of joint optimization (e.g., Reference [33] in the paper and References [1-4] below) as a means of improving the efficiency and (in some cases) performance of the resulting design. Also relevant, many of these papers similarly motivate the idea of learning a single policy that can control different designs as a form of multi-task RL (e.g., [42] in the paper and [5, 6] below). Instead, the primary novelty lies in optimizing over a set of different tasks as a pre-training phase in an effort to facilitate \"transfer\" to the target task. The benefits are fairly clear in the context of zero-shot and few-shot transfer, but the results don't demonstrate that such an approach is better than training from scratch on the target task. However, it's not obvious whether there is anything fundamental to the algorithm that enables multi-task pre-training as opposed to training a transformer-based framework that has a universal policy on a set of different tasks. Indeed, there is no discussion of how one might go about deciding on these pre-training tasks when there is a target task in mind. It seems that they were chosen randomly.\n- Related, Table 2 only includes one domain from Table 1. It is important to compare the performance of the resulting designs as a result of fine-tuning to those that are trained from scratch with the same architecture as well as a baseline (e.g., see the results in the \"Evolution Gym\" paper).\n- The paper is missing a few references to relevant work in soft robot co-design [1, 2, 5, 7]. Reference [5] seems particularly relevant given the focus on soft robots and the use of a universal controller, though it seems that they only train on a single task.\n- The paper doesn't provide sufficient details about the design space\n- The paper provides no discussion of how one might extend these results to physical hardware, which is likely difficult given the significant sim-to-real gap, particularly for voxel-based designs and differentiable simulators [3].\n- Certain claims in the paper are not supported. This includes the statement that the universal policy \"significantly facilitates exploration of the design space\". The results show that it improves performance including in a zero-shot setting, but there is no results that show improved exploration.\n- Lines 148-150: The statement regarding deficiencies in optimization as performed by existing methods that alternate between updating the design and the control are not substantiated. \n- While the appendix provides the hyper-parameter settings, it would be helpful if there was more discussion of the training procedure in the main paper, including the way in which tasks are chosen during pre-training (i.e., how often and from what distribution (uniform?)), how the method alternates between updating the design distribution and rolling out the current policy, etc.\n- Personally, I find the frequent parallels to \"brain\" and \"body\", \"genome\", and \"pleiotrophy\" to be a distraction since the relationship between biological evolution and what the paper describes is purely superficial.\n- Minor: Section 1 (first paragraph): After stating the importance of co-design, the motivating examples that are provided demonstrate the importance of optimizing the design, but not control.\n- Minor: Lines 240-241: Any claims about why the model learned a particular design are purely speculative\n\n\nGRAMMAR:\n* Line 187: \"Does our method, PreCo, ~~can~~ effectively perform\"\n\n\nREFERENCES\n\n[1] R. Deimel, P. Irmisch, V. Wall, and O. Brock. Automated co-design of soft hand morphology and control strategy for grasping. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.\n\n[2] S. Kriegman, S. Walker, D. Shah, M. Levin, R. Kramer- Bottiglio, and J. Bongard. Automated shapeshifting for function recovery. In damaged robots. in Proceedings of Robotics: Science and Systems (RSS), 2019.\n\n[3] S. Kriegman, A. M. Nasab, D. Shah, H. Steele, G. Branin, M. Levin, J. Bongard, and R. Kramer-Bottiglio. Scalable sim-to-real transfer of soft robot designs. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft), 2020.\n\n[4] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. In Advances in Neural Information Processing Systems (NeurIPS), 2019.\n\n[5] C. Schaff, A. Sedal, and M. R. Walter. Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer. In Proceedings of Robotics: Science and Systems (RSS) 2022\n\n[6] C. Schaff, and M. R. Walter. N-LIMB: Neural Limb Optimization for Efficient Morphological Design. arXiv:2207.11773 2022\n\n[7] A. Spielberg, A. Amini, L. Chin, W. Matusik, and D. Rus. Co-learning of task and sensor placement for soft robotics. IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1208–1215, 2021", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "cNIOeIRzCB_", "reviewer": "Reviewer_dKrr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the problem of automatically synthesizing designs and controllers for soft robots that are capable of generalizing (either zero-shot or with some fine tuning) to new tasks. Specifically, they study the co-design problem for 2D soft robots in environments of varying difficulties. The main innovation in the paper appears to be the use of a single shared backbone network (a transformer) for both the design and control policy networks (rather than separate representations as in prior works). The authors demonstrate better zero-shot and fine-tuned transfer to previously unseen tasks on a range of problems.", "summary_of_recommendation": "This paper presents a simple idea (a shared design/control policy backbone plus pre-training) that yields good empirical results in simulation. I think the core idea is worth sharing, but I recommend that the authors invest a considerable amount of time in revising and making sure that their explanation is clear. Please take care to disentangle causes and effects in your explanations; there are a number of distinct challenges in co-design, and you propose a number of distinct solutions, so it is important that you help the reader understand which solutions help address which challenges (both through explanation and empirical ablation).\n\n**Update post-rebuttal** I have no further questions and am content to leave my score at weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please address all of the questions and issues raised above. In addition:\n\n1. How are the design and control actions represented?\n2. Why do you train on a CPU? Wouldn't GPU training be more effective?\n3. Page 5, line 177-8: \"We estimate the overall value of the morphology by taking the average of the values assigned to each individual voxel.\" What does this mean? What is the \"overall value of the morphology\" and how is it used?\n4. You claim that the single backbone (\"pleiotropic representation\") yields better preformance. Can you comment on why MeCo (which also uses this shared representation) fails to learn or generalize on most tasks? Why does your pretraining work while the Reptile pre-training used by MeCo fail on most tasks?\n5. What is CPPN-NEAT, and how does it compare to your method?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n\n- The authors clearly motivate their use of a single backbone network for design and control, grounding it in a biological analogy.\n- The proposed method yields impressive empirical results.\n\n### Weaknesses\n\n- The authors identify a number of challenging aspects of the co-design problem, but it is not clear how their approach solves all of these challenges. For example:\n\t- Challenge: since relatively few design actions occur early in each rollout (compared to far more control actions), there is relatively less reward information on which to train the design policy. The authors claim that a shared backbone helps with this, and their ablation `PreCo-Sep` supports this.\n\t- Challenge: different design morphologies result in action spaces of different dimension. The authors claim that a transformer architecture helps with this, but they don't explain how. Please add more detail here.\n- You say that the genetic algorithm is used to provide an estimate of the upper bound performance for each task, but the GA doesn't seem to learn well on any task. Why is that?\n- The limitations section is not very comprehensive. In addition to the two limitations discussed (poor performance on the bridge environment and the risk of destructive mutations/forgetting), I can think of several others:\n\t- How would you generalize to robots in three dimensions? Would this problem suffer from a curse of dimensionality (e.g. 3D convolutions tend to be substantially harder than 2D convolutions; would a transformer-based architecture share this problem)?\n\t- How would you fabricate a robot with one of these designs? How could you enforce manufacturability constraints? Are there existing robot architectures or manufacturing methods that are compatible with the types of designs that your method produces?\n\n### Minor comments\n\n- \"Transformer\" is commonly written lower-case as \"transformer\" unless at the start of a sentence.\n- When discussing your results, please don't use terms like \"remarkable\" or \"astonishing\". It's up to the reader whether they are astonished or not.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "GMrGg3vzDZ", "reviewer": "Reviewer_Bcvj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel approach dubbed PreCo, which incorporates brain-body pre-training within the co-design framework for modular soft robots, aimed at bolstering their generalization and adaptability capabilities. The methodology hinges on the encoding of co-design principles into models achieved through pre-training a versatile co-design policy on a wide-ranging set of tasks. This pre-trained co-designer is subsequently employed to generate preliminary designs and control policies, which are then fine-tuned specific to a co-design task. The results reveal not only zero-shot generalization to unseen co-design tasks but also expedited adaptation with substantially fewer iterations. The work provides noteworthy contributions to co-design in robotics, more specifically within the field of modular soft robots.\n\n", "summary_of_recommendation": "The manuscript provides substantial contributions to the domain of co-design in robotics, particularly regarding modular soft robots, by presenting an innovative and pragmatic methodology, coupled with astute empirical findings and a comprehensive review of relevant literature. However, the study does fall short in certain aspects such as limited implementation details and hardware demo and evaluation studies, which necessitate further clarification. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Could the authors offer more insight into the choice of co-design tasks and the rationale behind their selection?\nIs it possible to compare the proposed methodology with recognized baselines in the field to offer a more comprehensive understanding of its level of improvement?\nWould the authors be able to provide further details for the replication of the study, in addition to more extensive evaluation and ablation studies of the proposed method?\nIs it possible to conduct physical experiments?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The introduction of the PreCo concept is novel, with practical implications for bolstering generalization within the co-design procedure for modular soft robots. The paper offers incisive empirical evidence concerning the efficacy of brain-body pre-training, zero-shot generalization, and fine-tuning. The structure of the pleiotropic co-design policy is robust and eliminates the requirement for a robot population, leading to enhanced flexibility and sample efficiency. \n\nAlso, a comprehensive and well-organized review of pertinent literature is presented. The experiments conducted on the modular soft robot system are thorough and meticulous, providing a robust foundation for the proposed methodology.\n\nHowever, the experiments do not encompass a diverse range of tasks, and further investigation into the choice of co-design tasks could serve as an intriguing area for future research. The manuscript could benefit from a comparison or discussion with established baselines in the field, thereby providing a more informed understanding of the degree of improvement proffered by the proposed methodology. Details for replication of the study are limited, and a more exhaustive evaluation and ablation studies of the proposed method are required.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "62eCtqAF0vr", "reviewer": "Reviewer_WFDJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The co-design methods in the literature mainly consider discovering an optimal robot morphology and controller for a specific task. Yet, biological morphologies can perform multiple tasks. Drawing inspiration from pleiotropy in biology, this paper proposes a learning framework for designing a modular soft robot capable of performing a wide variety of tasks. The authors achieve fast adaptation to new co-design tasks by leveraging past co-design knowledge and demonstrate the effectiveness of brain-body pre-training. The paper presents simulation results to evaluate the performance of the proposed approach. ", "summary_of_recommendation": "Motivated by the idea of facilitating effective body-environment interaction, the paper focus on co-designing both the physical bodies and the control systems that operate on the robot body. Their implementation of \"pleiotropic co-design policy\" is novel and can generate effective results against similar algorithms.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Provide a reference for the statement in lines 61-62.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well organized and the writing is clear. The presented methods are sound. The context of this paper is interesting because traditionally co-design methods focus on optimizing for \"one robot one task\", unlike biological system, while this paper presents a new learning approach aiming to address multiple co-design tasks simultaneously. The authors used modular soft robots to build their case allowing them to construct robot bodies by combining various types of deformable cubes and adjusting their volume to generate control signals. Aiming to design a single modular soft robot that achieves a wide variety of tasks, the paper first train the robot by adapting multi-task reinforcement learning. Then, the robot is further trained on unseen tasks. The authors implemented a new co-design policy, namely pleiotropic co-design policy, allowing them to integrate design and control into a unified decision process. \n\nThe authors illustrate that their proposed method could be more effective in learning and executing multiple tasks against various baseline models in simulation environment. However, the morphologies that their approach generated for locomotion purpose on various terrain profiles seem unconventional and not relevant to legged or wheeled locomotion morphologies. It would be interesting to see that their method converge to an efficient morphology for locomotion. Maybe varying the reward function towards legged or wheeled appendages might serve such a purpose.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper concerns a novel idea, and provides some working solutions to a long-held issue with robotic design.

The authors have been responsive during the review phase.

It is important to put this work in context of the large body of work on co-optimization that dates back several decades and that has received renewed attention within the last several years.

Furthermore, the biological framing for this work is unwarranted as any biological comparisons are superficial at best, and misleading at worst.
Decision: Accept (Oral)