Curriculum-based Co-design of Morphology and Control of Voxel-based Soft Robots
Authors: Yuxing Wang
Shuang Wu
Haobo Fu
QIANG FU
Tiantian Zhang
Yongzhe Chang
Xueqian Wang
Venue: ICLR 2023 poster (2023)
Keywords: Artificial Life
Brain-body Co-design
Robotics
Modular Soft Robots

Abstract: Co-design of morphology and control of a Voxel-based Soft Robot (VSR) is challenging due to the notorious bi-level optimization. In this paper, we present a Curriculum-based Co-design (CuCo) method for learning to design and control VSRs through an easy-to-difficult process. Specifically, we expand the design space from a small size to the target size gradually through a predefined curriculum. At each learning stage of the curriculum, we use reinforcement learning to simultaneously train the design policy and the control policy, which is enabled by incorporating the design process into the environment and using differentiable policy representations. The converged morphology and the learned policies from last stage are inherited and then serve as the starting point for the next stage. In empirical studies, we show that CuCo is more efficient in creating larger robots with better performance by reusing the practical design and control patterns learned within each stage, in comparison to prior approaches that learn from scratch in the space of target size.

Published as a conference paper at ICLR 2023
CURRICULUM-BASED CO-DESIGN OF MORPHOLOGY AND CONTROL OF VOXEL-BASED SOFT ROBOTS
Yuxing Wang Shuang Wu?, Haobo Fu Qiang Fu", Tiantian Zhang', Yongzhe Chang',
Xueqian Wang
'Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China Tencent AI Lab, Shenzhen, China
ABSTRACT
Co-design of morphology and control of Voxel-based Soft Robot (VSR) is chal- lenging due to the notorious bi-level optimization. this paper, we present a Curriculum-be based Co-design (CuCo) method for learning to design and control VSRs through an easy-to-difficult process. Specifically, we expand the design space from a small size to the target size gradually through predefined curricu- lum. At each learning stage of the curriculum, we use reinforcement learning to simultaneously train the design policy and the control policy, which is enabled by incorporating the design process into the environment and using differentiable pol- icy representations. The converged morphology and the learned policies from last stage are inherited and then serve as the starting point for the next stage. em- pirical studies, we show that CuCo is more efficient in creating larger robots with better performance by reusing the practical design and control patterns learned within each stage, in comparison to prior approaches that learn from scratch in the space of target size.
INTRODUCTION
The philosophy of embodied cognition (Pfeifer & Bongard, 2006; Pfeifer et al., 2014) inspires the domain of roboties that robot's ability interact with the environment depends both its brain (control policy) and body (morphology), which are inherently coupled (Spielberg et al., 2019; Gupta al., 2021). However, finding an optimal robot morphology and its controller for solving a given task is often unfeasible. The major challenge for this endeavor is the enormous combined design and policy space. Firstly, the freedom to pick the number of multi-mann materialt modules and the ways they are connected makes it notoriously difficult to explore the design space (Medvet et al., 2022). For instance, in robot simulator (Liu et al., 2020), there are over x 10 possible morphologies for robot composed of only 12 modules. Secondly, the evaluation of morphology requires separate training procedure for its unique controller.
In this work, consider the co-optimization of design and control of Voxel-based Soft Robots (VSRs) (Bhatia et al., 2021), 1), a form of modular soft robots composed of clastic, multi-material cubic blocks. Unlike fragile fully-integrated robots, they be easily disassembled and reassembled to adapt to a wide range of environments (Shah et al., 2020; Pigozzi et al., 2022). For efficiently exploring the modular robot design space, prior approaches commonly rely on artificial evolution (Sims, 1994; Cheney et al., 2013; Medvet et al., 2021), which maintains a population of design prototypes and adopts bi-level optimization loop, where the outer loop of morphology optimization is based on the fitness of individual controllers from the inner loop. These methods, however, tend to learn from scratch in the target design space where there is a significant combinatorial explosion. Thus, they spend a large amount of time on policy optimization and evaluation. Additionally, their separate training procedures significantly hinder the experience of design and control to be shared across different robots.
"This work was done during an internship at Tencent Al Lab, Shenzhen, China. Corresponnence Haobo Fu (haohofu@@ten (aaofff@ttnneet.ommm and Xueqian Wang www.g/ssssssinig.........m

---

Published as a conference paper at ICLR 2023
Enviromment
(5)
(0)
Designsags
Controll
Contrrll palky
Contral palicy
Conckeet Trrdhet-boeeti itt. mnertrt
palky
lociaa aree
Hosial and
Design f
Control staps
Empt vocal
Delggn Rip policy
Delggn pollcy
Dedggn
Rigld vorel
palicy
Softvosel
Cursoulan
Horiovetal
Stage
Vertoll actuator
Figure I: Schematic of approach CuCo. (A) We investigate the brain-bote body co-design problem of
VSRs by establishing an easy-to-difficult optimization process between a small design space (c.g.,
3 x 3) the target space (e.g.. x 7). (B) In 3 3 design space, the design policy performs finite number steps to develop a VSR which initiated by a single voxel. (C) VSR is composed of 5 kinds of voxels.
In view of these challenges, we propose a Curriculum-based Co-design (CuCo) method for learning design and control VSRs from easy difficult (Figure 1). CuCo draws inspiration from Cur- riculum Learning (CL.) that starting with easier tasks can help the agent learn better when presented with more difficult tasks later on (Bengio et al., 2009). The key to our approach is expanding the de- sign space from small size to the target size gradually through a predefined curriculum. Precisely, each stage of the curriculum, we learn practical design and control patterns via Reinforcement Learning (RL) (Schulman et al., 2017), which is enabled by incorporating the design process into environment and using differentiable policy representations. The converged morphology and the learned policies from last stage are inherited and then serve as the starting point for the next stage. Due to the exploitation of the previous knowledge of design and control, CuCo can quickly produce robust morphologies with better performance in larger dimensional settings.
To handle incompatible state-action spaces and make the control patterns transferable across vari- ous morphologies, we model the local observations of all voxels sequence and adopt the self- attention mechanism (Vaswani et al., 2017) in our control policy to capture the internal dependencies between voxels, which caters to the need for dynamically accommodating changes in morphologies. We also add material information to each voxel's observation, thus the control policy is conditioned the robot's morphology, which eliminates the need for a population of prototypes and enables experience to be shared across different robots, bringing better sample efficiency and flcxibility, For the design policy, we propose utilize a Neural Cellular Automata (NCA) (Mordvintsev et al., 2020), which takes multiple actions to grow a robot from an initial seed (morphology). NCA en- codes complex patterns in a neural network and generates different developmental outcomes while using a smaller set of trainable parameters.
In this paper, we make the following contributions: (1) We propose CuCo, a curriculum-basee co- design approach for learning to design and control VSRs. (2) We demonstrate that the practical de- sign and control patterns learned in small design spaces can be exploited to produce high-perg morphologies in larger dimensional settings. (3) We showcase that NCA-based design policy and self-at sulf-aatention-baae control policy can share design and control patterns across different sizes of morphologies, bringing better sample efficiency and flexibility.
(4) Comprehensive experimen- tal studies show that CuCo is more efficient in creating larger robots with better performance, in comparison prior approaches that learn from scratch in the target design space.
2 RELATED WORK
Brain-body Co-design. Co-designing a robot to solve a given task has been challenging for robot designers for decades (Sims, 1994; Wang al., 2019b; Nygaard et al., 2020; Medvet et al., 2021; Hejna et al., 2021; Ma et al., 2021). Related works can be roughly divided into two categories, gradient-free and gradient-based. Most gradient-free methods rely artificial evolution explore
2

---

Published as a conference paper at ICLR 2023
the vast design space. Cheney et al. (2013) evolve VSRs with 3D voxels over materials and actu- ators. Bhatia et al. (2021) create a large-scale benchmark and several population-bart co-design algorithms. Walker & Hauser (2021) evolve sculpting adaptation system that allows VSRs to change their bodies to adapt to the environment. Talamini et al. (2021) propose a task-agnostic approach for automatically designing morphologies. While the population-based approaches not easily trapped in local optima, they are sample inefficient and computationally expensive. Be- sides, most of these works rely on open-loop controllers, which simplify the control as a periodic sequence of actuation that depends only on the time (e.g., sinusoidal control signals), preventing robots from learning non-periodic behaviour. In contrast, gradient- based methods, especially rein- forcement learning methods, have been conducted for optimizing rigid-bodied robots. Ha (2019) uses a population-baser policy gradient method to improve robot design. Pathak et al. (2019) con- struct a self-organized modular robotic system that can generalize to unseen morphologies and tasks. Luck et al. (2019) train a surrogate for efficiently evaluating morphologies. Schaff et al. (2019) main- tain a distribution over morphological parameters. Chen et al. (2020) learn policy. Yuan et al. (2022) optimize a transform-and-contro policy based on Graph Neural Network (GNN) in a fixed design space. Our work falls into the second category, instead, focus scalable modular robot design space assigned by a curriculum. Within each learning stage, we unify the processes of design and control a single decision-makin However, both the design policy and the control policy of our work are exploited across stages as they are not restricted by robots' morphological topologies.
process like (Yuan et al., 2022).
Curriculum Learning. Curriculum learning follows the spirit that training the agent with series of tasks organized according to different complexity can speed up the convergence of the learning process (Bengio et al., 2009). It has achieved tremendous success in both RL and robotic commu- nities (Li et al., 2020; Iscen al., 2021; Jiang al., 2021b). In RL, one of the keys to curriculum learning is how to generate and assign tasks. Dennis et al. (2020); Jiang et al. (2021a); Parker- et al. (2022) focus on Unsupervised Environment Design (UED), which automatically constructs a distribution over environment variations, while Matisen et al. (2020); Portelas et al. (2020) inves- tigate teacher algorithms that monitor the learning progress and generate curricula to guide student algorithms learn in difficult environments. In robotics, Wang al. (2019a; 2020) generate an endless progression of increasingly challenging environments to evolve robust robots. Fang et al. (2021) propose method to learn skills via automated generation of diverse tasks. In this work, adopt a fixed curriculum by increasing the size of the robot design space according to the learning progress, as the number of possible robot morphologies increases exponentially with the expansion of each dimension of the design space.
Learning a general Controller. Another related line is learning generalizable control policies. Many such approaches utilize GNNs (Sanchez-Gonzal et al., 2018; Wang et al., 2019b; Huang et al., 2020; Yuan et al., 2022) that condition the policy input on graph representation related the robot morphology. However, for modular robot systems, GNNs have the limitation of aggregating multi-hop information (Pigozzi et al., 2022), which makes modules not completely interchangeable, and the system could be influenced by information loss, greatly limiting their flexibility. Instead, Transformer (Vaswani et al., 2017) is not restricted by the input graph topology, as it can be charac- terized fully connected GNN with attentional aggregation (Kurin et al., 2021). It is shown that generalization can be further enhanced by modularizatio, due to the success of modeling dynamic structures via self-attention (Gupta et al., 2022; Trabucco et al., 2022).
3 BACKGROUND
Voxel-based Soft Robots. We investigate VSRs composed of multi-material cubic blocks (e.g., soft voxels, rigid voxels and actuator voxels) organized in a 2-D grid (Figure (C)). The connectivity describes the connection pairs of adjacent voxels. During the simulation, voxels (except empty voxels) only sense locally, and based on the input sensory information, a controller outputs control signals to vary the volume of actuator voxels. Similar biological tissues, they can be easily expanded and contracted to complete a wide variety of tasks (Bhatia et al., 2021).
Reinforcement Learning. In contrast to prior co-design works which seldom involve the learning of control, we use reinforcement learning to simultaneously train the design policy and the control policy. In the domain of RI., the problem is usually modeled Markov Decision Process (MDP),
3

---

Published as a conference paper at ICLR 2023
Sliding window
Padding
st12
Flatten
Sample
Assign
MI.P
disiilation
Cropping
Figure 2: Visualization of single design step. multilayer perceptron is incorporated in NCA, which determines cell's state by taking its current state and its neighboring cells' states as the input and mapping it to categorical action distribution. As a result, each cell of the body is updated to the state with the sampled action.
which can be specified by 5-tuple (S,A, 7.r, With the state space S and the action space A, P S x S x A + [0, 1] is the transition function of the environment, r(s, a) : S x A + R is the reward function, and the discount factor > (0, I] specifies the degree to which rewards are discounted over time. The goal of the RL agent is to find a policy to maximize the total expected reward E,, Ei= 7'1(st, at)]. here the policy is represented by a deep neural network parameterized as To (o))) which maps from states to distributions over actions. In this work, we employ Proximal Policy Optimization (PPO) (Schulman et al., 2017), a popular RI. algorithm base on Policy Gradient (PG). PPO uses Kullback-Leibler (KI.) divergence to penalize the change between the old and the current policy to avoid instabilities from drastic changes in the policy's 's behaviour.
Neural Cellular Automata. Since we incorporate the design process into the environment, our ob- jective is to find a design policy that can take arbitrary VSRs' morphologies and output a series of actions modify them. Typically, VSRs can be characterized multi-cellui develop from a small set of initial cells (Horibe et al., 2021). We represent our design policy as a Neural Cellular Automata (NCA) (Mordvintsev et al., 2020; Sudhakaran et al., 2022; Palm et al., 2022), which begins with some initial seeds and updates their states according to local rules param- eterized by multilayer perceptron, thus, NCA is agnostic the robot's morphological topology, making it possible for scaling the learned rules to different design spaces.
systems which
Self-attention. Self-attention was proposed due the challenges in handling long sequences (Vaswani et al., 2017). The goal is to dynamically track the relationships between components, which can be viewed an adaptive weights mechanism. Given the imput vectors X, W W and WV are three trainable weight matrices. Q, K and V are the query, key, and value vectors where Q XWQ, K XWK and V XWV The importance scores are firstly computed by the dot product between Q and K and then scaled by non-linear function. Self-attention is typically inte- grated into the Transformer block, which also contains a feed-forward neural network, and each of these two submodules has residual connection around it and is succeeded by normalization layer. Our work here specifically uses a Transformer encoder, enabling our control policy to dynamically accommodate changes in morphologies.
CuCo: LEARNING TO DESIGN AND CONTROL FROM EASY TO DIFFICULT
Given a modular design space (e.g.. 7x7), exploring all possible combinations is unfeasible. The key to our approach to expand this design space from small size the target size gradually through a predefined curriculum, which divides the whole optimization process into several stages (Figure (A)). We maintain a design policy Wop and control policy The from the beginning to the end, thus the overall policy is represented as a binary value indicating whether the robot is being created or controlled, of and sf represent the state information received at each time step of design and control, respectively. Within each stage, we simultaneously optimize the design and control through reinforcement learning, which is enabled by unifying the two processes into single MDP. Specifically, at the beginning of each episode, the design policy Tep (af) V.) performs No steps to develop a robot from an initial seed, and rewards are assigned to these steps (Figure (B)). The resulting robot is then presented
{TO, (at st Vt), Toc (aflst W.)} where V, E {0, 1}
4

---

Published as a conference paper at ICLR 2023
Tranffomer
Local observation
Nocks
Autoc distrlbution
Norm
Hiner Liner Prjection
Pozittonn Prattoonerbbiiingg errbordding
Feod
Forward
Decoda
Mult Head
A Uietion
Concotenation
Task-related observation
Replication Concatenation
Figure 3: Visualization of the network architecture of the control policy. We model the local obser- vations of voxels as a sequence and utilize Transformer encoder to process the current state.
the simulator and controlled by The (af]sf, Vz). Thus, the design policy will informed by the future environmental rewards collected by the control policy under the designed morphology. After collecting the desired number of trajectories, the two policies are jointly updated using PPO (Schulman et al., 2017). From one stage to the next, the design space is expanded according to the curriculum and the converged morphology from last stage will serve as the new initial seed. The learned design policy and the control policy inherited and then fine-tuned to produce larger bodies. In the remaining section, we describe details about each component of CuCo.
4.1 DESIGN POLICY
Figure demonstrates single design step when developing a VSR. The design space is surrounded by empty voxels, and each voxel represented as discrete value that corresponds its mate- rial property (e.g., empty voxel=0, soft voxel=1, rigid voxel=2, horizontal actuator=3 and vertical actuator=4). The state vector sd for each voxel composed of its type and the types of its neigh- borhood (e.g., Neumann or Moore neighborhood) (Horibe et al., 2021). Practically, let N be the size of the design space, the MDP state of design step is represented by sf {( de ...., s..,, Here, the design policy Tep [s Wx,) is a multilayer perceptron parameterized by 0p. which maps of categorical action distribution created by the output logits. The dimension of the input layer corresponds to the number of cell's neighbors (e.g. in Figure 2), and the dimension of the output layer corresponds to the total number of material types (e.g. 5 in Figure 2). During training, the new cell state is determined by sampling from the distribution. While evaluating, the design policy selects the action that corresponds the highest logit value.
At the beginning of each episode of the first learning stage, a single voxel placed at the center of the design space is used the initial seed, and the rest voxels' states are set to zero. After Np design steps, all voxels are updated according to the policy's ' outputs. In the next learning stage, the converged morphology from last stage will be placed at the center of the design space as the new initial seed. Mcanwhile, the weights of the learned design policy are also inherited. Since the morphology defines the connectivity, here, we let the design policy learn the connectivity on its own. We do not assign any design-related rewards to the design steps and terminate the episode if the resulting design does not meet the connectivity or has no actuators.
4.2 CONTROL POLICY
Once a robot designed, is presented in the simulator and consumed by the control policy The (aflsf, sf, V,). We model the local observations of all voxels sequence. The input state at time step is represented as sf. = {s"", + where s = {s"" .8 s...."""" "x s is composed each voxel's local information which contains the relative position of its four corners with respect to the center of mass of the robot and its material information. sy is the task-related observation such as terrain information of the environment and geal-relevant information.
5

---

Published as a conference paper at ICLR 2023
To capture the internal dependencies between voxels and dynamically accommodate changes in morphologies, we incorporate the self-attention mechanism into the control policy. As shown in Figure 3, we apply learnable position embedding layer after the linear projection layer. The local observation, the output feature of the Transformer and the task-related observation are concatenated before passing them through decoder. In our work, the control policy outputs the mean js of the continuous action, and we use a constant standard deviation (a fixed diagonal covariance matrix 2) to create the Gaussian distribution Tho (af) si, V,) N (p(sf, Vz), 2.). All actions are clipped within the range of [0.6, 1.6]. During training, the action is randomly sampled from the distribution. While evaluating the policy, only the mean action is used for control.
4.3 POLICY UPDATE AND THE CURRICULUM
We simultaneously train the design policy and the control policy using PPO, which is based on the Actor-Critic architecture. In our work, the critic network has the same architecture as the control policy network. It estimates the value function Es, 12 7'r] which suggests possible policy distribution. Here, we estimate the value for the whole morphology by averaging the value per voxel. With the policy gradient method, all policies are updated in the direction of maximizing the predicted value. When estimating the value of a design step, we set the environment state s" to zero. We apply predefined curriculum according to the training progress as the number of possible robot designs increases exponentially with each additional cube. During training, we increase the dimension of the design space when the target number of policy iterations reached.
EMPIRICAL EVALUATION
In this section, we demonstrate the effectiveness of CuCo on various tasks. We seek to answer the following questions: (1) Does CuCo provide an effective mechanism for learning design and control VSRs? (2) How does CuCo compare to prior approaches which learn from scratch in the space of target size in terms of the final performance and the produced morphologies? (3) What impact does each component of CuCo bring to the learning process?
5.1 EXPERIMENT SETTINGS
We create modular robot design space described in Section 4 and 8 task-related environments based on Evolution Gym (Bhatia et al., (0.1). The environment details can be found in Appendix A and our codes available online' We compare CuCo with number of approaches, both RL-based and population-base based. All baselines are introduced as follows:
CuCo-NCU: CuCo-NCU utilizes the same network architectures as CuCo but eliminates the cur- riculum component. learns to design and control from scratch the space target size. This method helps us investigate the effectiveness of the curriculum.
Additionally, we compare CuCo with three evolutionary methods proposed in (Bhatia et al., 2021). The inner loop of control optimization is also driven by PPO, and the outer loop of morphology optimization is implemented by three state-of-the-at design optimization algorithms.
vector where each element is tailored to the voxel's material property in order. It keeps a proportion of robots with high fitness as survivors at the end of each generation, and then the survivors undergo mutation conducted by randomly changing each voxel.
GA: Genetic Algorithm directly encodes the VSR's morphology
CPPN-NEAT: In this method, the morphology of VSR indirectly encoded by Compositional Pattern Producing Network (CPPN) (Stanley, 2007; Cheney et al., 2013), which relies on having access to the whole design space. The training of CPPN accomplished by NeuroEvolution of Augmenting Topologies (NEAT) (Stanley & Mikkulainenn, 2002).
BO: Bayesian Optimization (BO) is widely used help solve black-box problems. trains surrogate model based on Gaussian Process (GP) to alleviate the computational burden of expensive fitness evaluation (Kandasamy et al., 2018).
hhtpp:://gt thub. com Yuxi ng- wang THU arkvoGym

---

Published as a conference paper at ICLR 2023
Table I: Mean and standard error of all results across environments and baselines. Results with the highest mean are shown in bold. * indicates that the method reaches the upper bound of the score.
Environment
CuCo
CuCo-NCU
GA
CPPN-NEAT
BO
Walker
*10.46 + 0.01
8.90 = 2.01
*10.45
0.02
+10.46 = 0.01
9.38 + 1.13
Pusher
11.74 1.43
1.73 + 2.19
7.88 + 1.30
9.66 1.24
8.73 + 1.16
Jumper
13.08 1008::0 1.09
0.79 = 1.98
188
3.82
1.28 = 0.78
UpStepper
4.40 1.40:00 0.28
2.45 = 1.70
2.58+0.74
3.65 +
2.43 + 0.53
PlatformJumper
3.13 0.93
1.58 = 0.19
2.550
2.36 0.07
1.60 0.03
Thrower
3.59 + 1.61
2.88 = 0.60
1.75+0.19
1.60 + 0.28
1.91 + 0.21
Climber
0.54 0.09
0.28 + 0.28
0.58
1.54 0.52
0.83 + 0.91
Lifter
0.00 = 0.00
0.00 = 0.00
0.04
0.00 0.00
0.10 - 0.16
CuCo
CuCo-NCU
GA
CPPN-EAT
BO
Stage
Stage
Stage3
2.95
R: 6.57
R: 11.95
R: 1.63
R: 7.07
R: 9.97
R: 8.71
11,26
12,80
R: 3,82
R: 7.71
R: 6.05
R: 7.26
5.14
11.06
12.80
R 0.13
R: 6.91
R: 8.03
R: 6,87
Figure 4: Visualization of converged morphologies of all methods across three different runs in Pusher. Below each VSR is its average performance over tests. More visual results can be found in this video.
Following the convention in the literature (Cheney et al., 2013; Horibe et al., 2021), we consider 2-D design space where all dimensions equal in size. The curriculum defines three stages (3 x 3 + 5 x 5 + 7 x 7) for Walker, Pusher, Jumper and Lifter. For the remaining tasks, the curriculum is set (3 3 5 5). We run all experiments with the same number of time steps. For CuCo, this amounts to 1000 policy updates per stage. We employ a population of 12 agents for all population- bbased methods and report the performance of the best individual. Further implementation details can be found in Appendix D.
5.2 OVERALL PERFORMANCE
In Table 1, we list the final performance of each method, and the result is reported over different runs per environment. Learning curves for all methods and visualization results of converged mor- phologies can found Appendix B.I. We observe that within error margins, CuCo outperforms the baselines in 5 of the 8 environments, while only CPPN-NEAT and BO achieve limited suc- cess in Climber and Lifter, respectively. In Section 6, we discuss in detail these two cases. Without the guidance of the curriculum, CuCo-NCU performs poorly on most tasks since learning to design and control from scratch in a high-dimensional space is challenging.
We demonstrate the converged morphologies of all methods across three different runs and their corresponding performance in Figure 4, and for CuCo, we visualize the converged design of each stage. Direct encoding methods like GA and BO prefer horse-like morphologies composed of ran-

---

Published as a conference paper at ICLR 2023
Pusher
Walker
12.5
10.0
10.0
Perforance
Performance
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
0.0
1.5
2.0
3.0
0.0
0.5
2.0
2.5
3.0
Policy iterations (x10*)
Policy iterations (x10*)
Figure 5: Ablations of the self-attention mechanism and the curriculum.
domly organized voxels. However, voxels from the last two robots created by GA often collide with each other, preventing robots from learning efficient control policies, CuCo-NCU fails to discover practical policies. In contrast, we find that CuCo can develop beneficial homogeneous tissue struc- tures by module reuse, which can also be observed in many creatures, such as insects and vertebrates with several repeated body segments determined by homologous genes different species (Dellaert, 1995). When the robot is moving, we can see that groups of voxels are expanded or compressed simultancously, leading to coordinated behaviour.
Interestingly, VSRs designed by CuCo obtain different skills for pushing a box. The first rhino-like robot chooses to push the box with its "horn". The second robot makes full use of vertical actuators to let the box roll forward, while the third one uses horizontal actuators pop the box out. As shown in the video, CuCo also produces robots that can keep the balance as well as find a better launch angle when throwing the box. Although GA and BO produce VSRs with arms, they not strong enough to throw the box further. CPPN-NEAT generates much simpler patches based on the global information of the design space. In Appendix B.2, we further investigate differences between CPPN and NCA when chosen as the indirectly encoded design policy for CuCo. In sum, CuCo holds great potential for exploiting practical design and control patterns learned in small spaces to produce high- hegh-performing robots in larger dimensional settings.
5.3 ABLATIONS
We perform ablation studies to take closer look at the behaviour of CuCo. Firstly, we demonstrate the learning curves of CuCo and CuCo-NCU (Figure 5) to show that following easy-to-difficult curriculum much more efficient than learning from scratch the target design space. Especially in Pusher, CuCo-NCU spends large amount of time learning the connectivity. Secondly, we remove the self-attention mechanism used in our control policy and construct a new ablation CuCo-NAT. Evidently, after removing the self-attention mechanism, the control policy fails to dynamically adapt to morphological changes, resulting in much lower performance. In Appendix C, we analyze the attention matrices that CuCo learns. These ablations indicate that the curriculum and the self- attention mechanism are indeed crucial for finding better solutions.
Another key feature of CuCo is the mechanism of inheritance. Between the two learning stages, the converged morphology, the learned design and control policies are inherited. We perform additional ablation studies to measure the impact of the inheritance the subsequent learning process. We design the following ablations: (1) CuCo-NIM: No inheritance of the converged morphology from last stage: (2) CuCo-NID: No inheritance of the design policy. We reset the parameters of the design policy at the end of each stage; (3) CuCo-NIC: No inheritance of the control policy. We reset the parameters of the control policy at the end of each stage; (4) CuCo-NIDC: No inheritance of the design policy and the control policy. We reset all network parameters at the end of each stage. The learning curves of all ablations are shown Figure 6, and for better visualization, we plot these curves from the second learning stage. By comparing CuCo with CuCo-NID and CuCo- NIDC, it is worth noting that the inheritance of the design policy plays a critical role during the learning process. Even if the control policy is reset (CuCo-NIC), the performance is less affected. Additionally, the inheritance the converged morphology also contributes to the learning process, morphologies can be quickly built by expanding the old morphology.
8

---

Published as a conference paper at ICLR 2023
Pusher
Walker
12.5
Ddbo D.V-a
10.0
10.0
Perforanne
Performanee
7.5
5.0
5.0
2.5
2.5
0.0
2.0
3.0
0.0
0.5
2.0 Policy iterations (x10*)
Policy iterations (x10*)
Figure 6: Ablations of the inheritance mechanism. The plots show that the inheritance of the design
policy plays critical role during the learning process.
DISCUSSION AND CONCLUSION
In this work, we proposed CuCo, curriculum-based co-design method for learning to design and control voxel-based soft robots through easy-to-diffiuult process. Compared with previous ap- proaches that learn from scratch in the target design space, our method is more efficient in producing larger robots with better performance as it follows predefined curriculum and effectively exploits the knowledge of design and control obtained in small dimensional settings. Empirical evaluations on various robot control tasks show that CuCo outperforms prior methods in terms of the final per- formance, producing intriguing robot designs with repeated body segments.
However, compared with GA and BO, also observe that CuCo tends to produce more regular shapes, resulting in failures on some complex tasks (e.g., Climber and Lifter) which may require robots have irregular body structures (e.g., thin limbs fingers). We speculate that there are two reasons for the failure: (1) Design. In our work, NCA is encouraged to learn the connectivity on its own and do not assign any design-related rewards. Specifically, we check the connectivity at each design step and terminate the episode if the resulting design does not meet the connectivity or has no actuator voxels. Therefore, NCA becomes conservative in using empty voxels. Due to the mechanism of inheritance, this kind of design pattern would be further extended in larger design spaces, restricting the exploration of irregular morphologies, (2) Control. For some challenging tasks, when it is hard for the control policy explore and exploit the environmental reward, the de- sign policy is also hard to be informed because we formulate the processes of design and control into single MDP. Fortunately, the failures also suggest that we add design rewards guide NCA to explore some counterintuitive structures, and using more complex NCA architectures (Sudhakaran al., 2022) may also help the exploration of irregular designs. Hence important future work is creating an efficient mechanism for balancing the exploration and exploitation abilities of NCA.
During the learning process, we do not constrain the number of modules for each type of material. Thus, the optimization is uncontrolled. A future direction would be to investigate the constrained co-design method. We found that CuCo performs well under the fixed curriculum, promising direction of future work would involve extending our method to automatic curriculum learning. In CuCo, two kinds of developmental processes happen on different timescales. Within each stage, the robots grow under the restriction of body size. From one stage to the next, robots will become bigger and stronger. However, size is not the only morphological characteristic living creatures. An important line of exploration is to extend CuCo help study the morphogenesis of virtual creatures. Co-designing physical VSRs (Kriegman et al., 2019; 99; 2020) using our method is expected to further extend the horizon.
7 ACKNOWLEDGEMENT
We thank the anonymous reviewers for their helpful comments revising the paper. This work is partially supported by Tencent Rhino-Bird Research Elite Program (2022).

---

Published as a conference paper at ICLR 2023
REFERENCES
Dylan S. Banarse, Yoram Bachrach, Siqi Liu, Guy Lever, Nicolas Manfred Otto Heess, Chrisantha Fernando, Pushmeet Kohli, and Thore Graepel. The body is not given: Joint agent policy learning and morphology evolution. In AAMAS, 2019.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICMI. '09, 2009.
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym: A large-scale benchmark for evolving soft robots. In NeurlPS, 2021.
Tianjian Chen, Zhanpeng He, and Matei Ciocarlic. Hardware as policy: Mechanical and compu- tational co-optimization using deep reinforcement learning. In CoRL, 2020.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. GECCO '13, 2013.
Frank Dellaert. Toward biologically defensible model of development. 1995.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 33: 1349-1 2020.
Kuan Fang, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Discovering generalizable skills via auto- mated generation of diverse tasks. ArXiv, 0abs/2106.19995 2021.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12, 2021.
Agrim Gupta, Linxi (Jim) Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. ArXiv, abs/2203. 11931, 2022.
David R Ha. Reinforcement learning for improving agent design. Artificial Life, 25:352-366 2019.
Donald Hejna, Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. ArXiv, abs/2102.13100 2021.
Kazuya Horibe, Kathryn Walker, and Sebastian Risi. Regenerating soft robots through neural cellu- lar automata. In EuroGP, 2021.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In /CML, 2020.
Atil Iscen, George Yu, Alejandro Escontrela, Deepali Jain, Jie Tan, and Ken Caluwaerts. Learning agile locomotion skills with mentor. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2019-2025, 2021.
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-18977 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950. PMLR, 2021b.
Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff G. Schneider, and Barnabás Póczos. Paral- lelised bayesian optimisation via thompson sampling. In AISTATS, 2018.
Sam Kriegman, S. Walker, Dylan S. Shah, Michael Levin, Rebecca Kramer- Botter--ottiilio,, and Josh C. Bongard. Automated shapeshifting for function recovery in damaged robots. ArXiv, hbs/1995.09264, 2019.
Sam Kriegman, Amir Mohammadi Nasab, Dylan S. Shah, Hannah Steele, Gabrielle Branin, Michael Levin, Josh C. Bongard, and Rebecca Kramer-BBii Kramer-Bottiglis Bottiglio. Scalable sim-to-real transfer of soft robot designs. 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft). pp. 359-366, 2020.
10

---

Published as a conference paper at ICLR 2023
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based based incompatible control. ArXiv, abs/2010.01855, 2021.
Rich Gang Li, A. Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manipu- lation using relational reinforcement learning. 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4051-4058. 2020.
Sida Liu, David Matthews, Sam Kriegman, and Josh C. Bongard. Voxcraft-sim, gpu-accelerated voxel-based physics engine. 2020.
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In CoRL, 2019.
Pingchuan Ma, Tao Du John Z. Zhang, Kui Wu, Andrew Spielberg, Robert K. Katzschmann, and Wojciech Matusik. Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation. ACM Trans. Graph., 10:132:1-132: 2021.
Tambet Matiisnn Avital Oliver, Taco Cohen, and John Schulman. Teacher Teacher-student student curriculum learn- ing. IEEE Transactions on Neural Networks and Learning Systems, 31:3732-3740, 2020.
Eric Medvet, Alberto Bartoli, Federico Pigozzi, and Marco Rochelli. Biodiversity in evolved voxel- based soft robots. Proceedings of the Genetic and Evolutionary Compulation Conference, 2021.
Eric Medvet, Giorgia Nadizar, and Federico Pigozzi. On the impact of body material properties on neuroevolution for embodied agents: the case voxel-based soft robots. Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2022.
A. Mordvintsev, E. Randazzo, Eyvind Niklasson, and Michael Levin. Growing neural cellular tomata. Distill, 2020.
Tønnes F. Nygaard, David Howard, and Kyrre Glette. Real world morphological evolution is fea- sible. Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, 2020.
Rasmus Berg Palm, Miguel González Duque Shyam Sudhakaran, and Sebastian Risi. Variational neural cellular automata. ArXiv, abs/2201.12660 2022.
Jack Parker-Holdde Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint 22X3v:2003.0/33 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-perffrmance deep learning library. In NeurlPS, 2019.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros. Learning to control self-assembling morphologies: study generalization via modularity. In NeurlPS, 2019.
Rolf Pfeifer and Josh C. Bongard. How the body shapes the way we think a new view on intelli- gence. 2006.
Rolf Pfeifer, Fumiya lida, and Max Lungarella. Cognition from the bottom up: on biological inspi- ration, body morphology, and soft materials. Trends in Cognitive Sciences, 18:404-413, 2014.
Federico Pigozzi, Yujin Tang, Eric Medvet, and David Ha. Evolving modular soft robots without explicit inter-module communication using local self-attention. Proceedings of the Genetic and Evolutionary Computation Conference, 2022.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yve Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning, pp. 835-85 PMLR, 2020.

---

Published as a conference paper at ICLR 2023
Alvaro Sanchez- Nicolas Manfred Otto Heess, Jost Tobias Springenberg, Josh Merel, Mar- tin A. Riedmiller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. ArXiv, abs/1800.01242. 2018.
Charles B. Schaff, David Yunis, Ayan Chakrabarti, and Matthew R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA). pp. 9798 900889905,000 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 707.003477
Dylan S. Shah, Bilige Yang. Sam Kriegman, Michael Levin, Josh C. Bongard, and Rebecca Kramer- Bottiglio. Shape changing robots: Bioinspiration, simulation, and physical realization. Advanced Materials, 33, 2020.
Karl Sims. Evolving 3d morphology and behavior by competition. Artificial Life, 1:353-372, 1994.
Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojciech Matusik, and Daniela Rus. Learning-in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations. NeurlPS, 2019.
Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of develop- ment. Genetic Programming and Evolvable Machines, 8:131-162, 2007.
Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary Computation, 10:99-127, 2002.
Shyam Sudhakaran, Elias Najarro, and Sebastian Risi. Goal-guided neural cellular automata: Learn- ing to control self-organising systems. ArXiv, abs/2205 200s02000500806, 2022.
Jacopo Talamini, Eric Medvet, and Stefano Nichele. Criticality- evolution of adaptable mor- phologies of voxel-based soft-robots. Frontiers Robotics and AI, 8, 2021.
Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: Learning transferable polices by inferring agent morphology. In ICML, 2022.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. ArXiv, abss/706.00362. 2017.
Kathryn Walker and Helmut Hauser. Evolution of morphology through sculpting in a voxel based robot. In ALIFE, 2021.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Poet: open-ended coevolution of environments and their optimized solutions. Proceedings of the Genetic and Evolutionary Com- putation Conference, 2019a.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In /CML, 2020.
Tingwu Wang. Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. ArXiv, abs/1 906.05337, 2019b.
Ye Yuan, Yuda Song. Zhengyi Luo, Wen Sun, and Kris M. Kitani. Transform2act: Learning transform transform-and-comm and-control policy for efficient agent design. ArXiv, abs/2110.03652) 2022.
12

---

Published as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
In this section, we provide details about the tasks simulated in Evolution Gym platform (Banarse al., 2019).
Position. p" is a 2-dim vector that represents the position the center of mass of an object o in the simulation at time pf. and p are and y components of this vector, respectively. p" is calculated by averaging the positions of all the point-masses that make up object at time I.
Velocity. 1 is 2-dim vector that represents the velocity of the center of mass of object in the simulation at time t. bg and by are z and y components of this vector, respectively. v0 calculated by averaging the velocities of all the point-masses that make up object at time
Orientation. 0 is a 1-dim vector that represents the orientation of an object o in the simulation at time L. Let pt be the position of point mass of object o. 0" is computed by averaging over all i the angle between the vector pp at time t and time O. This average is weighted average weighted by |p:
pl]| time
Other observations. hg(d) is a vector of length (2d+1) that describes elevation information around the robot below center mass. More specifically, for some integer S d, the corresponding entry in vector (d) will be the highest point of the terrain which is less than Pe between range of [z, x + 1] voxels from P. in the z-direction.
A.1 WALKER
Figure 7: Walker environment.
In this task, the robot is rewarded by walking as far as possible on flat terrain. The target design space is 7 x 7. The task-specific observation is rabot and the reward R is:
Ap robot
(1)
which rewards the robot for moving in the positive r-direction. The robot receives a reward of 1 for reaching the end of the terrain. The episode duration reaches a 500 time steps.
A.2 PUSHIRR
Figure 8: Pusher environment.
In this task, the robot pushes box initialized in front of it. The target design space is 7 7. The task-specific observation formed by concatenating vectors {u hour pboz p rooot robot }. and the reward R R1 Rg, where R is:
0.5 Ap. 0.75 Aphor
which rewards the robot and box for moving in the positive x-direction, and Rg is:
R₂
A 1p
box
robob
(3)
13

---

Published as a conference paper at ICLR 2023
which penalizes the robot and box for separating in the z-direction. The robot also receives a time reward of 1 for reaching the end the terrain. The episode duration reaches a 500 time steps.
A.3 UPSTEPPER
Figure 9: UpStepper environment.
In this task, the robot climbs up stairs of varying lengths. The target design space is 5 x 5. The task- specific observation is formed by concatenating vectors torbbot (rrboo hrobot (5)}, and the reward R is:
Ap Aprobut
(4)
which rewards the robot for moving in the positive r-direction. The robot also receives a one-time reward of 2 for reaching the end of the terrain. The episode duration reaches 600 time steps.
A.4 JUMPER
Figure 10: Jumper environment.
In this task, the robot jumps high possible in place flat terrain. The target design space is 7 x 7. The task-specific observation is formed by concatenating vectors robot h (2)}, and the reward is:
R = 10 Ap robot 5- App robot
(5)
which rewards the robot for moving in the positive y-direction and penalizes the robot for any motion in the r-direction. The episode duration reaches a 500 time steps.
A.5 PLATFORMJUMPER
Figure 11: PlafformJumper environment.
In this task, the robot traverses series of floating platforms different heights. The tar- get design space is 5 x 5. The task-specific observation is formed by concatenating vectors robot grabot hy (5)) and the reward R is:
R R= Ap robot
(6)
which rewards the robot for moving in the positive x-direction, The robot also receives a one-time penalty of -3 for rotating more than 90 degrees from its originally orientation in either direction or for falling off the platforms (after which the environment resets). The episode duration reaches 1000 time steps.
14

---

Published as a conference paper at ICLR 2023
A.6 THROWER
Figure 12: Thrower environment.
In this task, the robot throws a box initialized on top of it. The target design space x The task-specific observation formed by concatenating vectors yobo pboz p probot robot j, and the reward Rı + R2, where Rí is:
R Ap
Appor
which rewards the box for moving in the positive x-direction, and R2 is:
gabot Aprobot
if p probot 2
R2
(8)
otherwise
which penalizes the robot for moving too far from duration reaches a 300 time steps.
0 when throwing the box. The episode
A.7 CLIMBER
Figure 13: Climber environment.
In this task, the robot climbs as high as possible through flat, vertical channel. The target design space is 5 x 5. The task-specific observation is robot and the reward R is:
R Ap robot
which rewards the robot for moving in the positive y-direction. The robot also receives one-time reward for reaching the end of the channel. The episode duration reaches 400 time steps.
A.8 LIFTER
Figure 14: Lifter environment.
In this task, the robot lifts a box out of hole. The target design space is 7. The task task-specific observation is formed by concatenating vectors box probol robot yoor Obox }. and the reward R = R2 + R2 + R3, where Rg is:
R1 10 Apter
boz
(10)
which rewards the robot for moving the beam in the positive dirrection, and Rg is:
R₂ 10 A 0x
P
bow
(11)
15

---

Published as a conference paper at ICLR 2023
Jumper
Pofomaacc
10.0
Pororaacce
5.0
5.0
2.6
0.0
-8
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.5
1.5
2.0
2.5
Stmullation stopo (x101)
Elmuiation stopo (x110)
3imulasion stops (x107)
UpStappar
Threwar
Platforuuuimpr
0.50
1.00
0.50
0.75
1225
3.00
1.25
Senuliion stapa 10")
Senutadion atagu (x.)
(x:30)
Climber
Lifter
2.5
0.5
1.5
Peformenc
1.0
-1.0
0:25
1.00
(x10")
Figure 15: Learning curves of all methods.
which penalizes the robot for moving the box away from a goal x z-position.g.- This ensures that the robot lifts the box straight up, and R3 is:
if prooot 2 by otherwise
R3 = 20
(12)
App
which penalizes the robot for falling below a threshold height by (at which point the robot has fallen into the hole). The episode duration reaches 300 time steps.
B ADDITIONAL EXPERIMENTS
B. LEARNING CURVES AND MORPHOLOGY RESULTS
Figure 15 plots the learning curves of all methods across various environments. In Figure 16, we provide visualization results of morphologies constructed using CuCo and baselines. In Figure 17, we demonstrate the development of robot morphologies during CuCo's learning process in UpStep- per and Pusher environments, respectively.
B.2 THE CHOICE OF DESIGN POLICY
We perform an additional experiment investigate the importance of using the NCA architecture as CuCo's design policy. Here, we replace NCA with Compositional Pattern Producing Network (CPPN) (Stanley, 2007) which takes the global information (e.g., & and y coordinates of the design space) as the input and directly generates the robot design parameters. We refer to this method as CuCo-CPPN. Note that the converged morphology and the design policy are not inheritable in CuCo- CPPN as the input dimension of CPPN is related to the design space. Figure 18 shows the learning curves of CuCo-NCA and CuCo-CPPN in Pusher and Walker. Evidently. CuCo-NCA outperforms CuCo-CPPN in terms of convergence speed and final performance. It shows that NCA can encode practical design patterns and successfully leverage them in larger design spaces, while CPPN does not significantly facilitate the learning process.
16

---

Published as a conference paper at ICLR 2023
CuCo
CuCo-NCU
GA
CPPN-NEAT
BO
UpStepper
4.25
R: 1.95
2.37
R: 3.06
Walker
R: 10.46
7.98
10445
10446
R: 8.87
J umper
R: 14.50
1.-1.05
1.33
5.07
R: 1.67
Platform jmper
R: 4.02
R: 1.52
R: 1.80
2.03
Thrower
4.97
R:
1.67
R: 1.12
Climber
R: 0.63
0.81
R: 2.07
R: 1.77
Lifter
R: 0.00
R: 0.00
R- 0.002
R: 0.00
R: 0.26
Figure 16:
Visualization of converged morphologies of all methods across various environments. Below each VSR is its average performance over 5 tests.
C ATTENTION MATRICES ANALYSIS
Self-attention brings better interpretability than multilayer perceptron. In this section, analyze the attention matrices that CuCo learns. We use only one Transformer encoder layer, thus we visualize the generated attention matrix after input state passes through the attention layer. Figure 19 demonstrates the learned attention matrices the critic network when designing a robot, where the voxel is assigned with more attention scores. Figure 20 shows attention matrices produced by
17

---

Published as a conference paper at ICLR 2023
Curriculum
Stage 2
Stage
0.4
0.6
0.8
1.0
Policy iterations (x 10²)
Curiculum
Stage 3
Stage 2
Stage
0.2
0.4
0.6
0.8
1.0
Policy iterations 10a)
Figure 17: Visualization of CuCo's learning process. The æ-axis represents the number of policy iterations and the y-axis represents the curriculum. We show morphologies produced within each stage from left to right.
Pusher
Walker
12.5
Dubx MDX
Perfomance
Perfommenc
7.5
7.5
5.0
2.5
0.0
0.0
0.5
1.5
2.0
3.0
0.0
0.5
1.5
2.0
Policy llerations (x10*)
Policy lterations (x10*)
comparison of using NCA and CPPN as the design policy. The results are averaged over 7 runs and suggest that the NCA architecture contributes to the performance greatly.
Figure 18:
the control policy network. The color each attention score tells the strength of the compatibility between inputs and interprets what is driving the current behaviour. When the robot's front foot (voxel 9) or the rear foot (voxel 7) touches the ground, the corresponding voxels are assigned with greater wights, which is consistent with humans' intuition and common sense.
18

---

Published as a conference paper at ICLR 2023
Figure 19: Attention matrices of 10 design steps.
V-9
Figure 20: Attention matrices of control steps.
D IMPLEMENTATION DETAILS
We use Py PyTorch (Paszke et al., 2019) to implement our proposed method. We take the official implementation of Transformer from Pytorch and add the position embedding. All hyperparameters for CuCo are listed in Table For baseline algorithms, we use the official implementations of GA, CPPN-NEAT and BO from Evolution Gym (Bhatia et al., 2021). We employ population of 12 agents for population-based methods. Following
convention in the literature, we use multiple CPU threads to accelerate the training of PPO. For all the environments used in this paper, it takes around 1 day to train our model on a computer with 12 CPU cores and an NVIDIA RTX 3090 GPU.
19

---

Published as a conference paper at ICLR 2023
Table Hyperparameters of CuCo.
Hyperparameter
Value
Use GAE
True
GAE parameter
0.95
Learning rate
2.5 10-4
Use linear learning rate decay
True
Clip parameter
0.1
Value loss coefficient
0.5
Entropy coefficient
0.01
Time steps per rollout
2048
Num processes
4
PPO
Optimizer
Adam
Evaluation interval
10
Discount factor y Clipped value function
0.99
True
Observation normalization
True
Observation clipping
- 10, 10
Reward normalization
True
Reward clipping
- 10, 10
Policy epochs
8
Policy iterations per stage
1000
Neighborhood
Moore
Input dimension
9
Output dimension
5
NCA
Growth factor
False
Hidden layers
(64, 64)
Activation function
Tanh
Design steps
10
Number of layers Number of attention heads Embedding dimension Feedforward dimension
1
1
Transformer
64
128
Non linearity function Dropout
ReLU
0.0
20


Reviews:
[{"review_id": "gY-9RZKle-", "reviewer": "Reviewer_iFUz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the effectiveness of using a predefined curriculum over the size of a Voxel-Based Soft Robot (VSR) in co-training design and control policies simultaneously using RL (PPO). Their results in training a target 7x7 VSR configuration provides strong evidence that such curricula over VSR body size is effective for producing more performant morphologies and associated control policies on 4 environments in simulation. The architectural and training procedure used in this paper is largely taken from Yuan et al, 2022 (Transform2Act).", "summary_of_the_review": "Overall, this paper provides a clean investigation of the impact of body-size curricula in co-training morphology design and control policies for VSR. The experiment setting is sound, including several VSR environments, key ablations, and sensible baselines. The main finding—that body-size curricula lead to improved morphologies and control policies in simulation—seems valuable in informing future work in this area. ", "strength_and_weaknesses": "### Strengths\n- The paper is overall well-written with clear figures and diagrams.\n- The experimental setting and baselines are clearly communicated and seem appropriate for addressing their research question about whether body-size curricula is useful for co-designing VSR control and morphology.\n- The experiments comprehensively ablate each component of the proposed method: the use of a curriculum, the use of self-attention, and whether the morphology or design policy are inherited across curriculum stages.\n\n### Weaknesses\n- The study can benefit from an ablation of the choice of using an NCA architecture for the design policy.\n- Several key details are left unclear in the current paper, as detailed in the Clarity section.\n- A primary weakness with this work is that the results are only validated in simulation. The goal of VSR is to transfer to real robots, and the so the study would be much more impactful if the authors can demonstrate the effectiveness of the learned morphologies and controllers in a sim2real setting.\n\nLastly, there is important related work around automatic curriculum learning that the authors should cite, especially around unsupervised environment design, which describes the same problem setting that the authors investigate (where the environment is the combination of the task and morphology):\n- Matiisen, Tambet, et al. \"Teacher–student curriculum learning.\" IEEE transactions on neural networks and learning systems 31.9 (2019): 3732-3740.\n- Portelas, Rémy, et al. \"Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments.\" Conference on Robot Learning. PMLR, 2020.\n- Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. \"Prioritized level replay.\" International Conference on Machine Learning. PMLR, 2021.\n- Dennis, Michael, et al. \"Emergent complexity and zero-shot transfer via unsupervised environment design.\" Advances in neural information processing systems 33 (2020): 13049-13061.\n- Jiang, Minqi, et al. \"Replay-guided adversarial environment design.\" Advances in Neural Information Processing Systems 34 (2021): 1884-1897.\n- Parker-Holder, Jack, et al. \"Evolving Curricula with Regret-Based Environment Design.\" International Conference on Machine Learning. PMLR, 2022.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DcXFq7SCn3y", "reviewer": "Reviewer_Qe3n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new curriculum-based method for co-designing morphology and control of voxel-based soft robots. This curriculum-based method expands the design space from a small size to the target size using reinforcement learning with a predefined curriculum. To address incompatible state-action spaces, local observations of robot voxels are modeled as a sequence and self-attention is used to control the voxels.", "summary_of_the_review": "Although several technical details need to be clarified, this paper proposes a new method and addresses several specific challenges for morphology and control of simulated voxel-based soft robots.", "strength_and_weaknesses": "Strength: \n\n+ The paper addresses an important problem on co-design of morphology and control with the potential to greatly impact voxel-based soft robots.\n\n+ The paper identifies several interesting challenges for soft robot co-design, such as the high dimensionality of the joint design and control space and the difficulty in generalizability. Methods are designed to address these specific challenges.\n\n+ The paper is well organized and well written.\n\nWeakness:\n\n- The goal of the paper is the co-design of morphology and control. However, the approach assumes that \"The morphology of the robot is unchangeable during the interaction with the environment\". While the control policy uses the learned morphology, but how does the control inform or help morphology design?\n\n- In the related work section, the paper argues that methods such as Transform2Act by Yuan et al (2022) have \"the limitation of aggregating multi-hop information\". In the experiment section, the paper states that CuCo-NCU (CuCo without the curriculum component) is similar to Transform2Act. How is this multi-hop information aggregated by CuCo. Other than the curriculum, what are the key differences between CuCo and Transform2Act?\n\n- While simulators such as Evolution Gym and MuJoCo are helpful to evaluate theories, out of curiosity, how are these simulators, the proposed method, and the experimental results applicable to real soft robots? For example, what are the existing prototypes of real voxel-based soft robots that can use the proposed method for co-design?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "85B2xT0bqt", "reviewer": "Reviewer_vM9V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel curriculum-based co-design method for evolving soft voxel-based robots. The main contributions are:\n- A novel curriculum mechanism that effectively evolves robots from simple to large design spaces.\n- A transformer-based control policy representation, enabling adaptation to arbitrary dimensional observations due to morphology changes.\n- A novel design policy based on Neural Cellular Automata (NCA), making end2end training of both design and control policy possible.\n- Impressive visual behavior and empirical performance compared to baseline methods with abundant ablation studies.", "summary_of_the_review": "This paper proposes a novel method with sufficiently new insights into soft robot co-design. While the approach is theoretically sound, on the practical side, I still have several concerns (evaluations on more environments, discussions on limitations and failure cases).", "strength_and_weaknesses": "Strengths:\n- A novel and sound technical approach. Each component of the approach is intuitive and effective (curriculum, transformer-based generalizable control policy, NCA-based design policy).\n- Good empirical performance with abundant ablation studies (Sec 5.4), which explains the necessity of all the algorithmic components.\n\nWeaknesses:\n- This paper only performs evaluations on 4 environments/tasks from Evolution Gym [Bhatia et al. 2021], which is not sufficient enough in my opinion to show the generalizability of the proposed approach. I would suggest at least evaluate on 6-10 environments from Evolution Gym, to convince people that the results are not cherry-picked.\n- Limitations or failure cases are not discussed well in the paper, and I think this is crucial. Is there any case in which CuCo fails to beat the baselines? \n\nQuestions:\n- By looking at the results, CuCo tends to produce regular shapes for the robot design compared to GA. It does not appear to evolve robots with some thin limb structures. However, my intuition tells me that for some complex tasks (such as climbing/throwing/lifting in environments), thin structures like limbs or fingers are the key to success. Could you please evaluate your method on those environments in Evolution Gym as well and report the performance?\n- Could you show the performance/convergence curves for the baseline methods as well?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster