BodyGen: Advancing Towards Efficient Embodiment Co-Design
Authors: Haofei Lu
Zhe Wu
Junliang Xing
Jianshu Li
Ruoyu Li
Zhe Li
Yuanchun Shi
Venue: ICLR 2025 Spotlight (2025)
Keywords: Reinforcement Learning
Primary area: reinforcement learning
TL;DR: This paper introduces BodyGen:  a novel framework for efficient embodiment co-design.

Abstract: Embodiment co-design aims to optimize a robot's morphology and control policy simultaneously. 
While prior work has demonstrated its potential for generating environment-adaptive robots, this field still faces persistent challenges in optimization efficiency due to the (i) combinatorial nature of morphological search spaces and (ii) intricate dependencies between morphology and control.
We prove that the ineffective morphology representation and unbalanced reward signals between the design and control stages are key obstacles to efficiency.
To advance towards efficient embodiment co-design, we propose **BodyGen**, which utilizes (1) topology-aware self-attention for both design and control, enabling efficient morphology representation with lightweight model sizes; (2) a temporal credit assignment mechanism that ensures balanced reward signals for optimization. With our findings, BodyGen achieves an average **60.03%** performance improvement against state-of-the-art baselines. We provide codes and more results on the website: https://genesisorigin.github.io.

Published as a conference paper at ICLR 2025
BODYGEN: ADVANCING TOWARDS EFFICIENT EMBOD- IMENT CO-DESIGN
Haofei Lu' Zhe Wu', Junliang Xing"!, Jianshu Li", Ruoyu Li Zhe Li³, Yuanchun Shi! Department of Computer Science and Technology, Tsinghua University, AAt Group (luhf23, 224 @malmmlll iss........ tsinghua. edu, en, (iixing, shiye S @tssinghua en [jianshu. 1, ruoyu. 11, 11,,1,1,, lizhe. 12 wtte///aanntuu..oooooooo
Jiansuu
ABSTRACT
Embodiment co-design aims to optimize a robot's morphology and control policy simultaneously. While prior work has demonstrated its potential for generating dwvironment-adaplee robots, this field still faces persistent challenges in opti- mization efficiency due to the (i) combinatorial nature of morphological search spaces and (ii) intricate dependencies between morphology and control. We prove that the ineffective morphology representation and unbalanced reward signals be- tween the design and control stages are key obstacles to efficiency. To advance towards efficient embodiment co-design, we propose BodyGen, which utilizes (1) topology-aware self-attention for both design and control, enabling efficient morphology representation with lightweight model sizes; (2) a temporal credit assignment mechanism that ensures balanced reward signals for optimization. With our findings, Body achieves average 60.03 % performance improvement against state-of-the-art baselines. We provide codes and more results on the website: github.io.
https: htttps:/genesisoote
INTRODUCTION
Species in nature are blessed with millions of years to evolve for remarkable capaci- ties to adapt to the environment (Pfeifer & Scheier, 2001; Vargas al., 2014). Time has gifted them with perfect physical bod- ies for movement and navigation, powerful processors for centralized information pro- cessing, and effective actuators for rapid in- teraction with their surroundings. Inspired by this observation, embodiment co-design (Sims, 1994; Ha, 2019; Yuan et al., 2021; Wang et al., 2023), where a robot's mor- phology and control policy optimized simultancously, has gained increasing attention and demonstrates significant potential various downstream fields, such automated robot design and bio-inspired robot generation (Kriegman et al., 2020; Nakajima et al., 2018; Judd et al., 2019; Pan et al., 2021; Whitman et al., 2023). However, this task encounters extreme difficulties: (1) the morphology search space is quite vast and combinatorial, with each morphology corresponding to unique action and state spaces; (2) evaluating each candidate design requires an expensive roll-out to find its optimal control policy, which is almost unfeasible for the expensive computation.
(a) Initial Designs
(b) Morphologies with Policies
Figure I: Embodied Agents generated by BodyGen.
Traditional evolutionary strategies (Sims, 1994: Wang al., 2018b; Zhao et al., 2020; Gupta et al., 202 lb) address these challenges through mutation nutation-based population optimization but suffer from inefficient sampling and scalability limitations, requiring large amounts of computation. While structural constraints like symmetry (Gupta et al., 2021b; Dong et al., 2023) reduce constant search complexity, such human priors may compromise functionality (Yuan al., 2021). Alternative
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Temporal Credit Assignment
Direct Communication Routes with Self-Attention
Right Leg
Left Limb B
Update
Leg
Right Limb
Morphology
Marpeingy & Oloccnaices
Action
Obscrvation
TopoPE
Inputs
Environment Evaluation
Outputi
Figure 2: Overview of BodyGen, which leverages RL-based framework for joint evolving of morphology and control policy, and an attention- based network equipped with Topology Position Encoding (TopoPE) for centralized message processing.
approaches employ modular GNN-based controllers (Huang al., 2020; Yuan al., 2021) for cross-morphology policy sharing, yet struggle with effective joint-level message aggregation (Kurin al., 2020).
In this work, we aim to further push the potential of embodiment co-design, by introducing a method that enables efficient generation of niigh-performance embodied agents while maintaining computational affordability. Here, we announce BodyGen, a reinforcement learning framework for efficient, environment-adapy adaptive embodied agent generation. Inspired by recent co-optimization approaches (Yuan et al., 2021), BodyGen directly use auto- rugo-rregessive transformers to generate an agent's morphology before executing environment interactions. Given an initial design (a. prompt), BodyGen can output optimal morphologies and corresponding controllers at the same time.
In contrast to previous methods, BodyGen utilizes a joint-level self-attention mechanism achieve direct message communication using transformers. We further propose topology-awand positional encoding for effective, lightweight morphology representation. Additionally, we addresses the inherent reward imbalance between morphology design gero-reward-guides. and control (rich- reward-gen -ward-ggided) phases: our enhanced temporal credit assignment mechanism dynamically balances reward signals across both stages, enabling coordinated optimization. The parameters of the whole BodyGen model is less than 2M, and a high-performance embodied agent can be generated using a single Nvidia GPU within 30 hours. To summarize, contributions
follows:
We propose BodyGen, an end-to-end reinforcement learning framework for efficient embodiment co-design.
We design a Morphology Self-Attention architecture (MoSAT) provide joint-to-joint mes- sage transition, featuring our proposed Topological Position Encoding (TopoPE) for efficient morphology representation.
We propose temporal credit assignment mechanism that ensures balanced reward signals in the morphology design and control phases, thus facilitating co-design learning.
Comprehensive experiments across various tasks demonstrate BodyGen's advantages against previous methods in terms of both convergence speed and performance. BodyGen achieves average performance improvement of 60.03% % against the state-of-the-art baselines.
RELATED WORK
Universal Morphology Control Embodiment co-design requires controlling robots with changeable morphologies and adapting to their incompatible action and state spaces. Universal Morphology Control (UMC), which employs a shared network to control each actuator separately, presents a
promising solution to this problem. To better perceive the topological structures of various morpholo- gies, some methods (Pathak et al., 2019; Wang et al., 2018a; Huang et al., 2020) employed Graph Neural Networks (GNNs) to enable communication between neighboring actuators. Recent works
2

---

Published as a conference paper at ICLR 2025
also use Transformers (Vaswani et al., 2017) overcome the limitations of multi-hop information aggregation brought by GNNs (Kurin et al., 2020; Hong et al., 2021; Gupta et al., 2021a: la; Dong et al., 2022). Despite these advancements, several challenges remain. Many existing methods are limited to parametric variations of a limited number (e.g. 2-3) of predefined morphologies, whereas a design space remains largelyy unexplored. Furthermore, most previous works do not fully leverage morphology information or only consider its simple form and even the usefulness of such information remains controversial (Kurin et al., 2020; Hong et al., 2021; Gupta et al., 2021a; Xiong et al., 2023). In this work, we prove that morphology information plays a crucial role, and the correctness of morphology representation significantly influences performance. To this end, we introduce a simple yet effective positional encoding technique, TopoPE, which facilitates message localization within the body and enhances knowledge sharing among similar morphologies. By representing the 2D topological structure with position embeddings, we explore the potential of autoregressive transformers for robotic design generation.
comprehensive morphology-agnoste
Embodiment Co-design As for embodied artificial agents, control policy (Lillicrap et al., 2015; Schulman et al., 2015a; Haarnoja et al., 2018; Schulman et al., 2017: Lowrey et al., 2018) has been well studied in the reinforcement learning and robotics community, while another critical component, the physical form of the embodiment, is currently attracting more and more attention (Kriegman al., 2020; Bhatia al., 2021; Xu al., 2021; Huang al., 2024). Embodiment co-design aims to optimize a robot's morphology and control simultaneously and is considered a promising way to stimulate the embodied intelligence embedded in morphology. Previous methods (Sims, 1994; Wang et al., 2018b; Gupta et al., 20) typically utilize evolutionary search (ES) to learn directly within the vast design space, which unavoidably brings inefficient sampling and expensive computation. A line of works (Wang et al., 201 20; Gupta et al., 2021a) introduces more human morphology priors, such as symmetry, to reduce the search space. Yuan et al. (2021) proposes jointly optimizing robot's morphology and control policy via reinforcement learning. This paper focuses on the RL-based approach for joint optimization for both morphology and control. We aim to establish comprehensive framework for embodiment co-design, systematically addressing key obstacles against efficiency during training.
3 PRELIMINARIES
Morphology Representation. The morphology of an agent can be formally defined as an undirected graph 9 = (V. E, A", A²), where each node E V represents a limb of the robot, and each edge e ( vi) E E represents joint connecting two limbs. A" and A are two mapping functions that map the limb node u to its physical attributes A V - A", and map the edge
(1)) to its joint attributes A= : E + A respectively. Here A = {A*) is the limb attribute space, consisting of attributes A like limb lengths, sizes and materials, and A = {A** is the joint attribute space consisting of attributes A like rotation ranges and maximum motor torques. Consequently, the design space D is defined on all valid robot morphologies G E D.
Co-Design Optimization. The fitness F of an agent represents its performance a specific environ- ment and is typically evaluated by rewards. In traditional control problems with fixed morphology Go. we aim to optimize its control policy π towards the optimal "r" = arg max F(T,, Go) for maxi- mum fitness. In co-design problems, we not only optimize the control policy but also the morphology design simultaneously. This co-design process is formulated as a bi-level optimization problem:
g arg max F ()
(3.1)
s.t. T
arg max F(mg,9),
Sg
where the inner loop defines the optimal control policy of given morphology, and the outer loop defines the optimal morphology using its optimal policy. Previous works typically use evolutionary algorithms (Sims, 1994; Wang et al., 2018b; Gupta et al., 2021b) to solve this problem. In this work, BodyGen leverages an RL-based framework and jointly optimizes both loops:
(-19)), ).9" = arg max (-)()))).) x(-1)).
(3.2)
using the universal control policy <-119), to facilitate knowledge sharing among agents with similar morphologies.
3

---

Published as a conference paper at ICLR 2025
MoSAT
Collective Actions: [a,]a
a1 0g
Padding Masks: P
Lincar Projector
aayl
Layer Norm
Attention
N
Transformer Block
Masks:
MoSAT
11
Interact 11
Self-Attention
Padded
States: [s
Environment
Layer Norm
Linear Projector
Align Batchify
Morphology Tokens
Collective
Cross-Agents
Morphology Inputs
{s,
Trajectories
(a) MoSAT Architecture
(b) Batch Processing MoSAT
Figure 3: The Morphology Self-Attention (MoSAT) architecture. (a) The sensor observations from different limbs projected hidden tokens for centralized processing with several MoSAT blocks and generate separate actions. (b) The MoSAT network processes different morphologies in a batch manner and learns universal control policy x-11)), thus improving training efficiency.
Reinforcement Learning. We define the problem formulation of Norpholoyy-Conditoone Re- inforcement Learning for embodiment co-design. We consider the augmented Markov Decision Process (MDP), which can be described by a 6-element tuple M = (S. A. T. R. DD ②). & is a flag distinguish design and control stages. S denotes the state space. A( represents the action space, where A( Design) changes the morphology of the agent, and A($ Control) defines action space for motion control. S A($) x + [0, represents the environmental transitioning probability from one state to another 8(+1, given an action a₁ R: S A x S + IR
the state-action reward, and the fitness function /' defined as the episodic return > Fr (8t, az) based on rewards. As defined above, D represents the morphology design space, and our goal is to find some co-design policy π D that can maximize the environmental fitness F.
METHOD
The co-design process consists of two sequential stages. (1) In the Design Stage, an agent begins with initial morphology, Go. and iteratively refines through series of morphology transforming actions via design policy π until it achieves the final design Gdone. In the subsequent (2) Control Stage, the agent interacts with the environment with its corresponding control policy C
BodyGen addresses three key challenges that hinder co-design efficiency: (I) Message Transmission Decay which occurs when multi-hop communication fails to effectively propagate information to distant limbs (Kurin et al., 2020). BodyGen leverages self-attention for both auto regressssyy body building and centralized body control using transformers. (2) Ineffective Morphology Representa- tion (Yuan al., 2021; Hong et al., 2021). BodyGen employs simple yet effective topology position encoding mechanism better to align similar morphologies for knowledge sharing between them. (3) Unbalanced Reward Signals. BodyGen utilizes temporal credit assignment mechanism to ensure balanced reward signals between different co-design stages.
4.1 ATTENTION-BALE CO-DESIGN NETWORK
BodyGen divides the Design Stage into two sub-stages: Topology Design Stage and Attribute Design Stage, which transforms the topology (Vo, Eg) and the corresponding attributes (A AB) of the agent's 's morphology, respectively. Consequently, the design policy x is also divided into two sub-policies .! 2opo attrr) for according action control.
4

---

Published as a conference paper at ICLR 2025
During the Topology Design Stage, the agent modify the topology through three basic actions:
(1) Additi ion: add a new child limb Unew to U, along with a new joint Enew = (v, Unew) connecting (Up. 2) if is a leaf node.
them. (2) Deletion: delete the limb v and the joint to its parent conditioned on the current (3) NoChange: take no changes for node n. The agent's policy zoopo is
topology (V, F)) of timestep i, denoting the product of action distributions pepo from all limbs !
a topo π topo (a oopo (g) - II
topo (a. pp P)
Pus
(4.1)
DEV
where Pv represents the topology position of node v.
In the Attribute Design Stage, the agent further generates limb and joint attributes based on the given topology Goone (Vdone: Edone). The agent's s attribution policy gattr can be formulated as:
a attr ~ π attr (a attr (g) _
II NEV
attr attr (a (pe,
(4.2)
Finally, in the Control Stage, the agent uses the morphology generated in the Design Stage to interact with the environment using the control policy rC.
a andl (a (a Gdone)
II Te (at Pos Gamn))
(4.3)
where s = {8v] denotes the sensor states of every limp, including forces, positions, velocities, etc. We use a. to represent the torque of the joint connecting node v with its parent Up-
During the co-design process, we aim for the policy network to accommodate evolving morphologies in way that offers two key advantages: (1) a single agent can maintain unified control even as the robot's body grows, preserving consistency across different designs, and (2) direct point-to-point communication between joints allows for richer information exchange, enabling more coordinated actions throughout the entire system. Inspired by the centralized signal processing of mammals in real-world nature, we propose the Morphology Self-Atten Tion architecture (MoSAT) for efficient, centralized message processing. Figure (a) provides an overview of MoSAT.
Latent Projection. We encode information from each limb's sensor to enhance network processing capabilities and map it into a latent space as message tokens. Specifically, limb sensor states 8, are first processed through parameter-shared linear mapping layer (.):
m Φ (s) + Epos (V, E) E TR. Lxd
E RLXD
(4.4)
where d is the input state dimension and D the hidden dimension. We employ our proposed TopoPE for morphology representation, which will be further discussed later in Section 4.2. The position encodings e, are added message tokens m,, to get position-embeddee message tokens.
Centralized Processing, As illustrated in Figure 2. we aim for efficient message interaction. Body- Gen utilizes the scaled dot-product self-attention Attention(.) for point-to-point centralized pro- cessing. Specifically, each message m use qu; to query the key of another message kin, weighting its value un
QKT Jax V
Attention(m) (m) SoftMax
where Q = mWq: K mWK, mWv,
(4.5)
where Wo. WK. Wy E DDD are learnable matrices. For MoSAT block design, we adopt Pre-LN (Xiong et al., 2020) for layer normalization and add residual connections (He et al., 2016; Dosovitskiy al., 2020).
Forwarding. In the end, we need to output actions for each actuator. We decode the attended messages using a linear projector (a(+) to generate the action logits for each actuator:
(oxt)) x (als) (Va(: Φa (mx), E), Continuous Action Space.
Discrete Action Space
(4.6)
'We use the limb-level action distribution. where each limb corresponds to its own action distribution, and the entire agent's 's action distribution is composed of all limbs" distributions. This effectively resolves the incompatibility of state and action spaces across the changeable topological morphologies.
5

---

Published as a conference paper at ICLR 2025
0000
0000
0
000
0
00:0
0D:0
8000
6330
0000
(a) Morphology Changes
(b) Traversal PE
(c) TopoPE (Ours)
Figure 4: The motivation of our proposed topology-aware position encoding TopoPE. (a) During the co-design procedure, the agent's morphology keeps changing. (b) A typical traversal-based PE in previous works resulted in inconsistency across mythologies. (c) TopoPE can better adapt similar morphology structures using a reasonably alignable manner.
where N is the stacked block number of the attention layer. The above process has equipped MoSAT with the capability to handle various morphologies. As shown in Figure (b), to maximize training efficiency, we further offer MoSAT the ability process multiple morphologies in batch mode. We provide more implementation details in Appendix A.3.3.
4.2 TOPOLOGY-AWARD POSITION ENCODING FOR MORPHOLOGY REPRESENTATION
The vanilla attention operation treats each token equally, neglecting morphology information. How- ever, it is crucial to inject positional information during embodiment co-design, for: (1) Similar information from different morphology positions has varying meanings and message source local- ization is significant; (2) Similar morphology structures may share similar local control policies, and positional information facilitates knowledge alignment and sharing across different agents. To better capture the differences between morphological structures and share structural knowledge among similar morphologies, we propose Topology Position Encoding (TopoPE), a topology-aware position encoding mechanism to handle the above two issues efficiently.
As demonstrated in Figure 4, for traversal-based limb indexing methods (Hong et al., 2021; Gupta et al., 2021a; Xiong et al., 2023) slight morphological changes cause global indexing offsets. To mitigate the effect of offsets due to morphological changes, TopoPE uses a hash-map H(-)) for position encoding, which maps the path between the root limb Uroot and the current limb U; unique embedding e..:
Cv² = H([u: + vroot))
(4.7)
where (c: Mroot] [(v p(u)),, (1:))
Uroot)];
where p"( (v. P(U)) is denoted by the integer k allowing the path index to be represented sequence of integers.
(v) is the n-th ancestor v. Practically, v is the k-th child of its parent p(v), the edge
During the Topology Design Stage, BodyGen generates the robot's topology autoregressively (Fig- ure 2). The topology created at each step is passed to MoSAT in the following step, where newly added limbs are automatically registered and assigned their Topology Position Embedding. Meanwhile, dur- ing the Autrbute Design Stage and the Control Stage, this final topology remains fixed, Experiments demonstrate that TopoPE effectively adapts growing morphologies, facilitating knowledge alignment and sharing across agents, which leads to better performance.
4.3 CO-DESIGN OPTIMIZATION WITH TEMPORAL CREDIT ASSIGNMENT
To achieve efficient reward-driven co-design, Body- Gen leverages actor-critic paradigm based on reinforcement learning, which trains a value func- tion Va(sz) and policy function Tp(a, (s) and up- dates them using collected trajectories. We employ the Proximal Policy Optimization (PPO) (Schulman et al., 2017) to optimize the policy Ty in the actor- critic framework. PPO uses the advantage function
MoSAT
MoSAT
G
Policy Network
(b) Value Network
Figure 5: BodyGen leverages actor-critic
paradigm for policy optimization.

---

Published as a conference paper at ICLR 2025
Ae(a,, st) to define how better action ag is for current state st, and optimizes the following surrogate objective function as:
Coolicy
min
{ To(ar)s)) 9t) Totiaa (ap
A., clip Te (ae((s)) Toaa (a,)st) e, (1]))))11s1++)AA
c)
(4.8)
In the co-design process, vanilla PPO exhibits limited performance. Only the Control Stage directly receives environmental rewards, while theoretically, a body-modifylle action in the Design Stage influences all future timesteps, whereas a motion-control action in the Control Stage has a diminishing impact over time. To address this, we decouple the MDPs for body and policy optimization, linking them through a modified Generalized Advantage Estimation (GAE) (Schulman al., 2015b) for improved temporal credit assignment:
A, = 6, + 7dA+1 : (1-T, Ct),
for Control Stage
U. Vo(s)),
for Design Stage
(4.9)
where = r2 + (000941)-(11777
Vo(st)
- U441 (v)).
where is the discounting factor, À is the exponentially weighted for GAE and Tr, C, are two environment flags denoting environment termination and truncation, respectively. This decoupling enables us to apply distinct optimization algorithms to each stage, potentially improving overall performance. The value loss function Lealue is defined as:
Cualue (Vo(s:) R where R. =g Vo(ss) Ac),
(4.10)
where sg[+] stands for the stop-gradient operator.
During the transition from the Design Stage to the Control Stage, we shift from a GPT-style (Radford al., 2019) approach to BERT-style (Devlin et al., 2018) framework. Specifically, the token output of each limb is used to generate the action policy for its corresponding actuator (Equation 4.6), as illustrated in Figure 5(a). Meanwhile, the token output the root limb is used for value prediction of the entire body at timestep t (Figure 5(b)). To prevent conflicts in gradient descent (Yu et al., 2020; Liu et al., 2021) arising from different credit assignment strategies, each stage in the co-design process is equipped with separate value network.
5 EXPERIMENTAL EVALUATIONS
Our experiments aim to validate our primary hypothesis: that efficient message and reward delivery can effectively overcome bottlenecks the co-design process, leading embodied agents that can better adapt to the environment. Additional visualization results are presented in Appendix A.8. Visit our project website for more visualization results: https: http///gnessssiiiiggggssss github. io.
We conduct comprehensive evaluation of BodyGen with baselines in ten chal- lenging co-design environments (CRAWLER. TERRAINCROSSER, CHEETAH, SWIMMER, GLIDER- REGULAR, GLIDER- MEDIUM, GLIDER-HARD. WALKER-REGULAR WALKER -EDEUM and WALKER "ALKER-HARDD on MuJoCo (Todorov et al., 2012). These environments encompass diverse physi- cal world types (2D, 3D), environment tasks, search space complexities, ground terrains, and initial designs to provide a multilevel evaluation. See Appendix A.1 for detailed descriptions.
Environments.
5.1 COMPARISON WITH BASELINES
We compare BodyGen with the following baselines to highlight BodyGen's performance: 1) Evolution Based Algorithms: NGE (Wang et al., 2018b) maintains population of agents with different morphologies for random mutation and only preserves top-performing agents' children for further optimization. 2) RL Based Algorithms: Transform2Act (Yuan et al., 2021) propose to optimize a robot's morphology and control concurrently through reinforcement learning and achieve co- optimization. It utilizes graph neural networks (GNNs) and joint-specific MLPs (JSMLP) to foster knowledge sharing and specification. 3) Universal Control Algorithms: UMC-Message (Wang et al., 201 8a; Huang al., 2020) leverages localized message transition mechanism for information
7

---

Published as a conference paper at ICLR 2025
Crawler
Terrain Crosser
Cheetah
Swimmer
17/00
10030
Reward
Remoard
Reward
7230
2007
.000
2530
075
1055
30Ml Step
200
-CM
FPM
1056
170
1054
4002
200
080
1004
2054
1002
4.00
300
Step
Step
Step
Glider-Regular
Glider-Medium
Glider-Hard
Walker- Reguulr
11904
12500
1224
12200
Reward
Reward
Reward
Roward
2000
:008
2009
1.0
3.04
4000
Stop
Stop
Stop
Step
"Yalker-Mediim
Walker-Hard
BodyGen (Ours)
Transform2Act
12500
18103
UMC-Message
NGE
Roward
2306
5000
Ours w/o MoSAT
2000
2500
Ours w'o Enhanced-TCA
034
1054
1011
3.54
0.54
254
12M1
35V
12V
Step
Step
Figure 6: Performance of BodyGen, NGE, Transform2Act, UMC-Message*, BodyGen w/o MoSAT, and BodyGen w/o Enhanced-TCA on ten co-design environments, with error regions to indicate Standard Error over four random seeds.
exchange within the body, which is a typical method for universal morphology control. To make suitable for embodiment co-design, we equip it with policy network and our enhanced temporal credit assignment using reinforcement learning and denote as UMC-Message*. The implementation details and full syper-parameter of BodyGen and all baselines are provided in Appendix A.3 and Appendix A.4.
As shown in Figure 6, BodyGen achieves the highest task performance in all ten environments, with faster convergence speeds than baselines. Unlike the Universal Morphology Control (UMC) task, which focuses limited specific morphologies (Wang et al., 2018a; Huang et al., 2020), embodiment co-design deals with various changeable, morphology-agnosti robots. Consequently, UMC-Message' fails to converge within limited time for complex tasks such GLIDER-HARD, and HARD, due to its insufficient knowledge alignment mechanism for complicated, changable morphologies (e.g. JSMLP in Transform2Act and TopoPE in BodyGen).
Compared to evolutionary algorithms like NGE, we also find that RL-based methods demonstrate significant performance advantage due great sampling efficiency improvement within the same number of environmental interactions, supported by Yuan et al. (2021). By overcoming the bottle- necks in co-design, our approach goes even further: it achieves an average 60.03 % performance improvement over the strongest baseline in all the ten tasks.
5.2 ABLATION STUDIES
As mentioned in Section 4, our approach addresses inefficiencies in message and reward delivery, which includes the intra-agent level, inter-agent level, and agent-env level. better support hypothesis and understand the importance of our key corresponding components (MoSAT, TopoPE, (iraaneddTCAA) we designed four variants of our approach:
(i) Ours w/o MoSAT, which removes the MoSAT structure to remove our attention-based centralized information processing across different limbs; (ii) Ours w/o Enhanced-TCA, which removes our temporal credit assignment mechanism and employs original PPO for optimization; (iii) Ours w/o TopoPE, which removes TopoPE from our methods. For a more comprehensive comparison, we also introduced another position encoding method from recent UMC methods, (iv) Ours w/
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different position encoding choices for morphology representation. The reported values are Mean + Standard Error over four random seeds.
Methods
CRAWLER
TERSAINCROSSER
CHEETAH
SWIMMER
GLIDER REGULAR
TopoPE (ours)
10381.96 + 353.97
5056.01 + 703.57
11611.52 + 522.66
1305.17 + 15.25
11082.29 99.21
w/ Traversal
8582.24 + 987.44
4339.60 +: 260660
10581.62 + 846.69
1292.05 16.71
9801.31 748.13
w/o TopoPE
7490.83 + 267.70
1122.29 + 659.38
7451,37 = 2275.37
1371.20 30.74
10137.83 713.60
Methods
GLIDER-MEDIUM
GLIDER-HARD
WALKER-RECULAR
WALKER-MEDIUM
WALKER-HARD
TopoPE (ours)
11996.82 +: 595.51
10798.06 + 298.39
12062.49 1: 513.07
12962.08 1: 537.34
11982.07 520.78
w/ Traversal
10758.70 401.90
9106.77 679.59
1080.94
10972.13 584.0a
11255.89 121.04
w/o TopoPE
4099.99 +: 2057.92
109.48 + 10.03
10149.67 +: 255.99
6730.01 705.06
6529.87 + 1863.59
Traversal PE, where TopoPE is replaced with traversal-based position embedding (Hong et al., 2021; Gupta et al., 2021a). Figure 6 presents the ablation studies for TopoPE and Enhanced-TCA while Table 1 highlights the differences for different positional embedding choices. Additional detailed experimental results are available in the Appendix (Table 11, Table 12).
(1) Intra-agent level: The MoSAT module provides centralized information processing. Removing this module results in significant performance degradation. Transform2Act adds an MLP to each limb, enhancing local message processing and model performance, but it increases the model size to 19.64M, which grows linearly with the complexity of the morphology. In contrast, BodyGen is more lightweight, with each model only with 1.43M parameters. We provide model parameters of BodyGen and baselines in Table 2.
TopoPE facilitates morphological knowledge sharing among agents, aiding in adjusting knowledge for similar morphologies and reducing redundant learning costs. Compared to "Traversal PE" and "w/o TopoPE", TopoPE enhances agent performance and stabilizes learning.
(2) Inter-agent level:
(3) ggent-environment level: Our proposed temporal credit assignment ensures that agent receives balanced reward signals during both morphology design and control phases, markedly improving final performance across all the environments for embodiment.
Table 2: Model parameters of BodyGen and baselines. Note: For NGE, the total number of models required is calculated as 20 + 20 x 0.15 x 125 = 395 (population_siee sisooonssszeeeeee + population_se size x elimination_ran _ate generations). The total parameters are derived with population only.
Models
Agent Parameters
Population Size
Total Parameters
BodyGen (Ours)
1.43 M
1.43 M
Transform2Act
19.64 M
19.64 M
UMCC-eesagee
0.27 M
0.27 M
NGE
0.27 M
20
5.4 M
CONCLUSIONS AND LIMITATIONS
This work proposes BodyGen, end-to-end reinforcement learning framework for efficient embodi- ment co-design. Our approach delivers efficient messages and rewards through zero-decay message processing, effective morphological knowledge sharing, and balanced temporal credit assignment. Experiments demonstrate that BodyGen surpasses previous convergence speed and final performance methods while being efficient, lightweight, and scalable.
Limitations and Future Work. We acknowledge at least two limitations. Firstly, our approach remains focused on simulation environments, and further efforts are needed to transfer learned strategies to real physical systems. Secondly, our reward-driven reinforcement learning method focuses on improving control effects. Yet, cannot simulate the rich perception and execution capabilities of real biological intelligent systems. In future research, we expect embodied intelligence to evolve perception and execution components akin to biological evolutionary principles, realizing more efficient tasks for embodied intelligence.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by the Natural Science Foundation of China under Grant No. 62222606 and the Ant Group Security and Risk Management Fund.
REFERENCES
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojeiech Matusik. Evolution gym:
A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34:2201-22111 2021.
Brain for AI Fandom. Central nervous system, 2024. URL Accessed: 2024-05-21.
fandom.
com/ www//wiki//// Cemm/wiki/Centralae http://eeeeaaaannevvs__sssss
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01555 005.00000000, 2016.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8800008805 2018.
Heng Dong, Tonghan Wang, Jiayuan Liu, and Chongjie Zhang. Low-rank modular reinforcement learning via muscle synergy. Advances in Neural Information Processing Systems, 35:19861- 19873, 2022.
Heng Dong, Junyu Zhang. Tonghan Wang, and Chongjie Zhang. Symmetry-aware robot design with structured subgroups. In International Conference on Machine Learning, pp. 8334 8335-8355, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11999 2020.
http://wwww.iitnni........... Encyclopaedia Britannica. Diffuse nervous system, 2024. URL https://wwww bri Soritannico ca. com/ www.eenceenervousssssseeeee
systemss, Accessed: 2024-05-
21.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers
with transformers. In International Conference on Learning Representations, 2021a.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021b.
David Ha. Reinforcement learning for improving agent design. Artificial Life, 25(4):352-365, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim. Structure-aware transformer policy for inhomo- geneous multi-task reinforcement learning. International Conference on Learning Representa- tions, 2021.
Suning Huang, Boyuan Chen, Huazhe Xu, and Vincent Sitzmann. Dittogym: Learning to control soft shape-shifting robots. arXiv preprint arXiv:2401.1323.1 2024.
Wenlong Huang. Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455 4464, 2020.
10

---

Published as a conference paper at ICLR 2025
Euan Judd, Gabor Soter, Jonathan Rossiter, and Helmut Hauser. Sensing through the body non- contact object localisation using morphological computation. In Proceedings of the IEEE Interna- tional Conference on Soft Robotics, pp. 558-563,11 2019.
Sam Kriegman, Douglas Blackiston, Michael Levin, and Josh Bongard. A scalable pipeline for designing reconfigurable organisms. Proceedings of the National Academy of Sciences, 117(4): 1853-1859, 2020.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2020.
Boyu Li, Haoran Li. Yuanheng Zhu, and Dongbin Zhao. Mat: Morphological adaptive transformer for universal morphology policy learning. IEEE Transactions on Cognitive and Developmental Systems, (6(4):61111162) 2024. doi: 10. 1019/CCDS
Timothy Lillicraap Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509 150..0297, 2015.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. volume 34, pp. 18878-18890, 2021.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01848, 2018
Kohei Nakajima, Helmut Hauser, Tao Li, and Rolf Pfeifer. Exploiting the dynamics of soft materials for machine learning. Soft robotics, 5(3):339-347) 2018.
Xinlei Pan, Animesh Garg, Animashree Anandkumar, and Yuke Zhu. Emergent hand morphology and control from optimizing robust grasps of diverse objects. In Proceedings the IEEE International Conference on Robotics and Automation, pp. 7540-7547, 2021.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei Efros. Learning control self-assembling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019.
Rolf Pfeifer and Christian Scheier. Understanding intelligence. 2001.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):166) 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 201
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 15X6v 02438, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:7007.06347, 2017.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv: arXiv:/803.00 1803.02/55, 2018.
Karl Sims. Evolving virtual creatures. In Proceedings of the 21 annual conference on Computer graphics and interactive techniques, pp. 15-22, 1994.

---

Published as a conference paper at ICLR 2025
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033, 2012.
Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning, pp- 21677-21691. PMLR, 2022.
Patricia A Vargas, Ezequiel Di Paolo, Inman Harvey, and Phil Husbands. The horizons of evolationary robotics. 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. arXiv preprint 1rXiv:1912.12333 2019.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. International conference on learning representations, 2018a.
Tingwu Wang. Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. In International Conference Learning Representations, 2018b.
Yuxing Wang, Shuang Wu, Tiantian Zhang, Yongzhe Chang, Haobo Fu, Qiang Fu, and Xueqian Wang. Preco: Enhancing generalization in co-design of modular soft robots via brain-body pre-training. In Conference on Robot Learning. pp. 478 498, 2023.
Julian Whitman, Matthew Travers, and Howie Choset. Learning modular robot control policies.
IEEE Transactions Robotics, 2023.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. 2020.
Zheng Xiong, Jacob Beck, and Shimon Whiteson. Universal morphology control via contextual modulation. In International Conference on Machine Learning, pp. 38286-38300, 2023.
Jie Xu, Andrew Spielberg, Allan Zhao, Daniela Rus, and Wojciech Matusik. Multi-objective graph heuristic search for terrestrial robot design. In 2021 IEEE international conference on robotics and automation, pp. 9863-9869, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. volume 33, pp. 5824-5836, 2020.
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris M Kitani. Transform2act: granform--nd-commen policy for efficient agent design. In International Conference on Learning Representations, 2021.
Learning
Allan Zhao, Jie Xu, Mina Konaković-Lukoote Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Robogrammar: graph grammar for terrain-optimized robot design. ACM Transactions Graphics, 39(6):1116, 2020.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
The supplementary material provides additional results, discussions, and implementation details.
Our code is available our supplementary material for reproduction and further study. Visit our website for videos and more additional visualizations.
A.1 ENVIRONMENT AND TASK DETAILS
Crawler
TersainCrosser
Cheetah
Swimmer
Glider
Walker
Figure 7: Randomly generated agents in six different environments for visualization. Purple ground indicates agents in a 3D physical world, Green ground represents agents in the xy-plane physical world, Blue ground denotes agents in the xz-plane physical world, and Brown ground denotes physical world with variable terrain.
Initial Type-1
Initial Type-2
Initial Type-3
Initial Type-4
Figure 8: Visualization of four initial designs in the environments. Type-/ consists of structure with four limbs. Type-2 and Type-3 each includes two limbs connected by a joint, located the xy-plane and xz-plane respectively. Type-4 comprises three limbs connected by two joints. Note that BodyGen can support almost arbitrary initial designs and is not limited to specified types.
This section provides additional descriptions of the environments and tasks used in our experiments. Figure 7 displays randomly generated agents in six different environments. The first four envi- ronments: CRAWLER, TERRAINCROSSER, CHEETAH, and SWIMMER are derived from previous work (Yuan et al., 2021) to ensure fair comparison. We have also introduced two additional envi- ronments, GL.DER and WALKER, to broaden the testing scope and provide more comprehensive algorithm evaluation.
Each agent consists of multiple limbs connected by joints, each equipped with a motor for controlling movement. Sensors within the limbs monitor positional coordinates, velocity, and angular velocity, Each limb's s attributes include limb length and limb size. Each joint's attributes cover rotation range and maximum motor torque. Each episode starts with simple initial design, demonstrated in Figure 8. The agent evolves to its final morphology through a series of topological and attribute modifications. Meanwhile, the control policy is required to optimize concurrently.
13

---

Published as a conference paper at ICLR 2025
Crawler The agent inhabits 3D environment with flat ground O. The initial design is the Type-1 in Figure 8. Each limb can have up to two child limbs, except for the root limb. The height and 3D world velocity of the root limb are also included in the environment state. The reward function is defined as:
Pi+1 - Pi Tt
Σ
|
(A.1)
At
WVV
where w 0.0001 is the weighting factor for the control penalty term, J is the total number of limbs, and At = 0.04.
PeerainCrosser The agent evolves in terrain-variable variable environment, where the terrain features varying height differences. The maximum height difference the terrain is 2moz = 0.5. The agent must navigate these gaps to move forward. The initial design is the Type-3 in Figure 8. The terrain is generated from a single-channel image, with different values representing different height rates. Each limb of the agent can have up to three child limbs. For the root limb, its height, 2D world velocity, and a variable encoding the terrain information are included in the environment state. The reward function is defined as:
PF+1 P
(A.2)
Al
where At 0.008, and the episode is terminated when the root limb height below 1.0.
Cheetah The agent in this environment evolves with flat ground at = 0. The initial design is the Type-3 in Figure Each limb of the agent can have up three child limbs. The height and 2D world velocity of the root limb are added to the environment state. The reward function is defined in Equation (A.2). The episode is terminated when the root height is below 0.7.
Swimmer
The swimmer is designed to cover snake-like creatures in the water. The agent evolves in water with a ris- = 0.1 viscosity for water simulation. The initial design is the Type-2 in Figure Fach limb supports up to three child joints. The root limb's 2D world velocity is incorporated into the environment state. The reward function is the same as TerrainCrosser Equation (A.2).
Glider The agent in this environment evolves flat ground. The initial design is the Type-4, as shown in Figure 8. In Glider, the agent's search depth is limited to three times that of the initial design, encouraging full exploration of a relatively shallow search space. We also provide three different task levels: regular, medium, and hard, where each limb of the agent can have up to one, two, three child limbs. The reward function is defined in Equation (A.2).
Walker The agent evolves on flat ground. The starting design is Type-4 in Figure 8. The search depth for the agent is capped at four times the initial design promote thorough exploration within comparatively shallow search space. Similarly, three levels of task difficulty are offered, which have the same meaning as described in Glider. The reward function is specified in Equation (A.2).
Note that the reward functions are kept simple and consistent in all environments. Unlike com- practices in OpenAl-Gym (Brockman et al., 2016), we do not provide any additional reward priors galve bonus) to facilitate learning, which presents higher requirements to the algorithm robustness.
A.2 MOTIVATIONS
This section will detail the motivations behind designing MoSAT and TopoPE for embodiment co-design, aiming provide further insights.
A.2.1 CENTRALIZED MESSAGE PROCESSING AND MOSAT
As demonstrated in Figure 9, GNN-like neural systems commonly found in simple organisms such as planarians, where sensory information is connected through neural networks for distributed and localized processing. In contrast, advanced creatures such as humans utilize centralized signal processing approach, where signals from various body parts are centrally processed in the brain, leveraging scalability advantages similar to the self-attention mechanism within transformers.
Figure 10 further illustrates the different message delivery mechanisms between GNN and Trans- former. GNN uses aggregation and broadcasting for message transmission, resulting in progressive
14

---

Published as a conference paper at ICLR 2025
Bran
Spinal eard
Brochial
plexus
Neural System of Plunarian
Graph Neural Network
Neural System of Human
Trunsformer Neural Network
(a) Localized Message Processing
(b) Centralized Message Processing
Figure 9: Comparative overview of natural and artificial neural processing systems. (a) Localized message processing in planarians and GNNs. (b) Centralized message processing in human brains and Transformers. Relevant images sourced from Encyclopaedia Britannica (2024); Brain for Al Fandom (2024).
Step
Step [2]
Step
Step
Received
(a) Message via Aggregation and Broadcasting
(b) Messaging via Self-Attention
Figure 10: Comparison of different message delivery mechanisms between GNN and Transformer.
information reduction. As demonstrated in Figure 10 (a), the dog-like robot needs to adjust its posture throughout its motion. The GNN's localized message processing approach requires signals from distant locations to propagate multiple times before reaching the target actuator. contrast, Trans- formers can provide faster message transfer and interaction, by employing self-attention to facilitate direct point-to-point and po-ml-to-mullioint message delivery. Inspired by this, we propose MoSAT in Section 4.1. MoSAT first maps sensor information to the latent space and leverage self-attention for signal interactions for centralized decision-makingg
Meanwhile, in GNNs, the message propagation mechanism allows for an implicit representation of morphology. However, while transformers leverage self-attention for direct message delivery, they do not offer an asymmetric information propagation mechanism to differentiate positions between different body parts.
A.2.2 MORPHOLOGG POSITION EMBEDDING AND TOPOPE
Position encoding has proven effective in location representation within the natural language process- ing field (Vaswani et al., 2017; Shaw al., 2018; Raffel al., 2020; Wang et al., 2019).
Effectively representing the robot's morphology is crucial for co-designing morphology and control policies. our work, we propose the Topology Position Embedding (TopoPE) encode the morphology in way compatible with Transformer-basee architectures. TopoPE assigns a unique embedding to each limb based on its topological position within the robot's morphology tree. Specif- ically, the embedding index for limb is derived from the path from the root node to the limb, capturing the structural relationships within the morphology.
In previous works (Trabucco al., 2022; Gupta al., 2021a), morphology encodings often rely on traversal sequences like depth-first search (DFS) or manual naming (Trabucco et al., 2022; Li al., 2024) conventions based "full model" of the robot. When limbs removed generate variants, the names of the remaining limbs remain unchanged, facilitating consistent encoding.
However, in setting, there is no predefined "full model," and the robot's morphology is dynamically generated during co-design. Manually naming limbs is impractical in this context. Our TopoPE addresses this challenge using topology indexing mechanism, which uses the path the root as
15

---

Published as a conference paper at ICLR 2025
the embedding index. This method naturally extends to dynamically changing morphologies and ensures that similar substructures share similar embeddings, promoting generalization across different morphologies.
Moreover, unlike learnable position embeddings that are specific to particular morphologies, our approach can be extended using non-learnable embeddings, such sinusoidal embeddings (Vaswani et al., 2017), which offer better extrapolation to unseen morphologies and eliminate the need for training the embeddings.
To demonstrate the effectiveness of TopoPE, we conducted ablation studies comparing models with and without TopoPE. As shown in Table 1 and Figure 11, incorporating TopoPE: significantly improves performance across various tasks. This indicates that TopoPE provides more informative and stable encoding of the morphology, facilitating better learning of control policies.
In contrast to other morphology-awee positional encodings, our TopoPE is specifically designed to handle dynamic and diverse morphologies without relying on fixed full model or manual limb naming. Additionally, our approach aligns well with the Transformer architecture, allowing standard attention mechanisms to capture interactions between different limbs based on their topological relationships.
A.3 IMPLEMENTATION DETAILS
A.3.1 TRAINING DETAILS
In line with standard reinforcement learning practices, we employed distributed trajectory sampling across multiple CPU threads to accelerate training. Each model is trained using four random seeds on a system equipped with 112 Intel@) Xeon@ Platinum 8280 cores and six Nvidia RTX 3090 GPUs. Our main code framework is based on Python 3.9.18 and Py PyTorch 2.0. For all the environments used in our work, it takes approximately only 30 hours to train a model with 20 CPU cores and a single NVIDIA RTX 3090 GPU our server.
A.3.2 HYPERPARAMETERS
For BodyGen, we ran grid search over MoSAT layer normalization (w/o-IN, Pre-TN, Post-IN]. Policy network learning rate E (5e 5, le 4,3e - 4).
Value network learning rate E {le- 4,3e 4), and MoSAT hidden dimension E {32, 64, 128, 256). We did not search further for the environmental settings, optimizer configurations, PPO-related hyperparameters, or the training batch and minibatch sizes. Instead, we strictly maintained consistency with previous works (Wang et al., 2018b; Yuan et al., 2021; Kurin et al., 2020) to ensure a fair comparison. With further hyperparameter tuming, our algorithm could achieve higher performance levels. Table displays the hyperparameters BodyGen adopted across all experiments.
For Transform2Act, we followed previous work (Yuan et al., 2021) and its official released code repository 2, and used GraphConv the GNN layer type, policy GNN size (64, 64, 64), policy learning rate 5e 5, value GNN size (64, 64, 64), value learning rate 3e 4, JSMLP activation function Tanh, JSMLP size (128, 128, 128) for the policy, MLP size (512, 256) for the value function, which were the best values they picked using grid searches.
To make UMC-Message suitable for embodiment co-design, we equip them with policy network and employ our temporal credit assignment via reinforcement learning. The network parameters and training settings are consistent with those used in BodyGen and Transform2Act ensure fair comparison. It adopted GNN layer type of GraphConv, policy GNN size (64, 64, 64), policy MLP size (128, 128), policy learning rate 5e 5, value GNN size (64, 64, 64), value MLP size (512, 256). value learning rate 3e - 4. We followed previous work (Huang et al., 2020) and also referred to the publicly released code for implementation.
For NGE, we follow previous works (Wang et al., 2018b; Yuan et al., 2021) according to the public release code and used a number generations 125, agent population size 20, elimination rate 0. 15,
thub. wtw.t/th..ccccoooo////////ao 1aar111//mmmdu/////// 'https: // thub. http://hh......ooo//o/ooo//ga/aa
in
16

---

Published as a conference paper at ICLR 2025
GNN layer type GraphConv, MLP activation Tanh, policy GNN size (64, 64, 64), policy MLP size (128, 128). value GNN size (64, 64, 64), value MLP size (512, 256), policy learning rate 5e 5. and value learning rate 3e 4, which were the best searched values described by previous work.
Table 3: Hyperparameters of BodyGen adopted in all the experiments
Hyperparameter
Value
Number of Topology Design Niopp
5
Number of Attribute Design Nattr
1
MoSAT Layer Normalization
Pre-LN
MoSAT Activation Function
SiLu
MoSAT FNN Scaling Ratio r
4
MoSAT Block Number (Policy Network)
3
MoSAT Block Number (Value Network)
3
MoSAT Hidden Dimension (Policy Network)
64
MoSAT Hidden Dimension (Value Network)
64
Optimizer
Adam
Policy Learning Rate
5e-5
Value Learning Rate
3e-4
Clip Gradient Norm
40.0
PPO Clip
0.2
PPO Batch Size
50000
PPO Minibatch Size
2048
PPO Iterations Per Batch
10
Training Epochs
1000
Discount factor 7
0.995
GAE
0,95
A.3.3 THE BATCH MODE FOR MOSAT
To maximize training efficiency, further offer MOSAT the ability process multiple morphologies in a batch mode. For a batch of state inputs {s, ] n. we first pad them to equal length [s,]n e BB where T.m is the max limb number of morphologies within this batch, and generate padding matrix P E BxL,, where P.j = for j S L, and P.j 0 for j >> 14- To keep the messaging logic exactly equivalent the regular mode, we can eliminate the influence of padding by modifying the attention operation with an attention mask θ E R RBxx Lm x Lmm
QKT Vd
Attention([m/ IB) = SoftMax
+ θ)V.
(A.3)
where θizk log( Pss. + +). Finally, we remove the batch padding and re-allocate actions to joints of different agents via: {a} = [a]B @ P, where @ represents the bool-selection operation according to the padding matrix P.
17

---

Published as a conference paper at ICLR 2025
A.4 ALGORITHM DETAILS
Algorithm illustrates the overall training process of BodyGen, which is based on PPO for efficient reinforcement learning. We highlight three key components: the interaction process, temporal credit assignment based on GAE, and the main loop for iterative optimization.
Algorithm 1: Synchronous Learning Algorithm for BodyGen
Input: Replay Buffer B. Batch R, Optimizer optimizer
Initialize Policy networks: my {re topp #g zettr, wetr(f; J. Value networks: V₂ (Vgr Vater, Vigrl)
< 2, B < Ø, Discount factor ", GAE Exponential Weight
Function INTERACT(Policy:
Replay Buffer: B):
while B reaching max buffer size do
Go < initial design
<- topo
+ Topology design stage
for = 0, 0,1, ..., Noopo do
Sample topology actions from all limps Ge+1 <- apply a, an to modify the topology (V,, E,) of current design G.
a topo (a, (a.
topo ri = 0: St S; store fre, a. a St 0} into B -Uddtte Buffer B with transition
10
+ attr
Attribute design stage
for Niopo ...., Noopo Naltr
11
b Sample attrihute actions from all limps G t apply at modify the attribute (Ar, Af) current design G. T1 0: St = S; store (r., Ø, aetrr Gr S. 0, 0} into B Update Buffer B with transition
aethr ~ x n a
13
15
end
16
① + ctrl EnvvReccc((O)
Control stage 8, {su.c) denotes the sensor states from all limps
18
for = Ntopo + Nottr ,..., 1 do
aptri netrl (a sc, Gdone)
Tt, 81+1. Tú, C + Env. Steplaep(
D Tr, C2 denotes termination and trunction
21
S, = S: store 8r, aetri Gr, St, C, into B b Update Buffer B with transition
end
13
end
24
25
F Function ENHANCEDGAAE(ale Function: Vo, Replay Buffer: B):
for = T 1,..., do
27
U. ===,, + (1 T, Cc)
Calculate return
28
6: = rr +7 1)) TT)) Vo(ss)
D Calculate the TD-error term
if S ctrl then
A = 6z + +xAr+1 -1 V
b Calculate advantage for the control stage
else
A, = U, - Vo(st)
Calculate advantage for the design stage
end
RG Vols,) d
Calculate the target value Append A, and fi the corresponding transition item in B.
store (Ac, (ie) into B
36
end
end
38
Function MAIN0:
while reaching iterations do
40
The <- (INERA( ng. B) Tha.join))
We use multiple CPU threads sampling b Gather trajectories collected from threads Perform temporal credit assignment for co-design
41
ENHANCEDGAA(VO.. B)
while reaching max epochs do
44
Update B B
Sample random batch B from Buffer
Calculate PPO loss Eypo Cpolicy Lealue
According Equation (4.8) and (4.10) b Gradient descent update ma and V,
46
optimizer <- Gradient from Eppo
end
48
end
end
18

---

Published as a conference paper at ICLR 2025
A.5
ADDITIONAL RESULTS
A.5.1
QUANTITATIVE RESULTS
Table 4: Comparison of BodyGen, its ablation variants, and baseline methods.
Methods
CRAWLER
TURRAINCROSSER
CHEETAI
SWIMMER
GLIDUR-REGULAR
BodyGen (Ours)
10381.96 383.97
703.57
11611.52 $22.86
1305.17
11082.29
w/o MoSAT
818.92 = 57.78
-07.30
662.88
74.88
476.26
19.95
447,72 = 7.56
-/o Enhanced-TCA
4994,44 160.14
2668.66
844,22
8158.74 55:71
786.32
8317.88 498.26
Transform2Act
4185.63 + 334.04
2393.84 692.96
8405 815.64
732.20
22.61
6901.88 374.42
NGE
1545.13 1: 626.54
881.71 459,96
2740.79 515.51
395.90 173.85
1567.84 756.74
UMC-Message
6492.90 = 441.04
1411.51 705.68
$785.40 2110.77
961.20 183.03
7354.34 2145.22
Methods
GLIDER MEDIUM
GLIDER-MEDIUM
GLIDER HARD
WALKER REGULAR
WALKER MEDIUM
WALKER HARD
BodyGen (Ours)
11996.82 395.51
10798.06 1: 298.39
12062.49 1: 513.07
12962.08 :: 537.34
11982.07 :: $20.78
w/o MoSAT
489.75 = 5.74
533.17 14.20
555,33 18.15
708.32 12.72
827.33 47.71
w/o Enhanced-
7454.55 +: 289.93
7592.03 1023.70
7286. 735.55
6069.51 652.96
6126.73 572.85
Transform2Act
5573.44 519.22
6120.37 1380.74
8685.47 1008.88
6287.15 426.99
4645.31 294
NGE
1649.60 1: 763.55
2339,90
487.22
1402.85 595.54
2600.39 481.74
1575.87 508
UMC MMC-Message
4726.44 2406.35
425.49
141.02
5417.14 2019.43
5347.70 2397.85
2783.09
1587.06
As demonstrated in Figure 6, we present the full training curves for BodyGen with baselines including Transform2Act. UMC-Message, NGE, and ablation variants of ours w/o MoSAT and ours w/o Enhanced-TCA across ten co-design environments. Each model was trained using four random seeds. For all baselines, we employed the best performance configurations reported by previous works, as detailed in Section A.3. Table 4 further presents related metrics, with each cell showing the mean and standard deviation of episode rewards for the corresponding algorithm in each environment.
A.6 ADDITIONAL ABLATION STUDIES ON TOPOPE AND ENHANCCD--CCA
We provide additional ablation studies on our proposed TopoPE and Enhanced- -CA to provide more insights, demonstrated Figure and Figure
Crawler
TeosshnCrosser
Cheefah
Swimmer
Glider Regular
1900
Rewwad
Rewodd
Rewedd
Stop
Step
S0ep
Step
Step
Glider-MMedium
Glider-Hard
Walker-Regalar
Walker-Mediun
Walker--Han
Roval
Bovard
Stap
Step
MoSAT
TopoPE Enhanced- (Ours)
GNN
TopoPE
Enhanced-
McSAT Eahunced-TCA
GNN Enhanced-TCA
Figure 11: Extensive experiments our proposed simple-yet-effective Topology Position Encoding (TopoPE) across different architectures of MoSAT and GNN, validating TopoPE as an efficient and general method for morphology representation. (1) MoSAT: w/o TopoPE + with TopoPE; (2) GNN: w/o TopoPE + with TopoPE; Both sets demonstrated the obvious performance improvements brought by TopoPE.
19

---

Published as a conference paper at ICLR 2025
Crawler
TerrninCroseer
Cheetah
Swimmer
Glider- Regular
Rowand
Rewar
Roward
Skop
Strp
Step
Gilder-Meelium
Gilder-Hard
Walker-
Walker-Medium
Walkerr-Hare Hard
10xt
Rewend
Recsed
200
2006
Stop
MoSAT TopoPE Enhanced-TeC (Ours)
TopoPE Enhanced-TCA
McSAT TopoPE
GNN + TopoPE
proposed Temporal Credit Assignment Mechanism (Enhanced-TCA: across different architectures of MoSAT and GNN, validating Enhanced- TCA mechanism efficient method for enhancing bi-level optimization. (1) MoSAT: w/o Enhanced- TCA + with Enhanced-TCA: (2) GNN: w/o Enhanced-TCA - with Enhanced-TCA: Both sets demonstrated the obvious performance improvements brought by Enhanced-TCA mechanism.
Figure 12: Extensive experiments
A.7 COMPARISON OF BODYGEN'S DESIGN SPACE WITH UNIMAL
In addition better position BodyGen, we also compare its design space and computational require- ments to those of UNIMAL (Gupta et al., 2021b), a widely recognized framework for morphology design. BodyGen and UNIMAL (Gupta al., 2021b) share similarities and differences in their approaches to morphology design, search space, and computational demands, providing insights into the trade-offs between these systems. We will compare them from several perspectives:
Initial design. The search space of UNIMAL is similar with the design space in our "crawler" environment. Both BodyGen (in the crawler environment) and UNIMAL adopt an ant-like structure with single body and four limbs extending in perpendicular directions, as the initial design Go-
Morphology actions. UNIMAL offers three basic mutation operations: adding limbs, deleting limbs, and modifying limb parameters. BodyGen employs similar set of actions but organizes them into two types: the topology design type, which includes adding limbs, deleting limbs, and passing, and the attribute design type, which focuses modifying limb parameters. While these actions conceptually aligned, BodyGen provides a more structured framework for exploring morphology changes.
Search space. UNIMAL allows for a maximum of 10 limbs, whereas the "crawler" environment used by BodyGen supports up 29 limbs, offering significantly larger space for morphological exploration. This difference highlights BodyGen's 's broader scope in accommodating complex designs.
20

---

Published as a conference paper at ICLR 2025
A.8 MORE VISUALIZATION RESULTS
In this section, we provide additional visualization results for embodied agents generated by BodyGen across co-design environments.
Crawler
Crossir
Cheetah
Swimmer
Glider-Regular
Glider-Medium
Glider-Hard
Walker-Regular
Walker-Medium
Walker-Hard
Figure 13: Visualization of embodied agents generated by BodyGen different environments.
21

---

Published as a conference paper at ICLR 2025
Figure 14: Visualization for BodyGen's 's attention map during the control process Cheetah.
22


Reviews:
[{"review_id": "1xbTxjCwth", "reviewer": "Reviewer_JKmz", "summary": "The authors investigate the problem of joint optimization of agent design and control with reinforcement learning. They propose that two outstanding problems for addressing this are imbalanced reward signals between the control and design generation steps and the effective use of morphology information. They address these two problems with two components. First, they address this by modifying the advantage function to give a better signal during design development. The latter is addressed by proposing the MoSAT layer, which uses self-attention instead of a graph neural network and an embodiment-aware positional encoding approach. Experimentally the combination of these components is necessary for performance benefits based on their results", "questions": "1. What distinguishes the author's Topology Design Stage from the design stage in Transform2Act? They also seem to have a graph-generating component and attribute-generating policy. \n\n2. How is MoSAT different from a typical Transformer Layer?\n\n3. What exactly is U_t in comparison to the value function V? \n\n4. How was 60.52% performance improvement calculated?\n\n5. In the w/o TopoPE ablation, was any form of positional information provided? Also, is the adjacency information included explicitly in the limb embeddings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper proposes several sound components that address their proposed problems (reward mismatch and using morphology information). Modifying the training objective for the design policy makes intuitive sense, at leas twith how the author's describe it. Self-attention mechanisms have been effective in other embodiment-aware research, so using these mechanisms seems a natural step for co-design and control methods. The authors also compare their positional embedding mechanism to prior works, which adds support to the benefit of their embedding approach. The experiments used to validate the components, and the author provides supporting materials support their claims", "weaknesses": "Our biggest concern is that although the proposed framework yields strong performance, the authors need to pay more attention to the novelty of their contributions. We point out, for example, that the author's \"Topology Design Stage\" sounds similar to the previous work in Transform2Act (Yuan et al., 2021, in their citations). The MoSAT layer is simply a typical Transformer Encoder layer, so it seems strange that the authors renamed a previously established neural network model. Stating this would simplify their discussion on the MoSAT layers and allow more room for discussing other components. The reward signal modifications do not adequately provide readers with the intuition of the author's proposed solution in section 4.3. In our estimates, the major contribution is merging these components into a single design and control deployment framework, which is a good contribution. Still, the components in themselves do not seem notably novel alone or need to be justified more clearly to this reviewer.\n\n\nWriting comments:\n- Replace the statement \"remains a tough nut to crack\" with a more representative statement. \n- Consider citing MAT: Morphological Adaptive Transformer for Universal Morphology Policy Learning Li et al 2024 as related work for morphology-aware positional encoding. They focus only on the control aspects.  \n - In Ablation experiments, grouping the Figure 6 ablations might be better than discussing the Table 1 ablations separately. \"Figure 6 includes ablations of X & Y, where Table 1 shows ablations of Positional differences\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dULYx5BRst", "reviewer": "Reviewer_yGDd", "summary": "The paper introduces a robotics co-design framework that co-optimized morphology and control using a Morphology Self-attention (MoSAT) architecture, inspired by transformers. It employs Topology Position Encoding (TopoPE) to enhance the integration of positional data within self-attention mechanisms. The framework optimizes these configurations through a Proximal Policy Optimization (PPO) approach, incorporating specialized credit assignments to accommodate both immediate control actions and long-term design adjustments.", "questions": "1. What are the computational demands of the MoSAT architecture, particularly when scaling to robots with a high number of limbs or complex joint structures? Is there a limit to the number of nodes (limbs) and edges (joints) the system can effectively manage? \n2. How well does the Topology Position Encoding (TopoPE) generalize across robots with significantly different morphologies? Are there any limitations in the encoding approach when dealing with highly irregular or non-standard robotic structures?\n3. given the use of multiple networks (Topology Design, Attribute Design, and Control) within the Genesis framework, how is the synchronization of updates managed between these networks? Let's say with episode I, with timestep t that is not a terminal step, if a backpropagation is performed here. How does the t+1 work, does the morphology change and controls change?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors propose a new method to overcome the computationally heavy nested design of morphology and control of robotics systems. Based on the literature review provided, the authors have substantially improved the drawbacks of previous approaches, and the results are sufficient to prove it. The proposed method significantly will alter future research in the co-design domain.", "weaknesses": "I expect a big computational increase due to the 3 complex neural networks involved. The paper didn't talk about it or compare it with baselines. \nThe robots and simulation environments used are toy problems and not sure how this can be applied to complex robotics systems like UGVs and UAVs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iaf62eRpxu", "reviewer": "Reviewer_Qsbs", "summary": "This work tackles the problem of learning a control policy and improving the agent's morphology in a unified learning framework. The authors propose to improve both the morphology of the agent and the control policy via reinforcement learning.  Several previous works employed a combination of reinforcement learning for learning the policy and of evolutionary algorithms to improve the agent's embodiment, using a new neural network for each agent. Other papers used policies based on graph neural networks to enable weight sharing across morphologies, but suffered from inefficient communication between joints. With an attention-based policy, it is possible to both share the network's weights across different morphologies, and avoid the message passing bottleneck. The morphology self-attention architecture, paired with a mechanism to facilitate credit assignment for the policy improving the morphology, largely outperforms previous methods in co-evolving the agent's morphology and control policy.", "questions": "* How does TopoPE compare to other morphology encodings? E.g., the one used in [1]\n\n* Is the morphology search space the same as in [2] (unimal)? What is different and why?\n\n* How large is the delay in the Design stage? It is not obvious to me that the rewards come particularly delayed, unless the design stage has many steps. In fact, while the reward due to the updated design comes after some interactions of the agent with the environment, during the interaction there is no design decision being made, so this should not be perceived as a delay by the Design policy.\n\n* How do the agents' morphologies look at the end of the training? What is their performance, e.g., compared to agents with the base morphology trained with RL? Can the learnt morphology be used to train a new agent with it?\n\n[1] Trabucco, B., Phielipp, M., & Berseth, G. (2022, June). Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning (pp. 21677-21691). PMLR.\n[2] Gupta, A., Savarese, S., Ganguli, S. and Fei-Fei, L., 2021. Embodied intelligence via learning and evolution. Nature communications, 12(1), p.5721.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow. The motivation is clear: previous work employed certain design choices (specifically evolutionary algorithms, graph neural networks) which, if successfully replaced with more effective solutions (reinforcement learning and self-attention) should lead to a clear improvement, as indeed demonstrated by the final performance. The proposed framework required the introduction of new solutions for the positional encoding of the agent's joints and to improve the credit assignment for the \"slower\" reinfrocement learning process, which is the morphology improvement. Both problems have been successfully addressed. Therefore, the paper fills a a gap in this specific field, obtaining a large performance improvement over previous methods.", "weaknesses": "In the introduction and in the related work, this paper cites previous studies on policy-morphology co-evolution. However, the authors likely missed other papers proposing attention policies to deal with variable morphologies ([1] and [2]) and it would be important to at least compare MoSAT with those architectures. Furthermore, from the description of MoSAT I could not understand whether it is any different from a standard Transformer network. I would suggest explaining MoSAT in terms of its differences from a Transformer, rather than defining each component of the block.\n\nThe paper includes some details about methods introduced by other papers (e.g., details about PPO, or about the topology design, which seems identical to [3]), which could be shortened and the saved space used to better analyze the results. In fact, while the performance is convincing and the ablations satisfactory, the authors could draw inspiration from, e.g., the analysis in [4], to better grasp the effects of the morphology's evolution beyond the task performance.\n\nThe topology position encoding (TopoPE) is listed among the main contributions, so it would be good to better show its properties. For example, the authors claim that TopoPE is more stable than previous encodings when the morphology changes, but this claim is not supported by any specific experiment.\n\nThe modified GAE requires better explanation. For the \"Control Stage\", why is there a discounted advantage summed to the temporal difference? While for the Design Stage, U_t seems like a monte-carlo estimation of the reward, why using that instead of bootstrapping the value function?\n\nMinor:\n* line 080, facilitating -> facilitates\n* line 139, an -> a\n* line 242, \"achieve the co-design process\"?\n* line 369, what exactly is inspired by BERT?\n* line 443, Yuan et al. repeated\n* line 463, I would rather express the size of the model in number of weights\n* line 479, I would avoid sentences like \"despite the impressive results of our approach\"\n\n[1] Trabucco, B., Phielipp, M., & Berseth, G. (2022, June). Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning (pp. 21677-21691). PMLR.\n[2] Chiappa, A. S., Marin Vargas, A., & Mathis, A. (2022). Dmap: a distributed morphological attention policy for learning to locomote with a changing body. Advances in Neural Information Processing Systems, 35, 37214-37227.\n[3] Yuan, Ye, et al. \"Transform2act: Learning a transform-and-control policy for efficient agent design.\" arXiv preprint arXiv:2110.03659 (2021).\n[4] Gupta, A., Savarese, S., Ganguli, S. and Fei-Fei, L., 2021. Embodied intelligence via learning and evolution. Nature communications, 12(1), p.5721.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Io1G3fJ8Th", "reviewer": "Reviewer_GZKf", "summary": "The paper presents \"Genesis\", a reinforcement learning (RL) framework for the co-design of robot morphology and control policies. Genesis addresses the main limitations of current methodologies which struggle with the high-dimensional design space and optimize the morphology separately from the control policy. The key contributions include (1) a temporal credit assignment mechanism to balance reward between morphology and control optimization and (2) a topology-aware self-attention mechanism for efficient morphology representation. Experiments across multiple environments and ablations demonstrate that Genesis significantly improves performance compared to baseline models.", "questions": "- Does Genesis achieve better solutions than evolutionary search methods simply due to access to a superior hyperparameter space (e.g., varying thickness that may affect inertia and speed)? A direct comparison of best morphologies across methods, including an exploration of the role of hyperparameter configurations, would strengthen this aspect.\n- Does Genesis prioritize certain features to boost robustness or performance in various environments? Furthermore, it would be good to visualize how morphologies evolve over time. Are the retrieved morphologies consistent across different seeds, or does Genesis find alternative solutions depending on initialization? Additionally, it would be interesting to analyze the topographical encoder representation and how attention shifts based on morphology.\n- In real-world applications, slight morphological changes may occur. Ideally, the control policy should adapt to such changes in a zero-shot manner, as in [1,2,3]. A key question is how well Genesis handles IID morphologies (those seen in training) versus OOD morphologies.\n- Once the best morphology has been found, does optimizing it independently yield the same or better performance? Does the control policy of Genesis benefit from the co-training?\n- What are the main failure modes of Genesis?\n- Why is model size evaluated in MB rather than by the number of parameters?\n- What is U_t in equation 4.12?\n\n[1]: Chiappa, A. S., Marin Vargas, A., & Mathis, A. (2022). Dmap: a distributed morphological attention policy for learning to locomote with a changing body. Advances in Neural Information Processing Systems.\n\n[2]: Hong, S., Yoon, D., & Kim, K. E. (2021). Structure-aware transformer policy for inhomogeneous multi-task reinforcement learning. In International Conference on Learning Representations.\n\n[3]: Huang, W., Mordatch, I., & Pathak, D. (2020). One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Genesis introduces an innovative end-to-end co-design optimization that combines attention-based morphology processing with reward balancing between morphology and control.\n- The experiments demonstrate significant improvements over current methods, and the ablations highlight the importance of the topographical positional encoder.\n- The paper generally explains its contributions and mechanisms clearly, with nice illustrations.\n- This approach addresses important limitations in the field and, therefore, could influence future research in co-design optimization.", "weaknesses": "- Although Genesis demonstrates substantial improvements over current methods, the paper lacks a detailed explanation of specific solution differences between Genesis and previous approaches. \n- Genesis's morphological decisions are not fully interpretable, and a detailed discussion of the morphological features it favors could enhance the paper.\n- While Genesis aims for continuous co-optimization, this approach risks overfitting to specific scenarios or static environments.\n- The clarity of some paragraphs related to the methods could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)