Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization
Authors: Yanning Dai
Yuhui Wang
Dylan R. Ashley
Jürgen Schmidhuber
Venue: ICLR 2026 Poster (2026)
Keywords: Morphology-Control Co-Design
Stackelberg Game Theory
Stackelberg Markov Game
Phase-Separated Interactions
Non-Differentiable Interactions
Reinforcement Learning
Proximal Policy Optimization
Bi-Level Optimization
Primary area: reinforcement learning
TL;DR: The first work to introduce Stackelberg Game-Theoretical methodologies into the morphology-control co-design problem, formulated as a Stackelberg Markov Game with phase-separated and non-differentiable interactions.

Abstract: Morphology-control co-design concerns the coupled optimization of an agent’s body structure and control policy. This problem exhibits a bi-level structure, where the control dynamically adapts to the morphology to maximize performance. Existing methods typically neglect the control’s adaptation dynamics by adopting a single-level formulation that treats the control policy as fixed when optimizing morphology. This can lead to inefficient optimization, as morphology updates may be misaligned with control adaptation. In this paper, we revisit the co-design problem from a game-theoretic perspective,  modeling the intrinsic coupling between morphology and control as a novel variant of a Stackelberg game. We propose *Stackelberg Proximal Policy Optimization (Stackelberg PPO)*, which explicitly incorporates the control’s adaptation dynamics into morphology optimization. By modeling this intrinsic coupling, our method aligns morphology updates with control adaptation, thereby stabilizing training and improving learning efficiency. Experiments across diverse co-design tasks demonstrate that Stackelberg PPO outperforms standard PPO in both stability and final performance, opening the way for dramatically more efficient robotics designs.

Published as a conference paper at ICLR 2026
EFFICIENT MORPHOLOGY-CONTROL CO-DESIGN VIA STACKELBERG PROXIMAL POLICY OPTIMIZATION
Yanning Dai Yuhui Wang Dylan R. A 1,2,3,4 Jürgen Schmidhuber' 1,2,3.4 1,2,,4
Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia.
Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Lugano, Switzerland.
3 Università della Svizzera italiana (USI), Lugano, Switzerland.
4 Scuola universitaria professionale della Svizzera italiana (SUPSI), Lugano, Switzerland.
ABSTRACT
Morphology-control co-design concerns the coupled optimization of an agent's body structure and control policy. This problem exhibits a bi-level structure, where the control dynamically adapts to the morphology maximize perfor- mance. Existing methods typically neglect the control's adaptation dynamics by adopting a single-level formulation that treats the control policy fixed when optimizing morphology. This can lead to inefficient optimization, as morphology updates may be misaligned with control adaptation. In this paper, we revisit the co-design problem from game-theoretie perspective, modeling the intrinsic cou- pling between morphology and control as a novel variant of Stackelberg game. We propose Stackelberg Proximal Policy Optimization (Stackelberg PPO), which explicitly incorporates the control's s adaptation dynamics into morphology opti- mization. By modeling this intrinsic coupling, our method aligns morphology up- dates with control adaptation, thereby stabilizing training and improving learning efficiency. Experiments across diverse co-design tasks demonstrate that Stack- elberg PPO outperforms standard PPO in both stability and final performance, opening the way for dramatically more efficient roboties designs.
Training Process
0%
5%
20%
60%
100%
Stackelberg PPO (ours)
PPO
Figure Showcasing how our Stackelberg PPO can autonomously design task-specific robots for the "Pusher" task, starting with a bare-bones structure and ultimately evolving into so- phisticated design with arm-like structures for pushing boxes and log-like limbs for movement. This highlights the method's ability to create adaptive and complex designs. In comparison, the traditional PPO method generates simpler structures that can't support more complex behav- iors. For more examples of evolved designs and animations, well open-source code, visit: https//yanninggai, http://aammnnddiiiiiib...o...
"Equal contribution.
1 Corresponning author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Morphology-control co-design optimizes both an agent's body structure and its control policy. The morphology defines the agent's 's design, including topology, geometry, joint layout, and actuation limits, while the control policy dictates how this structure produces behavior interact with the environment (Paul, 2006; Ha & Schmidhuber, 2018). Both aspects are critical for task performance. For instance, a quadruped robot with rigid legs can't walk without an appropriate gait policy, and a locomotion policy is ineffective if the robot's morphology lacks the necessary joints to support movement, These examples show that morphology and control must be co-designed to ensure com- plementarity. Agents optimized under this paradigm are typically more versatile, robust, and effi- cient than those optimized for only one aspect (Sims, 1994; Lipson & Pollack, 2000; Bongard et al., 2006; Kriegman al., 2020).
A key challenge in morphology-contee control co-design is the dynamic interplay between morphology and control: the control policy must adapt to the evolving morphology to fully realize its potential; without this, the morphology's true performance may be underestimated (Schaff' & Walter, 2022). Existing methods often treat morphology and control as separate optimization processes (Cheney et al., 2018; Lu et al., 2025; Yuan et al., 2022). Specifically, morphology updates are typically made assuming a fixed control policy, neglecting that control must adapt to structural changes. As a result, morphology updates become misaligned with the optimal control response, leading to unstable and inefficient optimization in the morphology space and, ultimately, degraded performance.
In this paper, we revisit the co-design problem from a game- theerthe perspective, formulating it as chase-separated Stackelberg Markov Games (SMGs), where the leader first updates the morphol- ogy, and the follower adapts its control policy accordingly. The key insight is that the leader must anticipate how morphology changes will influence the follower's control dynamics, enabling more effective designs. This perspective leads to the development of Stackelberg Implicit Differentiation (SID), a technique that incorporates the follower's is adaptive dynamics into morphology optimization. However, applying SID to sypphology-contrro co-design is non-trivial due to the nature of the interaction and the non-differentiable interface between the leader and follower.
To address these challenges, derive Stackelberg policy gradients tailored to chase-separatee SMGs with non-differentiable interfaces, building on SID to explicitly account for the follower's adaptation in the morphology optimization process. Since direct differentiation is obstructed by the non-differentiable leader-follower interface, we apply the log-derivative technique (Williams, 1992) to derive new Stackelberg surrogate formulation that bypasses this issue and provides a tractable gradient estimator. We further provide theoretical guarantees, showing that these surrogates are lo- cally equivalent to the true Stackelberg gradients. To stabilize training under large policy shifts, we adapt PPO's likelihood -atioo clipping to our Stackelberg framework, ensuring robust optimization of the surrogate gradients. Our method, Stuckelberg PPO, outperforms otate-of-the-art baselines by 20.66% on average and by 32.02% on complex 3D tasks.
2 RELATED WORK
Morphology-Contwee Co-design Co-optimizing morphology and control is attracting increasing attention in embodied intelligence (Li et al., 2024; Huang et al., 2024b; Liu et al., 2025). Prior work optimizes only continuous attributes under a fixed topology, without generating new structural topologies (Banarse et al., 2019; Huang et al., 2024a). Early topology-editng work treated co-design as a discrete, non-differentiable search solved using evolutionary strategies (Sims, 1994; Cheney et al., 2018), requiring each morphology to be paired with a separately trained controller and thus incurring high computational cost. Subsequent methods introduced structural priors and parameter sharing to reuse experience across designs (Dong et al., 2023; Zhao et al., 2020; Wang et al., 2019; Xiong et al., 2023). More recent RL-based approaches cast structure generation as sequential edits in an MDP (Gupta et al., 2021; Yuan et al., 2022), with graph and attention architectures improving representation quality (Chen et al., 2024; Lu et al., 2025; Yuan et al., 2022). However, the discrete nature of morphology-editing operations blocks gradient propagation across the morphology-come control interface, preventing efficient learning by capturing their coupled dynamics. Our work establishes a gradient-driven pathway that allows controller adaptation to directly affect morphology updates.
2

---

Published as a conference paper at ICLR 2026
Stackelberg Game Learning systems with asymmetric components often exhibit directional de- pendencies, where one module's 's decisions influence another's 's adaptation in a non-reciprocal man- ner (Schmidhuber, 2015). Stackelberg games formalize this asymmetry, with leader commit- ting to strategies that followers then best-respond to. Classical approaches study static normal- form games (Başar & Olsder, 1998; Conitzer & Sandholm, 2006; Von Stengel & Zamir, 2010) while more recent extensions integrate this structure into sequential decision processes and RL & Parkes, 2023; Zhong et al., 2023; Bai al., 2021), often incorporat- ing confidence aanfieence-wwareeeee or optimistic mechanisms to manage follower uncertainty (Ling et al., 2023; Kao et al., 2022; Kar et al., 2015; Mishra et al., 2020). Another line of research applies implicit differentiation to enable direct gradient flow from the follower to the leader in Stackelberg games. A complementary line leverages implicit differentiation to propagate follower gradients to the leader (Zheng et al., 2022; Yang et al., 2023; Vu et al., 2022), typically under DDPG-style settings with ex- plicit action-level coupling and alternating updates. Our problem differs in two key aspects: leader actions (morphology edits) cannot be directly transmitted to the follower, and both agents use non- alternating PPO-style updates. We extend implicit Stackelberg gradient methods to this more general regime, enabling (to our knowledge) the first application of implicit Stackelberg differentiation to morphology-com control co-design under PPO algorithm.
Sanmeworks(Gersggassse
3 PRELIMINARIES
Proximal Policy Optimization (PPO) Our method builds upon PPO due to its empirical stability and simplicity. In reinforcement learning, an agent interacts with the environment by observing a state 8,, selecting an action a, according to its policy Tæ. and receiving feedback the form of rewards. Vanilla policy gradient methods (Sutton, 1984; Williams, 1992: Sutton et al., 1999) optimize Ty using a surrogate objective that locally approximates the true performance, which has been shown cause instability when policy updates become too large (Schulman al., 2015). PPO (Schulman et al., 2017) addresses this by constraining the likelihood ratio between new and old policies through a clipping technique:
CPO (0) = Ex min (rr())A,,
Te, (8)) is the likelihood ratio and A, is an estimator of the advantage function. The clipping mechanism prevents likelihood ratio r, (0) from deviating excessively, thereby limiting policy updates and balancing stability with performance improvement.
where re(0)-
Stackelberg Game A Stackelberg game models sequential and asymmetric interaction in which a leader first commits to strategy, and follower subsequently optimizes its strategy in response. One key characteristic of a Stackelberg game is that the leader explicitly accounts for the follower's best-response dynamics when making its decision. Let 0² and 0F denote the decision variables of the leader and the follower, and let (- O²) and J (0=, 0F.) denote their respective objectives. The leader solves the following bi-level optimization problem, referred to the Stackelberg objective:
J (LL 0, (B6))
s.t. 0., (0²) arg max J 8F
max gL
where 8. (8L) denotes follower's best response. The leader's gradient can be written as
Jon.b (0L, (00))
VOLJ 0F) + (VoL0f Vor (8L OF)
(2)
direct glreiingradient
indirect gradient vin influencing follower
The first term, the direct gradient, captures the steepest direction along which the leader can directly improve its objective. The second term, the indirect gradient via influencing follower, captures the indirect strategic effect: it characterizes the direction along which the leader can further improve its objective by influencing the follower's 's response. In particular, the term (VoL0," (0')) the follower's optimal decision changes in response to variations the leader's decision. The Ja- cobian V 0f (6L) follows from the first-order optimality condition of the follower's maximization problem, JF (8", (g000.") 0.) = O, which indicates that 05 (0²) is an implicit function of the leader's variable 0L. This yields
(o=)) captures how
OVL 0 (80)) = - gggg JF (0L,0P) (V2p o J
(3)
3

---

Published as a conference paper at ICLR 2026
af
af
Of
Input/Oripuu
Leader Policy
Leader Policy
Follower Policy
Follower Policy
Policy
Non-differentiable
7+1
Leader Phase
Follower Phase
Figure 2: Illustration of the phase-separated Stackelberg Markov Game for morphology-control co- design. In the leader phase (blue part), the agent incrementally edits the morphology via discrete topology-altering actions, producing a terminal morphology s4. In the follower phase (green part), the control policy is optimized based on this morphology.
This implicit differentiation framework, often referred Stackelberg implicit differentiation (SID), is widely used in Stackelberg reinforcement learning (e.g., Stackelberg DDPG (Yang et al., 2023)), bi-level optimization (Zucchet & Sacramento, 2022), and meta-learning (Pan al., 2023).
Morphology-Control Co-Design Morphology-control co-design aims to jointly optimize a robot's morphology (body structure) and its control policy. The morphology defines the robot's structural and physical properties, such as topology (limb and joint connectivity), geometry (body proportions, limb lengths), and material properties for soft robots. The controller determines the robot's s actions based on its body, using motor commands, torque signals, or higher-level behaviors.
The process unfolds in two stages. First, the morphology is constructed incrementally through sequence of step-by-step editing actions (e.g., adding/removing limbs, adjusting lengths, attaching joints), rather than producing complete morphology in a single step. This incremental approach is necessary due to the high-dimensional and combinatorial morphology space, which makes shot generation intractable. Formally, starting from an initial morphology st. morphology-cdittee actions of are applied sequentially until terminal morphology s is obtained:
df ~ T s!),, * p- (' t 0, . --
The transition function ph updates the morphology through discrete topology-altering operations, making the morphology dynamics non-differentiable.
Next, the control policy is optimized based on the generated terminal morphology s/- The mor- phology defines the controller's action space (e.g., available joints and actuators), state space (e.g., joint positions, forces, velocities), and underlying dynamics. At each timestep, given a state of the controller applies control action af and transitions new state:
of ~ x (.1 ( s4), s4 PP 1 s
st), t= T-
The overall process is illustrated in Figure 2.
Existing works (Lu et al., 2025; Yuan et al., 2022) commonly model the co-design problem as bi-level optimization structure, similar to the formulation in eq. (1), where 0f and oF represent the parameters of the morphology policy T/L2 and the controller policy The. respectively. However, to simplify implementation, these approaches typically adopt single-level shared objective, given by
max Jshared (0" = E gr. B
x Rd (s of ifis)))
(4)
where (sF af s/)) measures the controller's performance under the specified morphology, such locomotion speed or task success.
This objeed-o formulation treats the controller parameters 0² as jointly optimized with 6*, ignoring that o5 (0L) is implicitly determined by the morphology. Consequently, the leader up-
4

---

Published as a conference paper at ICLR 2026
date includes only the direct gradient term and omits the implicit response term, which may steer morphology optimization in a direction misaligned with the controller's s best response.
4 A STACKELBERG PERSPECTIVE ON CO-DESIGN
In this section, we formalize morphology-come co-design from a Stackelberg game-theoretic per- spective. We first introduce asymmetric objectives for the morphology designer and the controller, and then cast their interaction as leader-follower Stackelberg game.
Asymmetric Objectives In contrast to classical approaches that adopt single shared objective (eq. 4) (Lu et al., 2025), we introduce asymmetric objectives for morphology and control:
T-
10 0F) E
(s Σ
abf) Σ A-TRR ( (sf.af i
(5)
The reward (s)' "ls//,aa )
provides immediate feedback for copphology-ditinng actions, typically re- flecting structural costs such as material usage or design complexity. The morphology objective therefore combines immediate morphology-editing rewards with long-term control performance.
The control objective focuses solely on maximizing long-term return, conditioned on the terminal morphology induced by ApL:
(F E
Σ R af
This formulation makes asymmetry explicit: control adapts to fixed morphology, while morphol- ogy optimizes both its structural objectives and downstream control performance.
Stackelberg Formulation Although existing work formulates co-design bi-level optimization problem, it typically treats the control policy as fixed when optimizing the morphology (see eq. 4) (Lu et al., 2025; Yuan et al., 2022), thereby failing to capture adaptive control dynamics. We revisit this structure from a Stackelberg game-theoretic perspective, where the morphology policy acts the leader and the control policy acts as the follower. This perspective emphasizes strategic anticipation: the leader commits to a decision while accounting for the follower's rational response.
This perspective naturally gives rise to Stackelberg Implicit Differentiation (SID), leading to differ- ent optimization behavior by incorporating the follower's adaptive dynamics into morphology op- timization (see eq. 2). However, applying SID morphology-contee co-design is non-trivial due to two intrinsic characteristics: a) phase-separated interaction structure and b) non -in-differentiable leader-follower interface, both of which obstruct gradient propagation across the interface.
a) Phase-separater interaction. Unlike standard Stackelberg Markov Games (SMG) in which the leader and follower alternate actions (Li al., 2020), the interaction the co-design problem separated: the leader acts for T steps, after which the follower acts for the remaining horizon. Formally, we define this structure as Phase- Separated Stackelberg Markov Game
Definition 1. Phase-Separated SMG between leader policy T/ and follower policy T.P parameterized by 0² and OF respectively, is defined
g = ((S",A','') , ,T), (ST AF,PF ,P,, .RF. RRRRR))) ,p)),7).
(i) The leader's components are given by its state space S². action space A", transition function ph. reward function R², initial state distribution p²b, and acting horizon T.
(ii The leader-follower interaction is phase- separase (i.e., nontatternt:ing): the leader first acts for T steps, producing terminal state s/,, after which the follower acts until termination.
(ii) The leader and follower interact through the terminal state st E St,, induced by the leader's action sequence under the transition dynamics PLL The follower acts conditioned on terminal state s/ and all its components (S ,A, ,PF. RR... "R",""" .p²)) are defined conditionally s/.
(iv) The leader aims solve the Stackelberg objective defined in eq. (1).
5

---

Published as a conference paper at ICLR 2026
b) con-differnntiabee interfaces. In morphology-contrr co-design, the resulting Saasse-Sparated SMG exhibits an inherently non-differentiable Seader-follower interface. Specifically, the leader's transition function p updates the morphology through discrete topology-altering actions, so the terminal state of linking the leader and follower does not permit direct gradient propagation from the follower to the leader.
Value Functions For subsequent analysis, we introduce value functions induced by the asymmet- ric objectives. Analogous to standard RI, the leader's Q-function, induced from its objective in eq. (5), defined as
o (str =E Σ (s Σ ne of :s4)sst = st of a
This Q-function captures the leader's long-term return from given for both its rewards before morphology finalization and the follower's rewards conditioned on the final morphology. From this, the leader's advantage function is defined A A (sk, at)
pair, accounting
(sk,at) a The advantage function measures how much better ou (st, (5,.a))) -Bttoa @h
(or worse) a specific action at is compared to the leader's average behavior at state sy. A follower's advantage function A (s s/1s can be defined analogously from its objective in eq. (6).
METHOD
In this section, we present how to realize Stackelberg Implicit Differentiation (SID; see Section 3) within the phase-separated Stackelberg Markov Game (SMG) framework introduced in Section 4. This enables to account for the follower's adaptive response during morphology optimization, which is typically ignored in prior approaches (Lu et al., 2025; Yuan al., 2022).
A direct way to implement SID is via dients from the follower to the leader through the leader-follower interface, as in Stackelberg MADDPG (Yang et al., 2023). However, as highlighted in Section 4, the phase-separated SMG formulation of the co-design problem exhibits a non-differentiable temporally separated interaction structure, both of which complicate gradient propagation across the interface. These challenges require new derivations of the Stackelberg gradients, as presented below.
which propagates gra-
interface and
5.1 STACKELBERG POLICY GRADIENT
We now introduce Stackelberg implicit differentiation into phase-separaiee Stackelberg Markov Game defined in Definition 1. Since this formulation departs from the classical SMG, we develop new derivations for all gradient components in eqs. (2) and (3). We present each term in turn.
Cross-Derivative Verge JF (F"0", (see eq. (3)). This is the most challenging term. Unlike clas- sical SMGs where the follower directly takes the leader's 's action input, in our setting the interface the terminal state s// generated through the transition pL. Backpropagation-three www.tthhhoooueereeee methods (e.g.. Stackelberg MADDPG) are infeasible here, since reaching 6 would require differ- entiating through the non -in-differentiable transition pt. Instead, we derive the cross-derivativ using the log-derivative technique, analogous to the stochastic policy gradient (Sutton, 1984; Williams, 1992; Sutton et al., 1999). which bypasses the transition's ''s non-differentiabiliy while relying only on sampled trajectories. Let (0% 0²") denote the parameters of the behavior policies used for col- lecting data. Formally, we obtain the following theorem.
Theorem 1. We define the surrogate
m2 (s")
(a (a" ;
||
FT (0',0' 0% 00 0.) = =cB
TT
TF
A 10
(5 :aaa:stii
(7)
oc. (
To (a
Then, have V Vorgr JF (0+, (0. (04,0F)lrrzg, rr -0=05-
V or ff. LF (ot, oF:00 ob 05) 10r 10 -0-

---

which is positive Published as a conference paper at ICLR 2026
In eq. (7). the outer expectation is taken over s d/g. a² Tog. st dog. where dog d2' P(s mbp)) is the visitation distribution probability of leader policy at step and dz. (xt) a 1/T 1/T E. d/g (st). The inner expectation is taken over of (: (::4), of (3 where d/r denotes the follower's visitation distribution. The constant c- nor- malizes the distribution, and its effect can be absorbed by the learning rate in practice. Proofs of this and subsequent theorems are provided in Appendix B. This theorem shows that the cross-derivative can be expressed as an expectation involving likelihood-ree mators, thereby extending the classical policy gradient theorem to capture in our phase-separated SMG.
dLeT
sepporrtnce-weigheee advantage esti- coupling
First-Order Derivatives JL (0), and Vor.J( (see eq. 2). These first-order terms are relatively straightforward, as they follow the same structure as the policy gradient the- orem (Sutton, 1984; Williams, 1992; Sutton al., 1999). They quantify how the leader's objective changes with respect to its own parameters (leader's s direct gradient) and w.r.t. the follower's 's param- eters. Both can be expressed using advantage functions under importance weighting, as follows. Proposition 1. We have
OF)
Jo 00) oz oF ag Vo, Ct. (8). ;F; V 2 (0 (P) 8 -===0 =F=
; 9.0) oz ot oF =6 Vo, or C (0r. g0 0²)
0z === oF =6²-
L (a41s²)
where Ct. (0t, 00,00 (00,0%) =
AL % (0t, (ot,at)
oy (at1)2))
nst of (a s s ;
2 (00,0% 8L 06,00 05)
E
(s :a:444)
F Tog (a*1s s/p)
Inverse of Second-Order Derivative (Hessian) (2 J (0L, )0)) (see eq. 3). This last com- ponent involves the inverse Hessian. Although the Hessian can be computed from the derived loss function (see Appendix Proposition 2), the Hessian is typically indefinite due to the advantage term, making its inversion unstable. A standard remedy is to approximate it by the Fisher information matrix, F(OF) = o log " (aF sf; x/)) Ver log xΦ. (a { > ;+))
semi-definite and can be estimated via the KI. divergence between policies:
(9) This -auural-gradiett approximation, used in methods such as natural policy gradient and trust re- gion policy optimization (Kakade, 2001; Schulman et al., 2015), avoids indefiniteness and improves stability. Further stability is obtained by regularizing the Hessian with a small multiple of the iden- tity (2 CKL AT) with X O, which has been shown interpolate between the standard policy gradient (when + oo) and the Stackelberg gradient (when X + 0) (Yang et al., 2023).
F(0²) = V C 8" (00000 = V2r KL T
(=:ss ss; (-15
5.2 ALGORITTMS
Based on the surrogate functions in Eqs. (7) to (9), we compute the leader's Stackelberg gradient in Eq. (2). Since these surrogates are locally equivalent to the true Stackelberg gradients, we adopt the likelihood-rathe clipping technique from PPO (Schulman et al., 2017) to constrain policy divergence and ensure stable optimization. Note that this application is not a simple reuse of PPO clipping. Rather, it is grounded in our aprlxxapproiiitiion theory on the newly derived Stackelberg surrogate (see Theorem 1). Moreover, the expectation terms are estimated from sampled trajectories. This yields sample-based surrogates with PPO clipping, denoted by L. and the corresponding estimation of the leader's Stackelberg gradient can be expressed as
step
Vgl. Ft. (Or, (z.) = Vo. VOLOE (Vo + a)) '
(10)
stop
7

---

Published as a conference paper at ICLR 2026
Stackelberg PPO (ours)
BodyGen
Transform2Act
NGE
ESS
Crawler
Cheetah
Swimmer
12500
15000
1500
10000
12000
1200
Reward
7500
Reward
Reward
5000
6000
600
2500
3000
300
0M
10M
20M
30M
40M
50M
0M
10M
20M
30M
40M
50M
00
10M
20M
30M
40M1
50M
Step
Step
Step
Walker-Hard
Glider-Hard
TerrainCrosser
12000
5000
12000
4000
Reward
Reward
3000
6000
2000
3000
3000
1000
0
50M
Step
Stop
Step
Pusher
Stepper-Regule
Stepper-Hard
7500
6000
3000
6000
Reward
Reward
Reward
1500
2000
4500
3000
3000
1000
1500
1500
0
10M
20M
30M
10M
0
10M
20M
20M
40M
10M
30M
4084
Step
Step
Step
Figure 3: Performance curve with respect to the number of follower steps during training. Shaded
regions denote standard error across seven random seeds.
We refer to this overall procedure Stackelberg PPO, which integrates PPO-style clipping into the Stackelberg gradient computation. We first compute step in the above equation, which can be efficiently implemented using the conjugate gradient method. Conjugate gradient only requires Hessian-vector products, which can be obtained without explicitly constructing the Hessian via Pearlmutter's method (Pearlmutter, 1994; Møller, 1993): ViC(() (V v). We then compute step 2, which in turn only requires Jacobian-vector products. These can likewise be com- puted efficiently without explicitly forming the full Jacobian by using the Jacobian-vector product operation provided by automatic differentiation frameworks.
EXPERIMENTS
Our goal is to whether leveraging Stackelberg implicit differentiation regularize the leader's gradient can improve sample efficiency and final performance.
All experiments are conducted on MuJoCo-based morphology-conteee co-design tasks. We adopt benchmarks from prior work, including three flat-terrain tasks (Crawler, Cheetah, Swimmer Gl oli,, Wa Walker) and one complex-terrain task (TerrainCrosser) (Lu al., 2025). To further evaluate performance under more challenging conditions, we introduce two new tasks, Stepper-Regules and Haspper-Harde where the agent must climb stair-like structures. These tasks require morphologies capable of effective climbing in addition to robust control. To test generality beyond locomotion, we also include a contact-rich 3D manipulation task, Pusher, designed to evaluate whether co-design methods objectives. Additional results on other tasks are provided in Appendix due to space constraints. In all environments, morphologies are represented tree structures with constraints depth, branch-
evolve structures aligned with manipulation
8

---

Published as a conference paper at ICLR 2026
k=0.0
7500
Am0.5
6000
k-1.0
6000
Reward
5-5.0
Reward
4500
A=100
4500
3000
3000
1500
1500
Fisher Gradient
Analytic Gradient
0M
10M
20M
30M 40M 50M
6M
10' 20M
Step
(a) Visualization
(b) Regularization Parameter À
(c) Hessian
Figure 4: (a) Evolved morphologies. Ablation studies (b) À sweep from 0 to and (c) Fisher
information matrix on/off, both evaluated on Stepper-Rege Regular task.
ing factor, and joint degrees of freedom. While structural complexity and terrain difficulty vary, the reward function consistently emphasizes forward velocity, ensuring fair comparisons of how differ- ent methods balance morphology and control. Each algorithm evaluated with seven random seeds per task. All reported learning curves show mean values with shaded areas representing standard deviations. Further details and visualization of the environments are provided in Appendix C.
6.1 COMPARISON WITH BASELINES
We implement our Stackelberg PPO on top of BodyGen (Lu et al., 2025), a PPO-based framework that employs transformer-biat based co-design with graph-aware positional encodings, optimizing mor- phology and control independently under shared rewards. BodyGen serves as our primary baseline, with the only modification being the use of Stackelberg policy gradients. Implementation details are provided in Appendix C. In addition to BodyGen, we compare Stackelberg PPO against several advanced co-design methods:
Evolutionary Structure Search (ESS) (Sims, 1994): A canonical vvolutionary-algoriin approach to robot design, where candidate morphologies are scored by handcrafted fitness functions. Here we instead use a lightweight RL-based training loop for principled evaluation.
Neural Graph Evolution (NGE) (Wang al., 2019): Evolutionary search over graph-structured morphologies with GNN controllers, each generation training the inherited parent controller.
Transform2Acr (Yuan al., 2022): Concurrent RL co-design using separate GNNs for morphol- ogy and control within unified PPO training, with joint-specific MLP heads for universal control.
Figure 3 presents the learning curves across all environments. Stackelberg PPO consistently achieves the best performance, yielding an average +20.66% improvement over the strongest base- line. Compared to evolutionary approaches (ESS, NGE), it attains substantially higher sample ficiency by avoiding the costly rollouts required to evaluate each morphological candidate. Rel- ative to the vanilla gradient method without Stackelberg differentiation (BodyGen), Stackelberg PPO achieves superior results in both sample efficiency and final performance. The advantage is most evident on challenging 3D tasks with large design spaces (Crawler, Stepper-Reuilas stepper-Has Pusher), where our method delivers an average +32.02% improvement. Fig- ure 4(a) showcases examples of the evolved creatures generated by our method. Additional mor- phology examples and evolution processes are provided in the appendix E.1 and E.5.
6.2 ABLATION STUDIES
We conduct ablation studies to validate key components of Stackelberg PPO, including (1) the reg- ularization parameter that controls gradient interpolation (eq. 10); and (2) the Fisher gradient approximation of the Hessian for stability (eq. 9).
Regularization Parameter (eq. 10). The parameter À interpolates between pure Stackelberg gradients and standard policy gradients. We evaluate Stackelberg PPO the Stepper- ar

---

Published as a conference paper at ICLR 2026
10000
2.0
7 Stackelberg PPO
BodyGen
0.1
8000
6.2
KL Divergencc
0.2.
6188.99
3663.06g571.
Reward
6000
1.2
5-0.6
Moy M
7215 440.02
4685.94464s.22
4000
noc
0.8
no clip
8260. 2260.74211..88
6S79.60 2579.00
2000
0.4
MW
6739. 6739.51
3375. 2488.54
6874.342m0.4 2216.77.0077.
0.0
8 M 10 M M 30 M. 50
0 M
10M 20 M 3 40 M 50 M
Steps
Steps
(a) Cumulative Reward
(h) KI--divegeence
(c) Leader Horizon T
Figure 5: (a) Reward curves and (b) KI.-divergence traces for different clipping thresholds e. (c)
Performance comparison under varying leader horizons T. All evaluated Sepper-RReulan
environment with À {0.0, 0.5, 1.0, 5.0, 10.0, oo), where À corresponds to no regularization and À reduces to the vanilla gradient without Stackelberg differentiation. Figure 4(b) shows robust performance for E [0.5, 10). with degradation only at the extremes (A = or oo). This highlights both the robustness of the method to À values and the necessity of regularization.
Hessian Computation (eq. 9). We compare our Fisher approximation with direct analytic second- order gradients (eq. 11). As shown in Figure 4(c), the Fisher approximation achieves stable learning with nearly twice the performance the analytic gradient (6000 vs 2500). This improvement arises from the positive semi-definiteness of the Fisher matrix, which avoids the numerical instabilities caused by the indefinite raw Hessian.
Sensitivity to PPO Clipping Threshold e. We evaluate the sensitivity of Stackelberg PPO to the clipping parameter by sweeping over multiple thresholds and measuring its effect on task perfor- mance and KL--dierreence stability. Figure 5(a) and (b) shows that moderate clipping (e.g., S 0.4) yields stable learning with low KL divergence, while removing clipping causes rapid KL growth and clear performance degradation. Full quantitative results are reported in Appendix E.3.
Leader Horizon (eq. 7). We examine how the leader horizon T affects structural optimization. As shown in Figure 5(c), larger horizons generally improve performance by enabling richer mor- phology edits, while overly large values (c.g.. T = 11) become harder to optimize and cause mild degradation--ye still outperform very small horizons such as T = Importantly, increasing T does not introduce higher variance in the leader- leader-gradient update of eq. 7 relative to BodyGen base- line, confirming that the Stackelberg update remains stable across wide range of horizon lengths.
CONCLUSIONS
We introduced Stackelberg Proximal Policy Optimization (Stackelberg PPO), a reinforcement learning framework grounded in the Stackelberg game paradigm, which explicitly captures the leader-follower coupling between high-level design decisions and adaptive control responses. While this formulation is general, we instantiate it in the context of morphology-contre the leader specifies the body structure and the follower adapts the control policy. Instead of treating design and control as independent, Stackelberg PPO exploits the leader-follower coupling to antic- ipate how the follower will adapt, enabling the leader to update its policy toward morphologies that are more compatible with downstream control. Experiments demonstrate that this coupling yields superior performance and stability over standard PPO, particularly on complex locomotion tasks where tight coordination between morphology and control is essential.
co-design, where
Despite these promising results, several avenues remain for future work. A key direction is sim-to- real transfer, which remains challenging due to unmodeled hardware consiraints and material dy- namics. Bridging this gap could enable the real-world deployment of self-evolving robotic systems. We further envision advances in this area leading to truly adaptive artificial life forms capable of self-directed evolution, reshaping our understanding of intelligence, embodiment, and the boundary between designed and evolved systems.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
The demonstrations and open-source code are available at: https:/yamningdai.ibbe githuui.ggttb..... wwt.gggbboo.s..cceeeeeeeee io/stackelberg- ppo-co-design. The computational requirements, hyperparameters, and key implementation details are provided in Appendix D.
ACKNOWLEDGEMENTS
We gratefully acknowledge the insightful comments from the ICLR 2026 reviewers. The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940. This work additionally supported by the European Research Council (ERC, Advanced Grant Number 742870). For computer time, this research used lbex managed by the Supercomputing Core Laboratory at King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.
ETHICS STATEMENT
As fundamental Al research and the best of the authors' knowledge, there are no clear ethical risks associated with this work beyond the risks already posed by prior work.
REFERENCES
Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficient learning of stackelberg equilib- ria in general-sum games. Advances in Neural Information Processing Systems (NeurIPS), 34: 25799-258111 25811, 2021.
Dylan Banarse, Yoram Bachrach, Siqi Liu, Guy Lever, Nicolas Heess, Chrisantha Fernando, Push- meet Kohli, and Thore Graepel. The body is not a given: Joint agent policy learning and mor- phology evolution. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS'I9) pp. 1134-1142, 2019.
Tamer Başar and Geert Jan Olsder. Dynamic Noncooperative Game Theory. Society for Industrial and Applied Mathematics, 1998.
Josh Bongard, Victor Zykov, and Hod Lipson. Resilient machines through continuous self- -odelin.. Science, 31 314(5802): 11-1111111,111111 2006.
Runfa Chen, Ling Wang, Yu Du, Tianrui Xue, Fuchun Sun, Jianwei Zhang, and Wenbing Huang. Subequivariant reinforcement learning 3D multi-entity physical environments. arXiv preprint arXiv:2407.12505. 2024.
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of mor- phology and control in embodied machines. Journal of the Royal Society Interface, 15(143): 20170937, 2018.
Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy commit to. In Proceed- ings of the 7th ACM Conference on Electronic Commerce, pp. 82-90. 2006.
Heng Dong, Junyu Zhang, Tonghan Wang, and Chongjie Zhang. Symmetry-aware robot design with structured subgroups. In International Conference on Machine Learning (ICML), pp. 8334-8355. PMLR. 2023.
Matthias Gerstgrasser and David C Parkes. Oracles & followers: Stackelberg equilibria in deep multi-agent reinforcement learning. In International Conference on Muchine Learning (ICML), pp. 11213-11236, PMLR, 2023.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12(1):5721, 2021.

---

Published as a conference paper at ICLR 2026
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Kangyao Huang, Di Guo, Xinyu Zhang, Xiangyang Ji, and Huaping Liu. Competevo: Towards morphological evolution from competition. In Proceedings of the Thirty-Thing International Joint Conference on Arlificial Intelligence (IJCAT), 2024a.
Suning Huang, Boyuan Chen, Huazhe Xu, and Vincent Sitzmann. Dittogym: Learning to control soft shape-shifting robots. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024b.
Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems (NeurIPS), volume 14, 2001.
Hsu Kao, Chen- Yu Wei, and Vijay Subramanian. Decentralized cooperative reinforcement learning with hierarchical information structure. In International Conference on Algorithmic Learning Theory (ALT), pp. 573-605. PMLR, 2022.
Debarun Kar, Fei Fang, Francesco Delle Fave, Nicole Sintov, and Milind Tambe. "A game of thrones" when human behavior models compete in repeated Stackelberg security games. In Pro- ceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1381-1390, 2015.
Sam Kriegman, Douglas Blackiston, Michael Levin, and Josh Bongard. Scalable sim-to-real transfer of soft robot designs. In IEEE International Conference on Soft Robotics (RoboSoft), pp. 2187- 2200, 2020.
Henger Li, Wen Shen, and Zizhan Zheng, Spatial-temporal moving target defense: A markov stack- elberg game model. International Conference on Autonomous Agents and Multiagent Systems (AAMAS). AAMAS Conference proceedings, 2020.
Muhan Li, David Matthews, and Sam Kriegman. Reinforcement learning for freeform robot design. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 7799-8806. IEEE, 2024.
Chun Kai Ling, Zico Kolter, and Fei Fang. Funetion approximation for solving Stackelberg equi- librium in large perfect information games. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 5764-5772, 2023.
Hod Lipson and Jordan B Pollack. Automatic design and manufacture of robotic lifeforms. Nature, 406(6799):944-978 2000.
Huaping Liu, Di Guo, and Angelo Cangelosi. Embodied intelligence: A synergy of morphology, action, perception and learning. ACM Computing Surveys, 57(7):1-36. 2025.
Haofei Lu, Zhe Wu, Junliang Xing, Jianshu Li, Ruoyu Li, Zhe Li, and Yuanchun Shi. Bodygen: Advancing towards efficient embodiment co-design. In International Conference on Learning Representations (ICLR), 2025.
Rajesh K Mishra, Deepanshu Vasal, and Sriram Vishwanath. Model-free reinforcement learning for stochastic Stackelberg security games. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 348-353. IEEE, 2020.
M. F. Møller. Exact calculation of the product of the Hessian matrix of feed-forward network error functions and a vector in O(N) time. Technical Report PB-432, Computer Science Department, Aarhus University, Denmark, 1993.
Yunian Pan, Tao Li, Henger Li, Tianyi Xu, Zizhan Zheng, and Quanyan Zhu. A first order meta Stackelberg method for robust federated learning. In ICML Workshop on New Frontiers in Adver- sarial Machine Learning (AdvML- 23), 2023.
Chandana Paul. Morphological computation: A basis for the analysis of morphology and control requirements. Robotics and Autonomous Systems, 54(8):619-630, 2006.
12

---

Published as a conference paper at ICLR 2026
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6(1):147-160, 1994.
Charles Schaff and Matthew R Walter. N-limb: Neural limb optimization for efficient morphological design. arXiv preprint arXiv:2207.11773, 2022.
Jürgen Schmidhuber. On learning think: Algorithmic information theory for novel combina- tions of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09220 0.0999 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning (ICML), pp. 1889-1897. PMLR, 2015,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
Karl Sims. Evolving virtual creatures. In Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '94), pp. 15-22, 1994.
Richard S Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
Richard S Sutton, David A McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NewrIPS), volume 12, 1999.
Emanuel Todoroy, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5026-5033, 2012.
Bernhard Von Stengel and Shmuel Zamir. Leadership games with convex strategy sets. Games and Economic Behavior, 69(2):446 457, 2010.
Quoc-Liem Vu, Zane Alumbaugh, Ryan Ching, Quanchen Ding, Arna Mahajan, Benjamin Chas- nov, Sam Burden, and Lillian J Ratliff. Stackelberg policy gradient: Evaluating the performance of leaders and followers. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022.
Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. In International Conference on Learning Representations (ICLR), 2019.
Ronald J Williams. Simple statistical gradient-folloinng algorithms for connectionist reinforcmeet learning. Machine Learning. 8(3):229-256, 1992.
Zheng Xiong, Jacob Beck, and Shimon Whiteson. Universal morphology control via contex- tual modulation. In International Conference Machine Learning (ICML), pp. 38286-38300.
PMLR, 2023.
Boling Yang, Liyuan Zheng, Lillian Ratliff, Byron Boots, and Joshua R Smith. Stackelberg games for learning emergent behaviors during competitive autocurricula. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5501-5507, 2023.
Ye Yuan, Yuda Song, Zhengyi Luo. Wen Sun, and Kris M Kitani. Transform2Act: aaaesormm-andconnool policy for efficient agent design. In International Conference on Learning Representations (ICLR), 2022.
Learning a
Allan Zhao, Jie Xu, Mina and Wojciech Matusik. Rohogrammar: Graph grammar for terrain-optimized robot design. ACM Transactions Graphics (TOG), 39(6): 2020.
Josephine Hughes, Andrew Spielberg, Daniela Rus,
Liyuan Zheng, Tanner Fiez, Zane Alumbaugh, Benjamin Chasnov, and Lillian J Ratliff. Stackel- berg actor-critic critic: thne-thhcreeic reinforcement learning algorithmss In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 9217-9224, 2022.
13

---

Published as a conference paper at ICLR 2026
Han Zhong, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Can reinforcement learning find Stackelberg-Nee -ash equilibria in general-sum markov games with myopically rational followers? Journal of Machine Learning Research, 24(35):1 2023.
Nicolas Zucchet and Joao Sacramento. Beyond backpropagation: Bilevel optimization through implicit differentiation and equilibrium propagation. Neural Computation, 34(12):2339-2346, 2022.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We used LLMs for drafting and refining text extensively throughout the paper. LLMs were not used develop algorithms, provide theoretical results, run experiments. or contribute in any other way to the work beyond the aforementioned writing help.
B THEORETICAL ANALYSIS
In this section, we provide the theoretical foundations of our approach. We first present the trajectory factorization in the proposed phase-separated Stackelberg Markov Game, which serves as the basis for proving all the theorems.
Given the trajectory - (1st.af
21 T-1 az 12=0 24. for of ) ) the trajectory distribution naturally
factorizes into two phases:
P (T; (
=y²
$6
T-1
II 1=0
The (ab1of) pe (fftilof af) af) a² (ffi04) II TeP (oflof: P (chi aof, af 1=T
20
Proof of Theorem 1. Based on policy gradient theorem (Sutton, 1984; Williams, 1992; Sutton et al., 1999), we have
oF _ (0*,0%) P
Σ Vor log Thr (oflof
Af dr
Differentiating both sides with respect to 02 yields the cross derivative:
Vo JF (0',0F)) Vo (VonJ" (0L, 00))
Vol / p(T:0"
V or log F. (a 1s ;s) A!" dr
T
= / -VoOP(T::0 0F)
V ve log Top (af lof x4) Af
dr
t=
= =f | 00) Von log (7:0",00)
0
7
r log F (a (afls/; ; Al dr
t=
| 0F)
Σ oe log The (af1st))
8
oF log o (aflsf |sf s44 Af dr
1=
Vo or / p 0r)
Σ log The (afllot)
T
F log oer (a : A! dr
t=T
Evaluating this identity the reference parameters (0L, 0F) (00, 05) gives
15

---

Published as a conference paper at ICLR 2026
V go (0L,0F)
loc - 05,0F=05
Vo or / P(T:0%:00)
T-1 Tpp
(a/
# (af 1s :sf:sh 84)
A (sf,
at :fi dr
Tog
(afls))
Tog (af 1of : s4)
o0
T=
Voc / | ( 0%))
The
(a)
Meg (af
1=0
P (Tficci 0% )²¹)
A opp
(0, al i0)) dr
(a f s4)
" oor
Ve pr Σ
T-1
/
(TeeTi 0f 00%)
Téc (s
THe (aflst)
Σ
/
P (Tfrcci 0% 0% st)
p TeF (s s
A
oo (6 (FF dr
Tée (af lsf [s: s4)
Vgg or / dog (ot) (at1st) (st))
Tpr. (a+1)+)
Tog (1
I (s ; (a (a*1s" ;(p)
y Tée (a fs :44
A
(s aa :o; dr
Tog (a
|sF; s//
o0
-v or F (0t 0% 0/) le:=0 =
Proof of Proposition 1. The result follows directly by applying the likelihood-ratio trick in the same way the standard proof of the policy gradient theorem (Sutton, 1984; Williams, 1992; Sutton et al., 1999).
Proposition 2. We have
V2r 8 (04,0%)= = Vog
Exg, x
F. log oor (af 10 18f) Al"
I=T
| (II)
Var
Σ (=)
F log ohr
(of118;o7)) VØr
log oor F (a (oF1o0;; s4) Ar"
1=T
Proof of Proposition 2. Based policy gradient theorem (Sutton, 1984; Williams, 1992; Sutton et al., 1999), we have
Ver FF (04,85)= = f P(T:0L,85) Vor Σ log F
(afls. (f]ls::s))) Af
16

---

Published as a conference paper at ICLR 2026
Differentiating this expression again with respect to l", we obtain
Vig 00)
Vor / P ( Vor Σ log of 1e Af dr
ff p(T:0t, ,0) V0r log Tog (aflef 1of; ,04) Af
+fp(Ts04 0f) Vor log
Vor > log (of (oflefidt)) dr (1 (r,)
f P(T:0L, 00)) Vir Σ log (at (1]:))oo))) :04) afdr
+ - .0) (vor
log ffo (of (o(1(:3t)) > 1=T
(Tor) Σ t=T
log o (af115:04) Af) dr
17

---

Published as a conference paper at ICLR 2026
C ENVIRONMENT DETAILS
In this section, we provide comprehensive details about the nine environments employed in our experimental evaluation. To ensure fair comparison with existing methods, we adopt all 6 en- vironments from the BodyGen framework: Crawler, Cheetah, Glider, Walker, Swimmer, and TerrainCrosser. Additionally, we introduce three novel environments designed evaluate differ- ent aspects of our algorithm: Stepper-Regule structures to test robustness in challenging terrain, while Pusher evaluates manipulation capabilii ties. These additional environments are specifically crafted to assess the robustness and adaptability of our algorithm under more challenging conditions, thereby providing more rigorous assessment of the proposed method's is capabiliitie. Figure 6 provides visualizations of all nine environments.
and Stepper-Hard feature complex topographical
Each agent undergoes dynamic morphological evolution through topological and attribute modifi- cations during training. The observation includes the root body's spatial position and velocity, all joints' angular positions and velocities, and motor gear parameters. All joints use hinge connections enabling single-axis rotation. Joint attributes encompass bone vectors, sizes, and motor gear values. The action space consists of one-dimensional control signals applied to each joint's 's motor.
Crawler operates in 3D environment, where agents exhibit quadrupedal crawling locomotion. The initial morphology consists of a central root node with four limb branches extending outward. The maximum depth of levels, with each non-root node supporting at most 2 child limbs. Episodes are terminated when the agent's body height exceeds 2.0 units to prevent unrealistic vertical extensions. The reward function encourages forward movement while penalizing excessive control effort:
body tree is constrained
It+1
Dt
W
Σ JEZ.
(12)
where 2, denotes the agent's forward position at timestep t, uf represents the effective control input applied to joint (i.e., the raw action scaled by the joint's gear ratio) applied to joint j at time t, w = 0.0001 is the control regularization coefficient, N is the total number of joints, and r = 0.04.
Cheetah features 2D locomotion focused on fast rumning gaits. The agent begins with initial design comprising a root body connected to one primary limb segment. The morphological search allows maximum tree depth of 4 with up 3 child limbs per node. The root body's 's angular orientation is constrained within 20 degrees to maintain stable running posture. Episode termination occurs when body height falls below 0.7 or exceeds 2.0 units. The reward follows the velocity-based formulation:
Tt Ti+1 Ze
(13)
where T = 0.008.
Glider and Walker enable 2D aerial and terrestrial locomotion. Both environments share the same base morphology: agents start from an initial configuration with three limb segments attached to a central root, each limb node support up three children, and joints can oscillate within a 60 range to accommodate wide-range motion. The reward structure follows eq. 13, emphasizing for- ward displacement. The two tasks differ only in their allowable morphology depth: Glider restricts the body tree to a maximum depth of 3, while Walker permits up to 4 levels.
Swimmer enables undulatory, snake-like locomotion in 2D aquatic environment. The agent evolves in water with a viscosity coefficient of vis 0.1. The initial morphology consists of a root body connected segments, enabling flexible articulated structures suited for wave-based propulsion. This task serves as a lightweight validation environment, and thus imposes no carly-termination conditions such as height limits or joint-rotation thresholds. The reward structure follows eq. 13, emphasizing forward displacement under hydrodynamic resistance.
single limb segment, and each limb node may support up three child
18

---

Published as a conference paper at ICLR 2026
TersainCrosser presents challenging 2D terrain navigation task using the Cheetah agent figuration. The environment features fixed terrain heights with maximum elevation differences of Zmar 0.5. Agents must traverse gaps generated from single-channel height maps. Height con- straints maintain agent stability between 0.7 and 2.0 units, with violations leading to episode termi- nation. The reward function prioritizes forward progress defined in eq. 13.
Stepper-Regular and Stepper-Hard introduce challenging staircase navigation tasks that test agents' morphological adaptation capabilities for vertical terrain traversal. Both environments uti- lize the Crawler agent configuration in 3D setting. Stepper-Regular features stairs with step width of 1.0 units and height of 0.4 units; Stepper-Har to 0.8 units while maintaining the same width. Unlike the standard Crawler environment, height termination constraints are removed to allow full exploration of vertical climbing capabilities. The reward function follows eq. 12, focusing solely on forward progression, thereby maintaining reward consistency across environments.
increases the difficulty by elevating step height
Pusher is a challenging 3D manipulation task designed to evaluate whether the co-design system can generate morphologies and control strategies that effectively interact with external objects. This environment reuses the Crawler agent configuration in a 3D setting. A rigid cube of side length 1.0 m is placed in front of the agent and constrained to move in the horizontal (zt y) plane. The observation space augments the agent state with the 3D relative position between the agent's root body and the cube. The reward encourages forward displacement of the cube, penalizes lateral motion, and provides auxiliary shaping term based the proximity between the agent and the cube. A control-effort penalty identical to eq. 12 is applied. Formally, the reward is
cueh cuche
9i+1
yr
cabe
p
cube
P ToDE
17
N
Σ JEJ.
(14)
where wtube and yi denote the cube's forward and lateral positions at timestep t, pfu and pt are the 3D positions of the cube and the agent's root body, and 0.1 controls the penalty.
19

---

Published as a conference paper at ICLR 2026
Crawler
Cheetah
3D
2D Plane
Swimmer
Walker
2D xy Plane
2D Plane
Glider
TerrainCrosser
2D Plane
2D Plane
Stepper-Regular
Stepper-Hard
3D
3D
Pusher
3D
Figure 6: Visualization of the nine benchmark environments used in our experiments. Crawler, Stepper-Regulat Stepper-Hard, and Pusher are 3D tasks; others are 2D (x-z or x-y plane). The environments differ substantially in required morphology depth, symmetry, and limb arrangement, enabling evaluation on the generality morphology-contro co-design across locomotion and ma- nipulation tasks.
20

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATION DETAILS
D.1 COMPUTATION COST
Following standard reinforcement learning practices, we utilize distributed trajectory sampling across multiple CPU threads to enhance training efficiency. All models are trained with seven ran- dom seeds on nigh-peeffrmmance computing cluster equipped with dual Intel Xeon@ processors (totaling 64 cores) and 24 NVIDIA A100 GPUs. Our implementation uses PyTorch 2.0. for all neu- ral network models and MuJoCo 2.1.0 (Todorov et al., 2012) physics engine for the morphology- control simulation environments. The training process is computationally efficient, requiring ap- proximately 30 hours per model when utilizing 10 CPU cores alongside a single NVIDIA A100 GPU across all experimental environments.
D.2 HYPERPARAMETER CONFIGURATION
Stackelberg PPO (Ours): Our method introduces several Stackelberg-specinn hyperparameters that require careful tuning. We conduct grid search over key parameters: Fisher information ma- trix regularization coefficient À (0.5, 1.0, 5.0, 10.0). maximum conjugate gradient (CG) steps {10, 20, 30), and follower sampling steps per episode E {6, 15, 30, 60, 100} during leader up- date. For the underlying network architecture, we maintain the same configuration as BodyGen (Lu et al., 2025) without modification to ensure fair comparison, including their MoSAT transformer blocks and all network-related parameters. The final hyperparameter configuration, along with the underlying BodyGen network architecture we adopt, is detailed in Table
BodyGen: We follow their original implementation and released code, adopting the same hyper- parameter configuration as reported in their work (Lu et al., 2025). The settings include MoSAT Pre-LN normalization, SiLu activation, hidden dimension 64, policy learning rate 5e-5, value learn- ing rate 3e-4, and other parameters as detailed in Table 1.
Transform2Act:
Following the original implementation Yuan et al., 2022), this baseline uses GraphConv layers, policy GNN size (64, 64, 64), policy learning rate 5e-5, value GNN size (64, 64, 64), value learning rate 3e-4, JSMLP activation Tanh, JSMLP size (128, 128, 128) for policy networks, and MLP size (512, 256) for value functions.
Based on the original implementations (Wang al., 2019), this evolutionary baseline uses 125 generations, population size 20, elimination rate 0.15, with GraphConv layers, Tanh activation, policy GNN size (64, 64, 64), policy MLP size (128, 128), value GNN size (64, 64, 64), value MLP size (512, 256), policy learning rate 5e-5, and value learning rate 3e-4.
NGE:
D.3 GRADIENT NORMALIZATION
Recall Eq. 2, where the Stackelberg gradient for the leader decomposes into a direct term and a response-induced term. To avoid scale imbalance between these components, scale the response- induced term by a data-dependent factor a computed from the relative norms of the two terms (no extra pyramt-arameters). Let Gdir We update the leader using
Vo, JL( 0f) and Gresp (Voz 0. (0r.)) V Vag J (Or,, 0f).
min (1,
| 1/2
V 0r JL
Adir
Gresp:
Fresp ||2
(15)
where & > is small numerical constant for stability. This rule guarantees || 1l2 ensuring the follower-implicit component never dominates while preserving its direction. across all experiments for simplicity and consistency.
l1gair 1|2. We use
21

---

Published as a conference paper at ICLR 2026
Table I: Hyperparameters of Stackelberg PPO adopted in all experiments
Hyperparameter
Value
Fisher Regularization Coefficient X Maximum Conjugate Gradient Steps
5.0
20
CG Relative Error Tolerance
10-3
Follower Sampling Steps per Episode
6
Gradient Normalization Ratio
1.0
Structure Design Steps T
5
Attribute Design Steps Tattr
1
Transformer Layer Normalization
Pre-LN
Transformer Activation Function
SiLu
FNN Scaling Ratio
4
Transformer Blocks number (Policy Network)
3
Transformer Blocks number (Value Network)
3
Transformer Hidden Dimension (Policy Network)
64
Transformer Hidden Dimension (Value Network)
64
Optimizer
Adam
Policy Learning Rate
5c-5
Value Learning Rate
3c-4
Clip Gradient Norm
40.0
PPO Clip
0.2
PPO Batch Size
50000
PPO Minibatch Size
2048
PPO Iterations Per Batch
10
Training Epochs
1000
Discount factor ?
0.995
GAE Parameter AGAE
0.95
22

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
E VISUALIZATION AND QUALITATIVE RESULTS
Figure presents the diverse morphologies discovered by our Stackelberg PPO framework across different environments. The evolved body designs reveal the sophisticated structural complexity achieved by our approach, confirming that the Stackelberg game formulation enables continuous co- adaptation between morphology and control without premature convergence to suboptimal simple structures. Remarkably, these designs demonstrate emergent functional differentiation, developing specialized appendages for complementary tasks such maintaining equilibrium versus providing propulsive forces.
As illustrated in the training curves presented in Fig. provide quantitative performance compar- isons across all evaluated environments. Table 2 summarizes the final episode rewards achieved by each method, presenting All baseline methods are configured using their optimal hyperparameter settings as reported in prior literature, with detailed specifications provided in Appendix D.2.
values and standard deviations computed over seven random seeds.
Table Performance comparison of Stackelberg PPO against baseline methods across morphology- control co-design environments. Results show mean episode rewards and standard deviations over seven random seeds.
Methods
Crawler
Cheetah
Swimmer
Stackelberg PPO (Ours)
11047.90
13514.94
1334.98
BodyGen (Lu al., 2025)
9098.721s58.26
11575.87 1840005
23.771
Transform2Act (Yuan al., 2022)
23
6297.90+
737. 737.900g21.04
NGE (Wang et al., 2019)
1482.45go24.0
2534.76. 20341.664422..8
384.45
ESS (Sims, 1994)
631.67 122.441
671.67
190.62
Methods
Walker-Hard
Glider-Hard
TerseinCrosser
Stackelberg PPO (Ours)
13612.32
12414.50+4a0
4488.07
BodyGen (Lu et al., 2025)
11645.89
11049. 11049.95
1103.25.s71..
Transform2Aet (Yuan et al., 2022)
4420.63
6120.62+
236-4.63:4475.00
NGE (Wang et al., 2019)
1504.55155a.15
2081.20 2081.25
427.15
ESS (Sims, 1994)
636.03
541.55
426.811166.30
Methods
Pusher
Hlepper-Regular
Stepper-Hard
Stackelberg PPO (Ours)
34622.7
7215.20+440.00
6003.59
BodyGen (Lu al., 2025)
2779.95
4685.944845.22
4685.414800.0p
Transform2Act (Yuan al., 2022)
1015:28
2325.694664.00
1192.394.54.22
NGH (Wang et al., 2019)
551.57 120.66
870.561215.66
509. 109.772120.00
ESS (Sims, 1994)
243. 243.144000 14 93
351.02+ 28
392.54
23

---

Published as a conference paper at ICLR 2026
Crawler
Cheetah
Swimmer
Walker
AF
Glider
XL
Pusher
TerrainCrosser
Stepper-Regular
Stepper-Hard
Figure Visualization of co-evolved body designs generated through our Stackelberg PPO.
24

---

Published as a conference paper at ICLR 2026
E.2 EXTENDED ENVIRONMENT EVALUATION
Beyond the environments reported in the main paper, we also include results for the full sets of Glider and Walker tasks, each provided in three difficulty levels: regular, medium, and hard. In the main text we present only the hard variants, as they offer the largest design spaces and naturally encompass the casier tiers, providing clearer view of morphology structural budgets. Here, we report the complete results mainly for completeness, and to illustrate how our method behaves under different morphology complexity limits. The six environments differ only in their structural allowances: Glider uses maximum tree depth of 3 and Walker a maximum depth of 4, while the regular/medium/nna variants correspond to maximum child counts of (1, 2,3). All other environment settings are identical.
co-design under less restrictive
Figure presents the training curves and the final generated morphologies across all six tasks. Our method outperforms the baseline across all difficulty levels, with the performance gap increasing as the design space becomes larger and more challenging. Interestingly, the three difficulty tiers within each environment achieve similar final performance, suggesting that overall task success is not strictly tied to structural complexity: even simpler configurations can discover diverse, correct, and high-quality locomotion patterns.
E.3 ADDITIONAL ABLATION STUDIES AND MECHANISM ANALYSIS
Quantitative Results for PPO Clipping Sensitivity. To complement the qualitative trends shown in Figure 5(a), we provide the full quantitative statistics for the clipping sweep experiment. The purpose of this analysis is to examine how the surrogate objective behaves under different clipping thresholds and to identify when the underlying assumptions of policy-gradien theory remain valid. From theoretical perspective. large policy updates cause the surrogate objective diverge from the true return, leading to instability. PPO addresses this by bounding the likelihood ratio aa(a s) Ta, (a s) within [1 c, c]. which prevents overly aggressive updates and ensures that the surrogate remains a reliable approximation. In this experiment, we vary the clipping parameter and measure three quantities that together characterize the stability of the update rule: (1) average performance, (2) likelihood-ratii constraint violations, and (3) KL divergence. Table 3 reports the full numerical results corresponding to the curves shown in the main text.
Table Sensitivity of Stackelberg PPO to the clipping threshold e.
Clipping Parameter
Performance
Likelihood Ratio Violations (%)
Average KL Divergence
0.1
4934.522e46.w6
14.82
0.0030+0.00m
0.2
7215.20
13.39 +0.83
01.01644..000
0.4
7907.0 0108.00.000
9.92
0,03433 20.055
0.6
4778. 18 84
1.10
0.0665=.0.188
= 0.8
2656.92 503.99
7.02
00.088
No Clipping
1233.2644448.
0
1.772640.1588
Ablation on SID Components and PPO Clipping. To further disentangle the contributions of our Stackelberg Implicit Differentiation (SID) estimator and PPO clipping, we conduct an additional controlled ablation. Specifically, we evaluate three variants under the same phase-separated, non- differentiable Stackelberg setup:
SID+PPO (full) - our complete method using both SID and PPO clipping,
PPO-only - standard PPO updates without SID,
SID-only - applying our SID estimator without PPO clipping.
This ablation assesses whether (i) our SID estimator meaningfully improves leader optimization and (ii) PPO clipping is required to stabilize the induced surrogate objectives. As shown in Table 4, both components provide clear performance gains, and the full algorithm consistently achieves the highest returns across four environments.
25

---

Published as a conference paper at ICLR 2026
Stackelberg PPO (ours)
BodyGen
Transform2Act
NGE
ESS
Walker-Regular
Walker-Min Mediiu
Walker-Hard
15000
15000
12000
12000
12000
Reward
9000
Reward
90000
Reward
9000
6000
6000
6000
0M
10M
20M
30M
40M
50M
0M
10M
20M
30M
40M
50M
0M
10M
20M
30M
40M
SUM
Step
Step
Step
Glider- Reguuar
Glider-Medium
Glider-Hard
12000
12000
12000
Reward
9000
Reward
9000
Reward
9000
6000
60000
6000
3000
0M
10M
20M
30M
40M
50M
0M
10M
20M
30M Step
40M
50M
0M
10M
20M
30M
4UM
S0M
Step
Step
Ralker-Regular
Walker-Medium
Walker-Hard
Max Depth:
Max Depth:
Max Depth: 4
Max Children Per Node:
Max Children Per Node:
Max Children Per Node:
Glider- Regular
Glider-Medium
Glider-Hard
Max Depth:
Max Depth: 3
Max Depth: 3
Max Children Per Node:
Max Children Per Node:
Max Children Per Node: 3
Figure 8: Extended evaluation on Glider and Walker environments under different morphology com- plexity (ad Training curves for the regular, medium, and hard variants of each environment. (b) Final generated morphologies under each complexity tier.
26

---

Published as a conference paper at ICLR 2026
Crawler
5%
20%
40%
60%
80%
100%
Ours
Body Gen
Walker
40%
100%
Ours
Gen
Stepper-Reguha
5%
10%
20%
40%
60%
80%
100%
Ours
BedyGen
Pusher
5%
10%
20%
40%
60%
80%
100%
Ours
Body
Figure 9; Comparison of morphology evolution between Stackelberg PPO (ours) and Standard PPO (BodyGen) across four environments, BodyGen tends to collapse early into low-complexity designs, while Stackelberg PPO continues exploring structurally richer morphologies, yielding more capable final structures.
27

---

Published as a conference paper at ICLR 2026
Table 4: Ablation studies on the components of our SID estimator and PPO clipping, evaluated under the same phase-separated, non-differentiable Stackelberg setting.
Environment
SID+PPO (full)
PPO-only (no SID)
SID-only (no clipping)
Stepper-Regular Regular
Stepper-
7215.20 4499.02
4685.94 1845.23
1257 .33 1500.25
Crawler
11047.90=120 20
7098.722338.1 26
35.77+12.28
Cheetah
13514.94 153.62
11575.87 1640.65
472.89 472.89177.40 1.7.40
Glider
12414.50+498. 53
11049.95=200
566.81 466.81+899
Effeet of Leader Gradients on Controller Adaptation. To better understand the mechanism be- hind Stackelberg PPO's performance gains, we analyze how morphology updates interact with con- troller adaptation. Specifically, we investigate whether the improved performance originates from faster controller adaptation under changing morphologies, from more informative leader gradients that guide the structure search more effectively. To isolate these effects, we extract ten intermediate checkpoints from BodyGen training run (spanning 10%-100% of training progress). From each checkpoint, we initialize both methods with identical morphology, controller parameters, and opti- mizer state, and then train each method for single epoch. This setup ensures that any difference in performance improvement reflects differences in the leader update rule, rather than controller initialization or long-term training.
As shown in Table 5, Stackelberg PPO consistently achieves a larger one-epoch performance im- provement compared to standard PPO (BodyGen). This indicates that Stackelberg PPO does not rely on faster controller adaptation; instead, it provides more informative leader gradients that enable the morphology to improve even when the controller is only partially adapted. These results high- light the role of the Stackelberg update in stabilizing and accelerating the joint corphology-conter optimization process.
Table 5: Average performance change after one epoch of training from the same checkpoint model, averaged over 10 checkpoints and seeds, evaluated on Regula..
Stackelberg PPO (Ours)
BodyGen (PPO)
Performance Change After Epech
+0.39220.075: %
00.224
We further provide visual comparison of morphology evolution to illustrate this effect (Figure 9). Across multiple environments, BodyGen tends to converge carly to low-complexity designs, which restricts later improvements continues meaningful structural exploration throughout training, enabling richer and more adaptive morphologies. These qualitative trajectories align with the adaptation results above, reinforcing that the Stackelberg update produces more informative and better-aligned structural gradients.
the controller becomes stronger. In contrast, Stackelberg PPO
28

---

Published as a conference paper at ICLR 2026
E.4 SAMPLE AND TRAINING EFFICIENCY
Sample Efficiency. To assess the efficiency of different co-design algorithms, we measure how many environment interaction samples required to reach predefined performance threshold. As reported in Table 6, Stackelberg PPO consistently converges with substantially fewer samples across all environments. On average, reaches the threshold with approximately -39% fewer samples than BodyGen. In contrast, Transform2Act, NGE, and ESS fail to reach any threshold within the available training budget. These results highlight the advantage of explicitly modeling morphology- control coupling via a Stackelberg formulation, enabling faster convergence and more stable co- design dynamics.
Table 6: Sample efficiency comparison: number of samples (in millions) required to reach the performance threshold.
Environment
Threshold
Stackelberg PPO
BodyGen
Transform2Act
NGE
ESS
Crawler
9000
25.8
47.2
Cheetah
1 11000
19.2
42.1
00
00
00
Swimmer
1200
14.8
17.0
00
00
Walker-Hard
10000
18.1
30.3
00
Gaider-Har
11000
23.6
49.7
00
0
00
TerrainCrosser
3500
23.9
33.8
Pusher
2500
29.3
39.1
00
00
00
Stepper-Regular
4500
18.5
40.4
Stepper-Hard
4500
27.2
43.1
00 00 00
Training Efficiency. Despite incorporating bilevel update, Stackelberg PPO introduces only modest computational overhead. The method avoids explicit Hessian construction or inversion; in- stead, the sagpluggategrddiiin step relies solely efficient Hessian-vector vector products (upproximately one backward pass). As result, its cost scales linearly with morphology and controller dimension- ality, rather than quadratically. Moreover, rollout collection dominates overall computation in all co-design settings, so the additional optimizatinn cost has limited influence on total training time. Table 7 summarizes the training time under different morphology/control design spaces. Increasing the structural search space does not incur superlinear overhead, confirming the scalability of Stack- elberg PPO. The comparison with ES-based approaches in Table further shows that ES reduces wall-clock time only when substantial CPU parallelization is available, while its resulting designs remain far less effective than those produced by PPO-based methods.
Overall, our method achieves strong efficiency-peeformmacc trade-offs:
Compared to BodyGen, Stackelberg PPO achieves substantially better sample efficiency by requiring -39% fewer samples to reach the performance threshold while also obtaining +20.66% higher final scores. In terms of wall-clock time, the difference between the two methods is modest (+13%), keeping the overall training cost comparable.
Compared to ES-based baselines, although ESS attains shorter wall-clock time using 6x more CPU cores (64 cores), its performance is extremely poor, achieving only 0.16 frac- tion of our method's performance.
29

---

Published as a conference paper at ICLR 2026
Table Wall-clock training time comparison across environments with different design space sizes (10 CPU cores + A100 GPU).
Environment
Space Size (mean)
Space Size (max)
Stackelberg PPO
BodyGen (PPO)
TersainCrosser
10.7
14
33.8810m 10.42
27.87
Swimmer
5.50+0.76
14
30.64
28. 28.13+0.45
Cheetah
6.57 10.90
32.9610.67
29.52
Glider-Ham
7.33
32. 32.93+0.71
28.9341.50
Walker-Hard
8.43
27
32.54 10.62
30.21 30.21 11.22
Stepper-Hare
0.57
29
32. 22.001
30.25+2.06
Pusher
14.33 14.33
29
33.4 10.82
29.24 11.12
Stepper-Regular
16.40444.69
32.83
30.17+1411
Crawler
18.25
29
33.73
30.54
Table 8: Wall-clock training time across methods. NGE results are shown under both 10 CPU cores and cores to illustrate parallelization effects.
Stackelberg PPO (10 cores)
BodyGen (10 cores)
NGE (10 cores)
NGE (64 cores)
Wall-clock Time
33.07+0.49
29.48
45. 45.1643.72
13.52+1.52
E.5
MORPHOLOGY EVOLUTION PROCESS VISUALIZATION
Figure 10 showcases the morphological evolution trajectories discovered by Stackelberg PPO framework across diverse locomotion tasks and environments. Each row represents a distinct em- bodiment (Crawler, Cheetah, Swimmer, Glider, Stepper-Regular, Stepper-Hard, Terrain Crosser, Walker, and Pusher), and the columns depict the progressive morphological changes from early evo- lution (5%) through convergence (100%). The evolution demonstrates emergent specialization of appendages for task-specific locomotion requirements.
30

---

Published as a conference paper at ICLR 2026
Crawler
5%
10%
20%
40%
641%
80%
100%
Cheetah
5%
10%
20%
40%
60%
80%
100%
Swimmer
5%
10%
20%
40%
60%
80%
100%
Walker
20%
60%
80%
1.00
Glider
5%
10%
20%
40%
60%
80%
100%
Terrain Crosser
1000
5%
10%
20%
40%
60%
80%
Reeppar-Regular
60%
100%
Stepper-llard
100%
5%
10%
20%
40%
60%
80%
Pusher
5%
10%
20%
40%
60%
80%
100%
Figure 10: Morphological evolution trajectories across eight environments. Each row represents distinct robot embodiment, with columns showing progressive stages of morphological adaptation from 5% 100% training progress.
31

---

Published as a conference paper at ICLR 2026
E.6 RESULTS UNDER REALISTIC CO-DESIGN CONSTRAINTS
In the main paper, we adopt a unified forward-progret reward to ensure fair comparison across algo- rithms and to avoid introducing task-specific reward biases. While this setup standard and suitable for benchmarking algorithmic contributions, real-world robot design is often shaped by additional engineering constraints. To better understand the practical co-design behavior of Stackelberg PPO, we further evaluate five common realistic constraints under an identical crawler task and training budget.
These constraints span both morphology- and control-level considerations, including power usage, manufacturability, torque limits, payload handling, and robustness. Several of these factors are already captured by our experimental setup:
Power usage: Energy expenditure is discouraged through small effort penalty included in the reward (Equation eq. (12)).
Torque limits: Joint torque capacity is implicitly limited by bounding the "allowable torque" attribute during morphology design.
Manufacturability: Physical realizability is enforced by constraining morphology-cditing attributes such as limb length, joint count, and topology depth (see Appendix C).
Robustness: Robustness naturally emerges from the evaluation protocol: each morphology-conntollee pair is scored using multiple rollouts, causing non-robust designs yield lower averaged returns.
To complement these built-in constraints, we further provide more detailed quantitative experiments that isolate and measure their individual effects.
Power Constraint. We evaluate performance under various power penalty coefficients (0.001, 0.01, 0.1), extending beyond the mild penalty (0.0001) used in the main experiments. Table 9 reports the detailed performance and control-effort statisties under each penalty coefficient. The generated morphologies are visualized in Figure 11. Increasing the penalty produces three consistent effects:
Impact Performance (velocity reward). As the penalty coefficient increases, both meth- ods experience reduced
reward. However, Stackelberg PPO exhibits a substantially smaller degradation, maintaining stronger performance across all tested set- tings.
Impact on Control Effort. Larger penalties encourage more conservative actuation strate- gies for both approaches, reflected by the lower penalty terms in the table.
Impact on Morphology-Control Control Co-Design. With stronger penalties, the optimized mor- phologies tend to adopt shorter, thicker, and more symmetric limbs, paired with low-torque gaits characteristic of energy-efficient locomotion.
Penalty Coef. = 0.0001
Penalty Coef. - 0.1
Figure 11: Power constraint under different penalty coefficients. As the penalty increases, the co- designed morphologies transition toward shorter, thicker, and more symmetric limbs.
32

---

Published as a conference paper at ICLR 2026
Table 9: Power-constraint setting: performance and power penalties under different penalty coeffi-
cients.
Penalty Coef.
Performance
Power Penalty
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
0.0001
11047.900 10047.90.1200 122.02 20 10191.15 1371.81
9098.72 4358.20
5631.42 2221442
2745.8 2
0.001
7501.6 661.16111611..0 1671 .33
5342.16=684
5948. 164254.84
0.01
9853. 19 19..1994222... 4:299.37
8304100 4-497566
1582.48 6697.
168.122516.33
0.1
10585. 25 80
897/L.18+574.22
25.50+22..77
26.31=23.66
Manufacturability Constraint. A manufacturability cost penalty is applied by incorporating two components into the leader objective: structural complexity is measured by the number of body ele- ments, and material cost is defined as the total mass. Table 10 summarizes the resulting performance and morphology characteristics under different penalty coefficients. The trends are consistent with those observed in the power constraint experiments in (i): our method consistently achieves better reward cost tradeoffs across all penalty levels. As shown in Figure 12, the generated morphologies are compact than the original structure, with fewer distal branches, shorter limbs, and mass con- centrated near root. These structures exhibit lower inertia and more efficient force transmission, supporting stable forward locomotion under cost constraints.
Penalty Coef. =
Penalty Coef. = (15 Bodies / 1.56 kg)
Penalty Coef. = 10 (8 Bodies 0.99 kg)
(18 Bodies 1.73 kg)
Figure 12: Manufacturability constraint under different penalty coefficients. Higher penalties on structure complexity and material mass encourage designs with fewer body elements, reduced branching, and mass concentrated near the root, producing compact morphologies that are casier to fabricate.
Table 10: Manufacturability constraint setting: performance, morphology complexity, and material cost under different penalty levels.
Penalty Coef.
Performance
Morphology Complexity
Material Cost
Stackelberg PPO (ours)
Stackelberg PPO (ours)
Stackelberg PPO (ours)
BodyGen
BodyGen
BodyGen
11047. 11047.90+124 120
9098.77 1098.724410 725358822 26
16. 16.40.2.45
13.6742.08
23
1.57
7892.93,C 349.84
6531.37 +437.26
13.67=2.08
9.6741..1
1.59
1.32
10
6825.44 47 330.009
5372.10 1
8.25
7.50
0.94
0.93
33

---

Published as a conference paper at ICLR 2026
Torque Limits Constraint. A lorque-limit penalty is incorporated by enforcing 50 N-m cap all joints and adding a proportional violation cost to the leader objective. Table 11 summarizes the quantitative results, and the morphological effects are shown in Figure 13. As in the manufac- turability and control-effort settings, our method achieves stronger reward-cost tradeoffs when the controller retains sufficient expressiveness (penalty 0.01). Under the stronger penalty (0.1), the tightened actuation constraints reduce the feasible morphology space for all methods, narrowing the performance gap.
Penalty Coef. =
Penalty Caef. = 0.01
Penalty Coef. 0.1
Figure 13: Torque limits constraint under different penalty coefficients. Tighter actuation limits lead to noticeably simpler and more compact structures, with shorter limbs and reduced distal branching.
Table 11: Torque limits constraint: performance and torque- violation penalties under different torque-penalty coefficients.
Penalty Coef.
Performance
Limit Violation Penalty
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
0.01
7893.42
6311.75199.
20210.50, 20210.501esos.2
11350.15
0.1
3133.01 +70.44
3121.800 1121.80
1106.454or
899.35
Payload Constraint. To evaluate the agent's ability to maintain locomotion under additional load, we attach an extra mass to the root link to serve as a payload. During training, the payload value randomized within fixed range (0-0.6 kg) to promote generalization. After training, evalu- ate each method under three fixed payload levels (0.2 kg. 0.4 kg. 0.6 kg). As shown in Table 12, Stackelberg PPO consistently maintains higher forward progress across all payload settings. Figure 14 further compares morphologies trained with and without payload. Under load, the evolved struc- tures become more symmetric and better support the additional mass, indicating that Stackelberg PPO adapts the topology itself rather than relying solely on controller compensation.
Without Payload
With Payload MMrphology 1
With Payload MMrrpholgyy 2
Figure 14: Morphology comparison trained with and without payload. Payload induces more sym- metric and load-suppo structures.
34

---

Published as a conference paper at ICLR 2026
Table 12: Payload constraint: performance comparison under different payload weights.
Payload Weight
Stackelberg PPO (ours)
BodyGen
0.2 kg
8675 4
5186.311650.47
0.4 kg
6966.34 473.44 7347.46; 44.77...14788.1 1523.894536.99
5116.544540.55
0.6 kg
Robustness Evaluation. We evaluate robustness under two settings: random external forces ap- plied to the root body at every control step, and terraín friction noise created by randomly varying the ground's friction in each episode. For each disturbance level, all policies are tested across mul- tiple stochastic rollouts, and we report the resulting forward progress reward. Tables 13 and 14 summarize the results. Across all disturbance magnitudes, Stackelberg PPO consistently demon- strates substantially higher robustness. For example, when external forces increase from N to N, performance decreases by only 5.91% for Stackelberg PPO, compared to much larger 59.57% de- cline for BodyGen. A similar pattern holds under terrain friction noise. These improvements arise primarily from more symmetric, mechanically balanced morphologies that better tolerate external forces and friction variability.
forward-progress
Table 13: Robustness evaluation: performance under different levels of external disturbance forces.
Level
Stackelberg PPO (ours)
BodyGen
2.0
11557.B 124.66
6963. 6963.05.4450 4450
4.0 N
11290. 13 1
(621.82+597.77
6.0
10875. 23 2500.97
2816. 161.6.1666 11166.118877
Table 14: Robustness evaluation: performance under different levels of terrain friction noise.
Level
Stackelberg PPO (ours)
BodyGen
30%
11424. 66
2226.044421..7
50%
11333.C 11333.43 431144442
6795.5544900
70%
10892.0941484
5062.85
35

---

Published as a conference paper at ICLR 2026
E.7 DISCUSSION AND EXTENDED EVALUATION ON REALISTIC CO-DESIGN CHALLENGES
In this section, we present broader analyses of morphology-com coohorl co-design and extend our re- sults along four representative challenge dimensions: (i) diverse co-design environments. (ii) multi- objective and role-specific rewards, (iii) robustness and generalization under unseen disturbances, and (iv) the use of morphology priors. These studies highlight both the empirical advantages of Stackelberg PPO and the conceptual benefits of explicitly decoupling structure design from control learning. Together, they demonstrate that our framework scales naturally to more complex co-design settings that better reflect real-world robotic demands, and they point toward promising directions for building more adaptive and physically grounded worphology-contre systems.
Diverse co-design environments. Standard co-design benchmarks focus almost exclusively on flat- terrain locomotion, which poses limited structural or behavioral challenge. To expose a broader range of morphology-contro interactions, we introduce more demanding environments-moos no- tably difficult terrain and manipulation-thaa require non-periodic motions, contact management, and functional differentiation across limbs. In the Stepper environments, agents must coordinate structure and control to handle large discontinuities without exteroceptive sensing. On low stairs, they develop stable stepping and small hops; on high stairs, the difficulty induces long-range, high- amplitude jumping behaviors. These emergent solutions reflect the stronger morphological and dy- namical adaptation required by complex terrain. the pusher task, co-design must jointly support locomotion and precise force application. Learned morphologies exhibit clear role specialization: some limbs provide acceleration and stability, while others regulate contact orientation and apply controlled pushing forces. Baseline methods typically recover only the locomotion component, lying on collision-bass propulsion. These environments reveal aspects of the co-design problem that flat locomotion cannot capture, and they demonstrate that Stackelberg PPO scales richer settings requiring terrain adaptation, contact reasoning, and multi-role morphology design.
Multi-objective and role-specific reward design. As shown earlier in Appendix E.6, our frame- work naturally accommodates additional objectives such as power consumption or payload capac- ity. The resulting morphologies and controllers smoothly adapt the trade-off's introduced by these objectives, validating the method's ' multi- nulti-objective co-design capability. Furthermore, the leader-follower decomposition allows reward terms be assigned selectively to the structure-desie or control-learning stages. For example, complexity or material-cost penalties can be applied only to the leader (structure) updates, enabling constraints on morphology without interfering with con- troller learning. This role-specific reward routing provides a high degree of flexibility for real-world design requirements.
Robustness and generalization under unseen disturbances Although our current setting does not include exteroceptive sensing and is not intended for zero-shot transfer to arbitrary unseen worlds, evaluate generalization and robustness under obstacle- naviiatinn task not seen during training. Policies are trained only on flat terrain (Crawler task) and then tested in environments containing either sparse or dense grids of square obstacles. As reported in Table 15, Stackelberg PPO obtains higher forward progress than BodyGen across both difficulty levels. The visualization in Figure 15 further shows that the morphologies produced by method maintain motion, whereas baseline agents more frequently stall or deviate under unexpected contacts. These results illustrate that the co-designed morphology-police pair exhibits meaningful robustness to previously unseen disturbances and obstacle interactions.
consistent forward
Dense Obstacle Navigation
Sparse Obstacle Navigation
Figure 15: Visualization the unseen naviatinnn task environment.
36

---

Published as a conference paper at ICLR 2026
Table 15: Performance in the unseen obstacle-navigation task under two obstacle densities.
Performance
Obstacle Type
Spacing
Stackelberg PPO (ours)
BodyGen
Sparse Ohstacle
16 m (m-4 rohot width)
1790.45 1698.52 52333337
1061.551228.00
Dense Obstacle
8 m (~~2x robot width)
1007.21
Incorporating and benefiting from morphology priors. Our framework also supports reusing morphology priors obtained from related tasks. To examine this, we transfer morphologies evolved in the Crawler environment to initialize training in the Pusher task. Table 16 shows that both Stackel- berg PPO and BodyGen benefit from priors terms final performance and the number of environ- ment steps required to reach threshold reward. Stackelberg PPO consistently obtains higher final reward and requires fewer steps under both "with prior" and "without prior" conditions. Figure 16 visualizes representative morphologies produced under this setup. While priors accelerate training, generally advisable to choose priors that encode broadly useful structural patterns-Sti -ucch as stable support geometries balanced limb arrangements rather than narrowly specialized solu- tions. Such general-purpose priors provide more flexible foundation for downstream adaptation and reduce the risk of over-constraining the design space.
Crawler Task Prior
Pusher Task with Prior
Figure 16: Cross-task reuse of morphology priors: morphology (right).
Crawler prior (left) and the resulting Pusher
Table 16:
Performance and sample efficiency in the Pusher task with and without morphology priors.
Performance
Steps to Threshold (2500 Reward)
Condition
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours) BodyGen
With Prior
4822.59 114.32
4575.52 52
4575.5211122 112..78
8M
9M
Without Prior
3466..77 368.00
2779.93+509.16
32M
44M
37


Reviews:
[{"review_id": "f0bYUlSso2", "reviewer": "Reviewer_B8Qz", "summary": "The paper formulates morphology–control co-design as a phase-separated Stackelberg Markov Game (SMG) where a leader edits morphology via discrete actions and a follower optimizes control on the resulting body. Because the leader–follower interface is non-differentiable, the authors derive a trajectory likelihood-ratio loss for the Stackelberg cross-derivative and use the natural gradient approximation as a stable inverse-Hessian proxy. Combining everything gives the Stackelberg PPO. The experiments on various mujoco-based morphology control tasks demonstrate the competitiveness of the proposed method.", "questions": "The natural policy gradient is more computationally expensive than regular PPO. How does the computation efficiency compare with the other baselines, given the experiments are all in simulation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The formulation of morphology–control co-design as a phase-separated Stackelberg game is intuitive and matches the causal structure between design and control.\n\n2. The paper gives a clear algorithmic pipeline that can be implemented with existing PPO infrastructure.\n\n3. Experimental results are consistent across several benchmarks and ablations are reasonably detailed.", "weaknesses": "1. The technical novelty is modest, since the likelihood-ratio surrogate, natural gradient, and PPO clipping are all existing techniques; the contribution lies mostly in integrating them coherently.\n\n2. the cross-derivative estimator in eq. (6) seems to have very high variance. \n\n3. The baselines do not include prior Stackelberg RL algorithms (e.g., Stackelberg Actor-Critic), so it is unclear whether the improvement arises from the Stackelberg formulation itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aR1ntCakKa", "reviewer": "Reviewer_ry9P", "summary": "This work proposes efficient morphology control co-design via Stackelberg PPO. It formulates morphology–control co-design as a phase-separated Stackelberg Markov Game, where the leader is the morphology designer and the follower the policy controller. The Stackelberg PPO is a policy-gradient method that uses an implicit-differentiation surrogate (likelihood–ratio–based) across a non-differentiable leader–follower interface, plus Fisher-information preconditioning and PPO-style gradient clipping. On MuJoCo co-design benchmarks (Crawler, Cheetah, Glider, TerrainCrosser) and two new stair tasks, the method reports better stability and final reward than PPO-based co-design and evolutionary baselines (e.g., +22% on average, ~+32% on harder 3D tasks).", "questions": "- How sensitive are results to morphology editing horizon T?\n- Does the follower avoid full re-training when morphologies change significantly, or does the advantage mostly come from better leader gradients? Any evidence of faster controller adaptation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a novel formulation of the morphology co-design problem, where the gradient from the morphology design phase is not differentiable. The Stackelberg game formulation is interesting and well-motivated. This avoids ad-hoc joint updates and gives a clean bilevel control–morphology structure.\n    - The derivation for a Phase-Separated Stackelberg Markov Game’s **likelihood-ratio surrogate** with a **Fisher preconditioner** creates a tractable gradient signal for the leader without assuming differentiability of morphology parameters or unrolling large follower optimization loops. This is a **meaningful algorithmic advance** over evolutionary search or unrolled gradient-based bilevel RL, which are expensive and brittle.\n- The multi-step morphology evolution, compared to prior art (e.g. Transform2Act) is a good insight and brings in notable improvement.\n- Training efficiency improvement is notably higher than prior art as the morphology editors are now more informed with controller training results.", "weaknesses": "- Most evaluations reduce to forward-velocity rewards on stylized locomotion creatures; even the new stair tasks keep the same simple progress reward. This makes it hard to assess whether Stackelberg coupling helps with *real* co-design constraints (payload, torque limits, manufacturability, sensor placement, power, robustness).\n- The λ/Fisher ablations are informative, but I’d like visibility into **data efficiency**: how many follower steps are actually saved vs PPO co-design? Since the thesis is “less re-optimization,” show sample-complexity curves for controller updates per viable morphology. (There is a follower-sampling hyperparameter, but not a direct “control-retraining cost” metric.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ea2AsEgDoS", "reviewer": "Reviewer_vMFr", "summary": "The paper reframes morphology–control co-design as a phase-separated Stackelberg game and derives a likelihood-ratio–based Stackelberg PPO that captures follower adaptation without back-propagating through a non-differentiable interface. The method claims better sample efficiency and final reward across six MuJoCo tasks versus PPO-based and evolutionary baselines, with theoretical support (local equivalence surrogates) and stabilizing design choices (PPO clipping, Fisher metric).", "questions": "1. Please address the concerns in the Weakness section.\n\n2. Estimator properties. What are the empirical bias/variance characteristics of the follower-implicit gradient term (eqs. 6–9) as the leader horizon grows? Any variance-reduction beyond standard advantage baselines?\n\n3. Local equivalence radius. How sensitive are results to step size, i.e., when does local surrogate equivalence break, and how does PPO clipping mitigate that in practice? Provide KL traces or constraint violations.\n\n4. Reward shaping sensitivity. How do results change if you (i) increase control-effort penalties, (ii) impose morphology complexity/material costs as leader rewards, or (iii) evaluate robustness (disturbances/terrain noise)?\n\n5. Generality. Does Stackelberg PPO extend beyond locomotion (e.g., manipulation, multi-objective design)? Any results on unseen tasks or morphology priors to show transfer?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Clear problem framing for non-differentiable interfaces.** The phase-separated SMG formalization precisely matches co-design realities (discrete edits then control), addressing why prior Stackelberg methods fail here. Treating co-design as leader–follower nicely explains why joint training can wobble or fail.\n\n2. **Technically neat gradient path.** The trajectory likelihood-ratio surrogate for the cross-derivative (Theorem 1) avoids differentiating through morphology transitions and yields local equivalence to true Stackelberg gradients. This is practical for implementation, easy for adoption with downstream ideas.\n\n3. **Stability machinery.** PPO-style clipping and Fisher (natural-gradient) Hessian approximation are well-motivated for practical computation. The $\\lambda$ sweep and Fisher vs. analytic Hessian give me a bit of insight into stability.", "weaknesses": "1. There are concerns about the experiment outlines:\n- Prior Stackelberg RL (e.g., Stackelberg actor-critic / policy-gradient) is acknowledged, but there’s no controlled ablation that swaps in those estimators under the same phase-separated setting to isolate what PPO-clipping contributes vs. the SID term itself.\n- Only four seeds are used “due to cost”; yet the authors claim “+22.1% average, +31.9% on 3D”, which needs more statistical significance. These environments are not that complicated to run at least 10 seeds.\n- The paper argues for efficiency, but training still takes ~30 hours per model on an A100 + 10 CPU cores. There should be implementation optimization, or the absence of wall-clock comparisons to ES baselines (which can parallelize) leaves practical efficiency unclear.\n- Most environments reward forward velocity (with optional small effort penalty), so improvements might partially reflect optimization for locomotion speed rather than broader co-design quality (e.g., robustness, energy, material constraints).\n\n2. On the theory side, surrogates are locally equivalent; the practical bias/variance of the coupled gradient estimator (esp. through long leader horizons) is not deeply characterized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BUCY07g20H", "reviewer": "Reviewer_1urx", "summary": "This paper makes a step on further improving embodiment co-design based on a recent interesting work BodyGen[1]. How to balance the bi-level optimization of co-design, is indeed a critical point for this area. This paper introduces Stackelberg Game, making this paper a promising solution for this target. The authors provide sufficient experiment results, and achieve some advantages over previous works.\n\n[1] Lu, Haofei, et al. \"Bodygen: Advancing towards efficient embodiment co-design.\" arXiv preprint arXiv:2503.00533 (2025).", "questions": "Personally, I have several questions about this paper:\n1) As far as I know, the bi-level problem introduced in the previous work, e.g. BodyGen and Transform2Act, their formulation is $max_{morph} (max_{control, morph})$, which already based on that the inner control policy is the best control policy. They just flatten the formulation with a universal controller, in order not to re-train the entire controller from the scratch. However, in the formulation of this paper, aka Stackelberg Game, I don't find any core differences with the old formulation.\n2) Following 1), for practical training using modern neural network, I believe that, if you use an end2end training using PPO like BodyGen, the inner universal control policy is always the \"best one\".\n3) You emphasise the adaptiveness of the inner controller according to the outer morph-designer, can you provide any experiments, demonstrating the evolving (especially topology differences) during training, e.g. 1/5, 2/5 of total iterations, what the morphology is like.\n4) I am curious about why you do not use all the tasks provided by your baseline, BodyGen & Transform2Act, since you build your entire pipeline on the top of BodyGen. Instead, you add some your personal task for evaluation.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The experiments cover multiple challenging co-design benchmarks, including both 2D and 3D locomotion as well as complex terrain settings. The comparisons against strong and diverse baselines demonstrate the robustness and generality of the approach. The method is supported by principled mathematical analysis with explicit gradient derivations under the Stackelberg formulation. The theoretical insights connect naturally with the empirical benefits in sample efficiency and stability.", "weaknesses": "I hope the author could provide anonymous code, which is strongly encouraged by ICRL. For the remaining weakness, please refer to questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CukWDZ54AR", "reviewer": "Reviewer_Md1x", "summary": "This paper presents a novel reinforcement learning-based brain-body co-design framework, named Stackelberg Proximal Policy Optimization (Stackelberg PPO). The core innovation lies in formally framing the co-design process as a Phase-Separated Stackelberg Markov Game, where a \"leader\" agent sequentially constructs the robot morphology, and a \"follower\" agent learns the control policy for the resulting body. The authors derive a novel Stackelberg policy gradient using a likelihood-ratio trick, which allows the leader to anticipate the follower's adaptation without requiring backpropagation through the non-differentiable simulator. Integrated with PPO's clipping mechanism for stability, the method demonstrates significant improvements in final performance but is kind of computationally expensive.", "questions": "Please refer to the \"Weakness\" Section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper contributes a rigorous reformulation, which formulates the brain-body co-design problem as a Stackelberg game. \n\n2. The experiments are comprehensive and convincing.\n\n3. The paper includes thorough ablation studies that validate key design choices, such as the regularization parameter.", "weaknesses": "1. The proposed method introduces very significant algorithmic complexity. The requirement to compute or approximate inverse Hessians via conjugate gradient, even with efficient tricks, adds computational overhead and implementation hurdles compared to vanilla PPO. Furthermore, the Stackelberg gradient relies on estimating second-order terms. As the complexity of the morphology and control spaces increases, how does the computational cost of this estimator scale?\n\n2. The paper claims that \"Transform2act enables joint training of structure and control without retraining per design. However, the interface\nbetween structure and control remains non-differentiable: the two modules are optimized independently under a shared reward, with no explicit gradient flow between them\". I don't quite understand what this sentence means. I think it is differentiable via policy gradient modeling.\n\n3. The authors are encouraged to clarify and analyze differences between their work and other competion-based co-design methods, such as CompetEvo and \"The Body is Not a Given\".\n\n4. The proposed approach seems to be too complex, relies on a lot of tricks, and can cause a lot of trouble for engineering implementation. It's more like a new PPO algorithm than a specific algorithm for co-design problem. Is it possible to use this algorithm in the neural architecture search problem (which also has a bi-level structure)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)