Title: Efficient Morphology–Control Co-Design via Stackelberg PPO under Non-Differentiable Leader–Follower Interfaces
Primary area: reinforcement learning
Keywords: Morphology–Control Co-Design, Stackelberg Game, Policy Gradient, Proximal Policy Optimization, Non-Differentiable Leader–Follower Interactions, Reinforcement Learning

Abstract: Morphology-control co-design concerns the coupled optimization of an agent’s body structure and control policy. A key challenge is that evaluating each candidate morphology requires extensive rollouts to re-optimize control and assess quality, leading to high computational costs and slow convergence. This challenge is compounded by the non-differentiable interaction between morphology and control---stemming from discrete design choices and rollout-based evaluation---which blocks gradient flow across the morphology-control interface and forces reliance on costly rollout-driven optimization. To address these challenges, we formulate the problem as a novel variant of Stackelberg Markov game, a hierarchical framework where the leader specifies the morphology and the follower adapts the control---a natural yet underexplored perspective. Building on this formulation, we propose {Stackelberg Proximal Policy Optimization (Stackelberg PPO)}, a policy gradient method that leverages the intrinsic coupling between leader and follower to reduce repeated control re-optimization and enable more efficient optimization under non-differentiable interfaces. Experiments across diverse co-design tasks demonstrate that Stackelberg PPO outperforms standard PPO in both stability and final performance.

TL;DR: We propose Stackelberg PPO, a policy gradient method that efficiently co-optimizes robot morphology and control under non-differentiable leader-follower interfaces.