Motion Control of High-Dimensional Musculoskeletal Systems with Hierarchical Model-Based Planning
Authors: Yunyue Wei
Shanning Zhuang
Vincent Zhuang
Yanan Sui
Venue: ICLR 2025 Poster (2025)
Keywords: Model predictive control
High-dimensional embodied system
Primary area: applications to robotics, autonomy, planning

Abstract: Controlling high-dimensional nonlinear systems, such as those found in biological and robotic applications, is challenging due to large state and action spaces. While deep reinforcement learning has achieved a number of successes in these domains, it is computationally intensive and time consuming, and therefore not suitable for solving large collections of tasks that require significant manual tuning. In this work, we introduce Model Predictive Control with Morphology-aware Proportional Control (MPC$^2$), a hierarchical model-based learning algorithm for zero-shot and near-real-time control of high-dimensional complex dynamical systems. MPC$^2$ uses a sampling-based model predictive controller for target posture planning, and enables robust control for high-dimensional tasks by incorporating a morphology-aware proportional controller for actuator coordination. The algorithm enables motion control of a high-dimensional human musculoskeletal model in a variety of motion tasks, such as standing, walking on different terrains, and imitating sports activities. The reward function of MPC$^2$ can be tuned via black-box optimization, drastically reducing the need for human-intensive reward engineering.

Published as a conference paper at ICLR 2025
MOTION CONTROL OF HIGH-DIMENSIONAL MUSCU- LOSKELETAL SYSTEMS WITH HIERARCHICAL MODEL- BASED PLANNING
Yunyue W Shanning Zhuang", Vincent Zhuang", Yanan Sui! Tsinghua University Google DeepMind fweiyy20, zhuangsn2 (maaiis htt.e/attii...tiniuaaaaaaaaaaaaa edu. en vincent zhageggeu com
ysuištsinghua edu.cn
ABSTRACT
Controlling high-dimensional nonlinear systems, such those found in biologi- cal and robotic applications, is challenging due to large state and action spaces. While deep reinforcement learning has achieved a number of successes in these domains, it is computationally intensive and time consuming, and therefore not suitable for solving large collections of tasks that require significant manual tun- ing. In this work, we introduce Model Predictive Control with Morphology-avee Proportional Control (MPC²), a hierarchical model-based learning algorithm for zero-shot and near-real-time control of high-dimensional complex dynamical sys- tems. MPC² uses a sampling-bee planning, and enables robust control for high sigh-dimensional tasks by incorporat- ing a morphology-awee algorithm enables motion control of high-dimensional human musculoskeletal model in a variety of motion tasks, such as standing, walking on different terrains, and imitating sports activities. The reward function of MPC² can be tuned via black-box optimization, drastically reducing the need for human--ntenssie reward engineering.
model predictive controller for target posture
proportional controller for actuator coordination. The
Stand
Walk
Muscle Injury
Manipulation
Sports Imitation
Figure 1: Movement control of whole-body human musculoskeletal system over a diverse set of motion control tasks. The videos of the control performances are the project page.
INTRODUCTION
High-dimensional nonlinear dynamical systems are prevalent in the real world, with important ex- amples including biological musculoskeletal systems. The system complexity laid the foundation of flexible motion due to their over-actuated nature. The presence of redundant actuation enhances the safety and robustness of the system, reducing the risk of performance degradation from actuator faults (Hsu et al., 1989). However, also leads to large state and action spaces, posing significant challenges to achieving stable control performance. We take the human musculoskeletal system as

---

Published as a conference paper at ICLR 2025
Full-body Musculoskeletal System
MPC²
Model Predictive
Morphology-aware Proportional Controller
Position Cotroller
State
Target Posture
Actuator
Control
Instant
Optimization
Control
MPPI
Rollout Model
Figure 2: Workflow of Model Predictive Control with Morphology-wwe aware Proportional Control
(MPC²). Solid arrows indicate control pipeline, and dashed arrows indicate planning procedure.
key example, where hundreds of muscles coordinate to facilitate various movements, Understand- ing and optimizing control in such systems is crucial for applications in healthcare and human robot interaction (Kidzinski al., 2018; Vittorio al., 2022).
Deep reinforcement learning (DRL) is promising approach for controlling high-dimensional sys- tems. However, RL approaches often struggle in high-dimensional state and action spaces, and typically require the use of lower-dimensional representations. More importantly, the immense computational requirements of DRL imposes a severe bottleneck on the iteration speed of reward engineering, meaning that researchers often need days (or longer) to discover effective control poli- cies. Furthermore, the high nonlinearity of the musculoskeletal system necessitates considerable sequential computation, complicating its parallel deployment on GPUs and impeding the training speed of DRL algorithms. Being able to generate effective control policies for high-dimensional nonlinear dynamical systems in near real-time is an open challenge.
Clinical studies on motor control of human movement revealed that predictive sampling is a crucial strategy in human movement control, such as maintaining balance during walking (Winter, 1991; Patla, 2003), where planning over finite horizon determines the controls to be executed. Recent works have started incorporate model predictive control (MPC) as the control backbone, offering faster behavior synthesis and more efficient reward design compared to DRL (Howell et al., 2022; Yu al., 2023). However, effective planning nigh-dimensional control spaces remains challenging, limiting the success of MPC primarily to low-dimensional systems. To the best of our knowledge, no training- free methods have achieved stable movement control of a whole-body musculoskeletal model across varying task conditions.
In this paper, we propose Model Predictive Control with Morphology-aware Proportional Control (MPC?), a hierarchical model-based planning algorithm designed to address the challenges of high- dimensional musculoskeletal control. We introduce a sampling- based model predictive controller to plan the target posture of the agent, while morphology-aware proportional controller serves as the low-level policy, adaptively coordinating the actuators to achieve the target joint positions. We demonstrate that our method can achieve stable control of 700-actuator whole-body musculoskele- tal model without training, enabling tasks such as standing, walking over varying terrain conditions, and sports motion imitation (Figure 1). Furthermore, facilitates efficient cost function optimization, improving task performance, especially for perform- ing complex sequences of movement. The bottleneck in achieving real-time control with MPC is the speed of the additional model forward dynamics computation, which can be solved by using more powerful computing devices or by controlling systems with reduced complexity.
show that MPC fast control generation
Our contributions. (1) We propose MPC the first MPC-based method capable of achieving near real-time stable control of high-dimensional musculoskeletal systems. (2) We demonstrate that our hierarchical model predictive control algorithm enables zero-shot high-dimensional full-body motion control across a wide range of motion tasks, many of which have not been achieved by state-of-the-ap DRL-based methods. (3) We show that the much faster control generation latency of 2PC facilitates automated cost function optimization via Bayesian optimization, demonstrating a pathway for reducing the human burden of reward engineering to near
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
High-dimensional musculoskeletal control. The control of musculoskcletal systems challeng- ing due to both high dimensionality and non-linearity, with deep reinforcement learning (DRL) being the predominant choice in existing solutions (Kidziñski et al., 2018; Geiß et al., 2024). Hi- erarchical architectures
often employed to decompose control across different modules, where DRL provides high-level actions and a low-level policy generates muscle controls (Lee et al., 2019; Park al., 2022; Feng al., 2023). These approaches typically require large collections of motion data for imitation learning. Several works have also explored strategies to improve sample effi ciency in over-actuated regime. including bio-inspired exploration (Schumacher al.. 2022), latent space exploration (Chiappa et al., 2023), model-based planning (Hansen et al., 2023), and multi-task learning (Caggiano et al., 2023). Recent studies have leveraged muscle synergies to reduce control dimensionality, enabling stable control across various musculoskeletal models (Berg et al., 2023; He et al., 2024). These methods typically require many hours or days of training to achieve effective control, posing significant bottleneck on the iteration speed of reward engineering.
Model-predictive control. Compared to DRI., model predictive control allows for real-time con- trol (Tassa et al., 2012), and thus has seen increasing application of MPC in roboties, including tasks such as quadruped locomotion and dexterous manipulation (Kim et al., 2023). Recent works have also integrated MPC into the reward design process due to its waining-ffree nature (Jain al., 2021; Yu et al., 2023; Liang et al., 2024). However, MPC typically succeeds only in low-dimensional settings and often struggles when applied to high-dimensional problems. The most complex systems handled by existing MPC-based methods are typically torque-driven humanoids (Meser et al., 2024).
3 PRELIMINARIES
3.1 MUSCULOSKELETAL SYSTEM CONTROL
High- digensiimnii over-actuated system. In this paper, used musculoskeletal models as the target high-dimensional over-actuated system, where the model dynamics can be formulated as fol- lows:
(1) where q denotes generalized coordinates of joints, M(q) denotes the mass distribution matrix, and cly, () denotes Coriolis and the gravitational force applied over the generalized coordinates, Jm and
(q) c(q.9) J. + J fc+ Texts
J. denote Jacobian matrices that map forces to the generalized coordinates, fe is the constraint force,
Jm denotes actuator forces, and Text denotes all external torque when interacting with environments.
Our used musculoskeletal models are implemented in the MuJoCo physics simulator (Todorov et al., 2012), where actuators are modeled as first-order systems. The force generated by one actuator can be formulated as follows:
da
fT Fv. Fp:
Øt
(u -) + T2²
where is the actuator activation, Fk, Fp represents the gain and bias of the actuator force dynamics, u denotes the actuator control, Tı and 72 denote the time coefficients of the first-order actuator system. Our primary experiments conducted the MS-Human-700,, body musculoskeletal model comprising 90 rigid body segments, 206 joints, and 700 muscle-tendon units (Zuo et al., 2024). Additionally, employ upper limb model of the human body and an ostrich model (La Barbera et al., 2021) to showcase the generalization of MPC across different models and tasks.
comprehensive whole-
Problem formulation. We treat the high-dimensional over-actuated control problem as a finite horizon Markov decision process with state S, control u U. and dynamics f. For given initial state of the model so and a desired horizon 7' we aim to find control sequence u (uo, .... 11 that enable stable control, which can be achieved by minimizing the cumulative value of task-specific cost function Co parameterized by 0:
u argmin
Calsi, uz), 8141 f((,, un)
(3)
3

---

Published as a conference paper at ICLR 2025
In this paper, the definition of cost function Co is equivalent to the reward functions used in rein- forcement learning (with negative value for maximization). For MS- 105-Hummnn-00 we consider the action space is d, 700-dimensional control of actuators (muscle-tendon units). The state space of the full-body model consist of joint positions and velocities, actuator activations and lengths, and task-related observations, leading space dimensionality d, over 1500.
3.2 SAMPLING-BAN BBALED MODEL PREDICTIVE CONTROL
Model predictive control is general framework for model-based control, which optimizes local control sequence using an approximated dynamics f within a short horizon H & T:
0 H argmin BrctH
Colseth: (eth), 84th+1 = f(seth, ucth).
(4)
h=0
The optimized action sequence UH (if..... 0l -1+ is a local approximation of optimal controls U H. In real-world deployment where the action execution and planning are asyn- chronous, the planning horizon H should be chosen balance accuracy and instantaneity.
Among various implementations of MPC frameworks, sampling- baseed based MPC is a popular choice which samples local control sequences from a distribution of open-loop control sequences, ü ~ (-).), and update the sample distribution via parallel rollouts of the sampled action sequences. The objective of sampling-based MPC is to find distribution parameter & that minimize the cumula- tive cost function value of sampled action sequences. The distribution update process usually only depends on the rollout performance without direct operation on the states, which has been demon- strated success in the control of high fegree-of-frredon systems, such torque-driven humanoid models (Meser et al., 2024).
Model Predictive Path Integral (MPPI) control (Williams et al., 2016) is commonly used sampling- based MPC method, which assumes the sampling distribution is a factorized Gaussian with = (j4,' 114HH-11,0 0t+n-1):
H-1
Po(ūr14H) II N(e++hi Meth, 04th)
(5)
During the rollout process, N action sequences (fea J are sampled and executed via ap- proximated transition f. For each sampled sequence U. the cumulative cost function Cø ΣH--1 Colseth, a))n) is collected and used for distribution update:
ΣN Uith
ΣN Wn (6ith meth)2
Me+h ΣN Wn
otth
Σ Wn
0 < h H-1, (6)
where wn, =1 A r(n) is the increasing- rank of cumulative cost function value of rollout n. i is the number of elite rollouts, and À is the temperature parameter.
3.3 OPTIMAL COST FUNCTION DESIGN
The finite-horizon optimization of MPC can result in a myopic policy, which may suboptimal when evaluated in the long term. Recent studies demonstrate that the parameters of the cost function can be optimized to compensate for the issues induced by local optimization, which can be different from the true cost function measured in the full horizon T (Jain et al., 2021; Le & Malikopoulos, 2023). The objective of optimal cost function design is to find parameters 6* that minimizes the cumulative value of true cost function Co over the horizon T::
0" argming
Σ t=0
Colse, uy Se+1 = f(st, a
where (in ized by 0'. As only coro-rrder cost function value be accessed, e.q. 7 can be considered black-box box optimization problem, which can be addressed by Bayesian optimization or evolutionary algorithms.
2% 1) is the control sequence MPC optimized under cost function parameter-
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Model Predictive Control with Morphology-aware Proportional Control (MPC*)
Input: Model dynamics f. rollout horizon H, total rollout number N, instant rollout number N, iteration number r, distribution parameter p. a, current state St
for i 1,.... do y.... ~ MNMMpaa(st)) N+1 NN ~N(H,
11 Instant rollout // MPPI rollout
Cp..... ,C RMp(=t, "
RMPl= ,H)
Update ye, using e.q. (6)
end
=" <- 14, up t TMP( (5, z*)
return if, x", 1
4 MODEL PREDICTIVE CONTROL WITH MORPHOLOCY-AWARE PROPORTIONAL CONTROL (MPC²)
Existing approaches for controlling high-dimensional musculoskeletal systems often incorporate deep reinforcement learning a central component, where a state-feedback policy, Tuuss), is learned from interactions with the model dynamics. While substantial efforts have been made to reduce the dimensionality of the action space, the large state space continues present significant challenges for policy training. In this paper, we opt to use model predictive control instead of deep reinforcement learning for the following reasons: 1) the overall control is conducted in simulation, where the exact dynamics is accessible, that is f f: the use of sampling- -baeed MPC cireum- vents the challenge of decision-making in high-dimensional state spaces; 3) MPC offers much faster control generation, enabling more reward design optimization iterations than DRL.
However, directly deploying MPC on musculoskeletal systems is challenging because of the high dimensionality. this section, we demonstrate that applying MPC such problems indeed possi- ble. Our approach is motivated by the observation that many biological systems such as vertebrates utilize hierarchical control strategies, in which sensory information is processed by high-level controller for planning, while motor commands proprioception (Merel et al., 2019). To this end, we introduce MPC², a hierarchical MPC method that facilitates stable control of high high-dimenssooal musculoskeletal systems, shown in Figure and Algorithm 1. MPC² has two major components: (1) a model predictive position controller the high-level planner which optimize for the target posture 2" given current state s, and (2) a morphology-ave -wware proportional controller TMP TMP(u|s. (u|s, =) as the low-level policy which computes actuator controls to achieve the target posture from given state.
generated by low-level controller based on
4.1 MODEL PREDICTIVE POSITION CONTROL.
We employ MPC over the planning major joint coordinates z that determine the system posture. For 700H,man--00,, the dimension of is de 37. Compared torque, choose lower-order joint position as the MPC objective to reduce the control frequency. Therefore, only one target posture is required to optimize during one rollout, where morphology proportional controller adapts control signals based on the instant states:
C. RMP H)
Σ C(sreh Wenk) U44h TMP(SE4A, 2).
Compared to planning over original action space, MPC² significantly reduce the number planning parameters from H d, to dg. enabling optimizing controls via sampling. While the original Model Predictive Path Integral (MPPI) can be directly employed we find that it lacks the ability to respond quickly to rapidly changing states, such as when the agent is falling. This issue cannot be easily mitigated by simply increasing the number of rollouts, only limited number (at most few dozen) can be executed in parallel when controlling nigh-dimensional systems, due to both computational budget constraints and the need for real-time responsiveness.
high-level planner for target positions,
5

---

Published as a conference paper at ICLR 2025
To equip MPC with rapid response capabilities for changing states, leverage the feature of posi- tion control and propose the use of instant rollouts during planning (line 2 in Algorithm 1). Rather than sampling based the policy from the previous planning iteration, instant rollouts sample tar- get postures based on the model's current posture, which can be extracted from the current state using posture mask: 24 Mpn((S)). When the current state significantly deviates from the pre- vious planning state, this approach provides better initial point compared to the original MPPI samples, increasing the likelihood of sampling more effective controls re-stabilize the agent. We will demonstrate the necessity of instant rollouts for position control in the experimental section.
4.2 MORPHOLOGY-AWARR PROPORTIONAL CONTROL
Here we introduce the morphology-aware proportional controller TMP(u[s, 2). a key component for reducing the control dimensionalityy which coordinates actuator controls to adapt to the target posture. Given the target joint coordinate 2", the target actuator length ! can be computed with model forward dynamics. We define proportional controllers for each actuator, which determine the actuator force required to achieve the target actuator length given current actuator length I:
f min(0, k- (I" ()):
where the proportional gain parameter. Utilizing the first-order actuator dynamics in 2, we able to derive the control signal u" to achieve target actuator force fmh given current actuator activation a:
72 a) At - (a a)
u a
(10)
where a- = -f F is the target actuator activation, and At is the duration of each time step. The proportion gain vector K = .k...... ke.)) controls the scaling of target forces, which is critical for the control performance. Improper gain settings can result in excessive collisions (if too large), insufficient force generation (if too small), which should be individually set for each of the 700 actuators.
From system dynamics in e.q. 1, the conversion from actuator forces to joint torque is computed using the Jacobian matrices of the model, Jms which can represent the influence of actuators on joint movements. Based on this observation, morphology. Instead of manually setting these gains, we set them based on the Jacobian matrices of current state and the target posture:
propose to set proportional gains according to the system
Re Σ Icol;(Jm)) [2]
(8e)s)
(11)
where le is the only scaling parameter, I, is the indices of major joints x over all joints, |.| is the absolute value operator, and () the column operator of Jm. The Jacobian values vary according to different system posture, allowing for adaptive and efficient control of different motion.
Note that 2PC achieves high-dimensional musculoskeletal control through online planning using model dynamics, allowing for the control of complex behaviors without the need for training procedure. This zero-shot motion control also enables rapid evaluation of cost function designs, facilitating efficient optimization of the cost function.
EXPERIMENTS
In this section, we aim to comprehensively evaluate MPC², and seek to answer the following ques- tions: 1) Can MPC² achieve robust and performant control over a wide variety of motion tasks and models? 2) Can MPC² generalize across different model morphology? 3) Can we leverage the fast generation speed of MPC serve as the inner loop in a reward function optimization problem? 4) How do the individual components of MPC contribute to its overall effectiveness?
Implementation details. We implement MPC² using the Mujoco MPC (MJPC) platform (Howell et al., 2022), framework designed for real-time model predictive control. The MJPC platform supports asynchronous simulation between the main thread and planning, which we find to more practical than freezing the main thread during planning. In all experiments, we set the iteration

---

Published as a conference paper at ICLR 2025
(a)
(d)
(e)
(ba)
(f)
Figure 3: Control sequences of MPC in (a) Stand, (b) Rough, (c) Walk, (d) Slope and (e) Stair (f) Ostrich walk tasks. The simulation speed of Stair task set to 10% due to slower contact computation.
number of MPC to 1 for rapid response the changing states in the main thread, and sample 64 rollouts (containing N = 10 instant rollouts) across 0.3s horizon during each round of planning.
Unless otherwise noted, the simulation in main thread are run with 20% of the real-time speed (following Howell et al. (2022)), where control sequences to complete the task can be generated within minutes. The experiments of MPC were conducted on a server equipped with an AMD EPYC 7773X processor, an NVIDIA GeForce RTX 4090 GPU, and 512 GB of memory. The cost function design of all movement tasks detailed Appendix
5.1 ROBUST MOVEMENT CONTROL
Motion control over different terrain. We design the following control tasks, which consists of wide range of human full-body motion: (1) Stand. This task requires standing still and keep balance for 10 seconds.
(2) Walk. This task requires walking forward over flat floor for 10 meters. (3) Rough. This task requires walking forward over rough terrain for 10 meters. (4) Slope This task requires walking up and down slopes. (5) Stair. This task requires walking up and down stairs.
We show control sequences of the Stand, Walk, Rough, Slope, and Stair tasks using MPC² in Figure While no previous control methods have demonstrated success in whole-body musculoskeletal systems for these tasks, MPC² exhibits consistent and stable control performance across various tasks, enabling navigation over different terrain conditions.
Adaption to model changes. We demonstrate that MPC" effectively leverages the over-actuated nature musculoskeletal systems to achieve stable control even in the presence of actuator failures. As shown in the Figure 4(a), disabled the biceps femoris, gastrocnemius, semitendinosus, and semimembranosus muscles on the back the right leg at the 5th second of walking to test whether the overdrive of the system to adap- MPC² continue walk. This requires the controller

---

Published as a conference paper at ICLR 2025
Pre-injury
Post-injury
(a)
(
Figure 4: Walking control under sudden muscle failure. (a) Illustration of muscle injury. (b) Control sequences of MPC². The dotted line shows the time when the muscles fail.
tively adjust the control strategy. As illustrated in Video W10 and Figure 4(b), MPC dynamically adapts its control strategy to maintain forward walking despite the sudden disablement of the pos- terior muscles in the right leg. We also find trained DRL agent fails to walk with actuator faults, as shown in the Video W3.
Robustness with respect to perturbation forces.
The MPC² algorithm
10
demonstrates robustness
Average Walk Distance (m])
Average Stand Time (s)
MPC?
to certain perturbations, as
DynSyn MPPI
DynSyn MPPI
evidenced in two walking
Sum muscle act
300
scenarios. As shown in Video W11 and W12, MPC² successfully main- tains forward walking
200
DynSyn
despite the application of significant external forces, including large, random, short-term forces (500N applied for 0.2 seconds every second) and consis- tent, random forces (100N applied continuously).
10a
102
10a
10a
102
10a
10
Time (min) (a)
(min) (b)
Time (s)
(c)
Figure 5: Control performance versus clock time of (a) Stand task, and (b) Walk task. Results show the mean performance with standard error, averaged over 50 independent trials. (c) Energy consumption during walking.
These results highlight the system's 's ability to adapt and maintain stability under challenging conditions.
Comparison to RL and MPC baselines. In the Stand and Walk tasks, we compared the control performance of 2 with the current state-of-the-art DRL-based al- gorithms, DynSyn (He et al., 2024), which identify and utilize muscle synergies to reduce control dimensional- ity, and demonstrates stable walking control over whole- body musculoskeletal model. We also included the origi- nal MPPI (Williams et al., 2016) baseline perform an ablation of our hierarchical pipeline. Figure (()) show the total time (training time + deployment time) re- quired for control sequence generation. We observe that DynSyn requires at least one day achieve effective con- trol in both tasks. While MPPI is capable of maintaining balance in the Stand task, struggles to generate control sequences for forward movement in the high-dimensional action space. MPC² enables stable standing and walk- ing control within 2 minutes, demonstrating a significant time efficiency advantage over DynSyn for deployment. energy
Target orientations
Figure 6: Dexterous manipulation se- quences of MPC² over arm muscu- loskeletal model.
We record the sum of muscle activations sumption measurement during walking in Figure 5(c). Although energy regularization terms is included in the cost function, observe that MPC
8

---

Published as a conference paper at ICLR 2025
+
Figure 7: Control sequences of MPC² in soccer sports imitation. The simulation speed is set to 1% for more frequent planning for rapidly changing motion, where the entire control sequence learned within 4 minutes.
reduces muscle activation by over 75% compared to DynSyn. We also ran other DRL and MPC baselines in Appendix C.5, where none of them capable of achieving walking.
5.2 CONTROL ACROSS DIFFERENT MORPHOLOGIES
In addition to the challenging task of human boll-body locomotion control, we demonstrate that MPC² generalizes zero-shot the control of another whole-body musculoskeletal model with different morphology, as well as to a local human arm model for dexterous manipulation.
Motion control of ostrich model. In Figure 3(f) we demonstrate that MPC successfully performs morphology computation and achieves stable control for ostrich musculoskeletal models with 120 muscle-tendon units (La Barbera et al., 2021) using the same controller applied to the full-body human model. Notably, the cost function used for the ostrich model is identical to that used for human walking. This highlights MPC ability perform planning across systems with varying morphologies without requiring training, cost function tuning, or controller parameter adjustments.
Dexterous manipulation of Arm musculoskeletal model. In addition the whole-body move- ment task, we also evaluate the performance of MPC² on the dexterous manipulation task. As shown in the Figure 6(a), we need to control a hight-hand arm model with 85 muscle-tendon units. By driving the shoulder joint, elbow joint, wrist joint and finger joints, the cube in the hand is ad- justed to the specified direction. Compared with whole-body movement, dexterous manipulation requires the controller to have a high reaction speed and more precise control. As shown in Fig- ure 6(b) and Video W28, MPC" successfully achieves dexterous manipulation of the arm museu- loskeletal model and can reach two different target block directions without training, which reflects the generalization of our method to model and control tasks.
5.3 AUTOMATIC BEHAVIOR SYNTHESTS WITH OPTIMAL. COST FUNCTION DESIGN
The fast control generation speed of 2PC enables rapid evaluation and iteration of cost function design. In settings where the true objective can be simply described, we can leverage black-box optimization algorithms to discover MPC cost functions that best optimize the true cost function, resulting in automatic behavior synthesis. If possible, this functionality is especially crucial in massively multi-task settings, where many complex behaviors must be generated.
We consider this problem in the setting of sports, which often require diverse and complex move- ments. As a case study, we investigate whether MPC² combined with a black-box optimizer can au- tomatically learn to kick a soccer ball. We specifically use Gaussian-process- process- baased Bayesian opti- mization algorithm to optimize the weights of the position error terms for different body parts (Jones et al., 1998; Rasmussen, 2003; Ament et al., 2023). The optimization objective the quadratic po- sition error of each body part, with results shown in Figure 10(a). We observe that the cost objective substantially improves compared to the initial settings. Thanks to MPC?'s training-fres control generation, our 100 cost design iterations take only around 5 hours, whereas DaL-based methods cannot even complete a single reward evaluation (i.e. a single trained policy) in that time frame. MPC² successfully imitates the reference trajectory and enables sports motion control, generating

---

Published as a conference paper at ICLR 2025
sufficient speed and force kick the ball (Figure 7). We additionally demonstrate in Appendix C.3 that the automatic cost function design is able to significantly increase the walking speed in both the human and ostrich models.
10
Sum of Jacobian
Walk Walk Distance
0.3
Stand Time
0.2
Stance phase
Swing phase
semitendinosus hiceps femoris
gluteus minimus gluteus medius gluteus maximus adductor magnus
MPC Constant 10 control
MPC?
w/o instant
(b)
Figure 8: Analysis of MPC Results show the mean performances with one standard error over 20 trials. (a) Control performance of lean backward standing, with initial position shown on the left. Blue axis indicates the vertical direction. (b) Control performance of the Walk task. (c) The distribution absolute Jacobian summations of a walking trajectory.
5.4 ALGORITHM ANALYSIS
To understand superior control performance behind MPC², investigate both model predictive position controller and morphology-avee aware proposition controller. The analysis results shown in Figure 8.
Instant rollout for rapid planning. We modified the standing task to evaluate the effectiveness of the instant rollout component in high-level posture planning. As shown in Figure 8(a), instead of starting from an upright position, we set the initial posture of the model to lean significantly backward, requiring a rapid response to recover balance. Our results show that MPC² significantly outperforms its variant without the instant rollout, demonstrating that the instant rollout enables timely response in unstable states.
Morphology-aware gain design. We compare MPC² with two variants over the low-level actuator controller side: (I) a proportional controller with constant gain settings for all actuators, which has similar average actuator force MPC², and (2) proportional-derivaiive (PD) control, setting the derivative gains based on the proportional gains. Figure 8(b) shows that MPC" significantly outperforms both the constant gain and PD control variants. In Figure 8(c). observe that the system's 's Jacobian effectively identifies the major muscles involved during walking and adapts to different phases of motion. Our morphology-awan gain design automatically prioritizes the major actuators for more efficient control, demonstrating its fidelity in biomechanics.
CONCLUSION
In this paper, we propose MPC?, a hierarchical model predictive control method designed to en- able near real-time motion control of high-dimensional musculoskeletal systems without the need for training. The algorithm employs a high-level model predictive position controller for posture planning and utilizes morphology-avve aware proportional controller to coordinate actuators in achiev- ing the target posture. Using whole-body model with 700 actuators, we demonstrate the stable control performance of MPC² across wide range of movement tasks, as well as its fast controller generation for efficient cost function optimization. Ablation studies over the algorithm components further verify the principled design and biomechanical fidelity of MPC².
ACKNOWLEDGMENTE
This work is supported by Tsinghua University Initiative Scientific Research Program and STI 2030- Major Projects 2022ZD0209400. Correspondence to: Yanan Sui (ysui saui@sssinhua..du....)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Un- expected improvements to expected improvement for bayesian optimization. Advances Neural Information Processing Systems, 36:20577-20612, 2023.
Cameron Berg, Vittorio Caggiano, and Vikash Kumar. Sar: Generalization of physiological agility and dexterity via synergistic action representation. arXiv preprint arXiv:2307.03776, 2023.
Vittorio Caggiano, Sudeep Dasari, and Vikash Kumar. Myodex: a generalizable prior for dexterous manipulation. In International Conference on Machine Learning, pp. 3327-3346. PMLR, 2023.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Huang, and Alexander Mathis. Latent ex- ploration for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
Yusen Feng, Xiyan Xu, and Libin Liu. Musclevae: Model-based controllers of muscle-actuated characters, 2023. URL https: /ttp//aaavvv//////a//eee/eaae.l/ org/ abs/ /ttss/2312 2312222222 07340.
Henri-Jacques GeiB, Firas Al-Hafez, Andre Seyfarth, Jan Peters, and Davide Tateo. Exciting ac- tion: Investigating efficient exploration for learning musculoskeletal humanoid locomotion. arXiv preprint arXiv:2407. 1658, 2024.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for contin- uous control. arXiv preprint arXiv:2310.1683 16828, 2023.
Kaibo He, Chenhui Zuo, Chengtian Ma, and Yanan Sui. Dynsyn: Dynamical synergistic rep- resentation for efficient learning and control in overactuated embodied systems, 2024. URL https://aaxiie httll//rgggbbb//////// 22/////s/222.... 11472.
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, and Yuval Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco, 2022. URL https: // //arxivv.ccoo org/ abs
Ping Hsu, John Mauser, and Shankar Sastry. Dynamic control of redundant manipulators, Journal of Robotic Systems, 6(2):133-148, 1989.
Avik Jain, Lawrence Chan, Daniel Brown, and Anca Dragan. Optimal cost design for model predictive control. In Learning for Dynamics and Control, pp. 1205-1217. PMLR, 2021.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 33:455-492, 1998.
Lukasz Kidziński, Sharada Mohanty, Carmichacl F Ong, Jennifer Hicks, Sean F Carroll, Sergey Levine, Marcel Salathé, and Scott L Delp. Learning challenge: Synthesizing physiologi- cally accurate motion using deep reinforcement learning. The NIPS' Competition: Building Intelligent Systems, pp. 101-120. Springer, 2018.
Gijeong Kim, Dongyun Kang. Joon-Ha Kim, Seungwoo Hong, and Hae-Won Park. Contact-implicit mpe: Controlling diverse quadruped motions without pre-planned contact modes or trajectories. arXiv preprint arXiv:2312.0890 2023.
Vittorio La Barbera, Fabio Pardo, Yuval Tassa, Monica Daley, Christopher Richards, Petar Kor- mushev, and John Hutchinson. Ostrichrl: A musculoskeletal ostrich simulation study bio- mechanical locomotion. arXiv preprint arXiv:21 12.06061 2021.
Viet-Anh Le and Andreas A Malikopoulos. Optimal weight adaptation of model predictive con- trol for connected and automated vehicles in mixed traffic with bayesian optimization. 2023 American Control Conference (ACC), pp. 1183-1188. IEEE, 2023.
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG), 38 2019.
11

---

Published as a conference paper at ICLR 2025
Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, et al. Learning to learn faster from human feedback with language model predictive control. arXiv preprint arXiv:2402.1 10440....45,,, 2024.
Josh Merel, Matthew Botvinick, and Greg Wayne. Hierarchical motor control in mammals and machines. Nature communications, 10(1):1-12, 2019.
Moritz Meser, Aditya Bhatt, Boris Belousov, and Jan Peters. Mujoco mpe for humanoid control: Evaluation on humanoidbench. arXiv preprint arXiv:2408.00 00342, 2024.
Matthew Millard, Thomas Uchida, Ajay Seth, and Scott L Delp. Flexing computational muscle: modeling and simulation of musculotendon dynamics. Journal of biomechanical engineering, 135(2))021005,22) 2013.
Jungnam Park, Sehee Min. Phil Sik Chang, Jacdong Lee, Moon Scok Park, and Jehee Lee. Genera- tive gaitnet. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022.
Aftab Patla. Strategies for dynamic stability during adaptive human locomotion. IEEE Engineer- ing in Medicine and Biology Magazine, 22(2):48-52 2003.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Em- bodied exploration for reinforcement learning in overactuated and musculoskeletal systems, 2022.
URL https:://... /tt.//aavvvv../// www.siii.oorraaeooeeee abtp:///////// 2206 00484.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906-4913, IEEE, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi:
Caggiano Vittorio, Wang Huawei, Durandau Guillaume, Sartori Massimo, and Kumar Vikash. Myosuite a contact-rich simulation suite for musculoskeletal motor control. https: //github. http///yohub/myosuiiii 2022. URL https: /tap:s://axxv...o orroorrg//ss.. atag//bb/22555 13600.
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Ag- gressive driving with model predictive path integral control. In 2016 IEEE International Con- ference Robotics and Automation (ICRA), pp. 1433-1440, 2016. doi: 10.1109//CRA.2016 7487277.
David A Winter. Biomechanics and motor control of human gait: normal, elderly and pathological. 1991.
Isabell Wochner, Pierre Schumacher, Georg Martius, Dieter Büchler, Syn Schmitt, and Daniel Haeu- fle. Learning with muscles: Benefits for data-efficiency and robustness in anthropomorphic tasks. In Conference on Robot Learning, pp. 1178-1188, PMLR, 2023.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Frez, Leonard Hasenclever, Jan Humplik, al. Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08644 2023.
Chenhui Zuo, Kaibo He, Jing Shao, and Yanan Sui. Self model for embodied intelligence: Mod- eling full-body human museuloskeletal system and locomotion control with hierarchical low- dimensional representation. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 13062-1306s 13069, 2024.
12

---

Published as a conference paper at ICLR 2025
A NEURO-MUSCLE DYNAMICS
We use the muscle-tendon units in MuJoCo as our actuator. The input control signal of muscle- tendon units is the neural excitation, denoted as The muscle activation, denoted act, is caleu- lated by a first-order nonlinear filter as follows:
Oact act Ōt T(u, act)
( Fact (0.5 + 1.5 act)
act u S act
T T(, act)
[ Tdeact/ (0.5 + 1.5 act)
where Tact and represent the time constants for activation and deactivation latency, with default values of 10 ms and 40 ms. where r(u, a) is the the effective time constant (Millard et al., 2013), which have been smoothed using sigmoid function.
The force produced by a single muscle-tendon unit is given by:
fm = fmx [F(()) F.(v)) a a Fç(1)
where fmax is the maximum isometric muscle force, and a, I, and v represent the activation, nor- malized length, and normalized velocity the muscle, respectively. The term (!(() accounts for the passive force-length relationship, and the terms (I) and (v) are the force-length and force- velocity functions, which have been fitted using data from biomechanical experiments (Millard al., 2013).
We use the following major joint positions that determine the whole-body posture: hip (6) knee (2), ankle (2), subtalar (4), spinal (9), shoulder (6), elbow (2), and wrist (6).
The output range of the reinforcement learning policy is typically [-1, 1], and it then normalized to [0, 1] in order to control the musculoskeletal system. We use the following equation to normalize action of the policy, which widely used MyoSuite environments.
+ (..
The reward design is follows:
reward rewardheatth cOStasks
where rewardhealch is the healthy reward given in each step, We subtract costasks and add it to rewardheathh to ensure that the reward remains positive. We find that the original cost weight in the cost function is sufficient for the reinforcement learning algorithm learn effectively, so we adopt the same weight used in the cost function.
B TASK SETTINGS
The common objective terms are defined as follows:
Height. This item limits the difference between the distance from the model's head to its feet and the desired height, encouraging the model to stand.
Cheight = Hnead
Hfcet
Hurget
where Hheal is the head height, Hise is the average height of the four feet, and Hunget is the target height as designed parameter for each task
Upright. This term encourages the character to maintain an upright posture.
Capright |[1 (1- - kap kpetvis) + (1 (1 kup knead) 0.1 1.1(1-kuu)) kup k + 0.1 kup Refwox)
where kmp is the up direction vector of the world coordinate, and Anants Anwnn Rypevics Antiont Ryteo are the up vectors of the body coordinate for the head, torso, pelvis, left foot, and right foot respectively.
Balance. This term encourages keeping the center of mass above the support polygon formed by the feet.
Oalance
COMMy
Feetsy
13

---

Published as a conference paper at ICLR 2025
where COMsy is the horizontal projection of the center of mass, and Feetzy is the average horizontal position of the feet.
Forward velocity. This term encourages maintaining specific forward velocity.
Ctt VCOM hioowardd
Utarget
where VCOM is the center of mass velocity, ktorward is the forward direction vector, and Uarret is the target velocity as a designed parameter for each task.
Forward angle. This term discourages sideways motion.
Cudir
VCOM
(VCoM Rrorward kitorward
2
Pelvis forward. This term encourages the character to face forward.
Co kforward kiptvis)
where kipetvis is the forward direction of the pelvis.
Joint velocity. This term penalizes excessive joint velocities.
Cj+ ||1l
Joint position. This term penalizes extreme joint positions.
cost cost = ||q/|2
Feet cross. This term discourages crossing of the feet and maintains proper leg alignment.
C min knip Rrreet 0.15)
+ min(O, (0, knip hice 0.15) + min (0, Anip hunce 0.15)
where Raips hisees Ruses krnee are the direction vector between hip joints, feet centers, toes and knee joints.
The cost function design of each task is as follows:
Stand
Cstand =100(Cneighr + Cupright + Cbalance)
10C 10C 100Cbr + 0.01C Cp
H Frgget 1.55
barget 0
Walk
Ostand 100(CCheigg Cupright + Chalance
+ 10Cst + 10C + 1000 + 50pp + 50C
Htarget 1.55
Utarget = 1
Rough
Ostand (Oneiiigh + Cupright + Chulance
+ 10Cst + 10C 1000br 2Cip 500
Husget 1.55
Uraret 0.5
14

---

Published as a conference paper at ICLR 2025
Slope
Ostand hheight + Cupright + balance
+ 10C, + 10Cydir + 1000br + 2Cpp + 50C
Hurget 1.5
Utarget = 0.5
Stair
Cstand 1heiChei + Cupright + Oullance
+ 100,r + 10C + 1000 + 2C + 500k
H Hurget 1.5
Vtarget = 0.5
Catair min (orwanlleet H sairr 0)l
where Sauir encourages raising leg before the stair.
For arm musculoskeletal manipulation task, The cost function is adopted from the MPJC Allegro Task'.
For soccer task, adopt the same cost function as the MJPC Humanoid Track task? without using the control and joint velocity term.
C ADDITIONAL EXPERIMENT RESULTS
The video and figures of our experiment is demonstrated in our project page.
C.I PLANNING UNDER UNCERTAIN MODEL.
We conducted additional experiment demonstrate that MPC² can effectively handle models with uncertainty. Following the implementation in sh MPC, we introduced perturbations to the 'rollout' model during planning by applying Gaussian random forces torques to each body of the model at every timestep. Figure 9 shows that MPC² maintains its control performance until the force standard deviation increases to 8 N or N-m, demonstrating the capability of planning with uncertain model. We consider states from the "reality' model help correct the errors caused by uncertain model during planning.
C.2 CONTROL OVER OSTRICI MODELS
In Video W13-W14, we demonstrate that MPC² successfully achieves stable control for ostrich musculoskeletal models using the same controller applied the full-body human model. Notably, the cost function used for the ostrich model is identical to that used for human walking. This high- lights 2 ability to perform planning across systems with varying morphologies without re- quiring training, cost function tuning, or controller parameter adjustments.
C.3 AUTOMATIC COST FUNCTION DESIGN
We consider cost function design with MPC² is more easier and efficient than reward engineering with DRL for its fast control generation combined with black-box function optimizer. We further demonstrate cost function optimization in both human and ostrich model, with cost function opti- mization results shown in 10.
As shown in Video W14-W15, starting with same cost function terms and weights as human walk- ing, we utilized Bayesian optimization in weight tuning, improving the walking speed of the human
htttp::/gitthbb.co tttb//gittthhcoooo googl deepmt httpe//gggaaggoooooooooo///////r
all llgerro / / www. www./oog//oooooooooooooooooooo le deepm nd nd/mu http:////aooooo/////////////////
humanoid/tracking
15

---

Published as a conference paper at ICLR 2025
10
10
Walk Distance (m)
Stand time (s)
7
0
2
4
8
10
0
2
4
8
10
Perturbation std during planning (N Nxm)
Perturbation std during planning (N Nxm)
(a) Stand
(b) Walk
Figure 9: Performance of MPC² under uncertain planning model
from 0.79 m/s to 1.24m/s, and the walking speed of the ostrich from 0.90 m/s to 2.08m/s without manual tuning.
1.2
Metion Tracking Error
-300
Forward Speed
Forward Speed
1.8
-350
1.6
-400
1.0
1.4
0.9
1.2
-500
-550
0.8
25
50
75
100
20
40
20 Iteration
40
Iteration
Iteration
Soccer
Human Walk
(c) Ostrich Walk
Figure 10: Cost function optimization
C.4 CENTER OF MASS POLYGON SUPPORT
In the Video W16-W17, we plot the centre of mass polygon support during walking for MPC² and DynSyn. We observe that MPC² is able to maintain larger polygon support compared to DynSyn, enhancing the stability during walking.
C.5 PERFORMANCES OF DRL AND MPC BASELINES
We ran MPO, the RL baseline in Wochner al. (2023), and its succeeded work, DEP-RL (Schu- macher et al., 2022), in the standing and walking task. However, as shown in Video W24-W27,v observe that these two method failed to achieve stable standing or running over full-body model with same training steps (5e7) as DynSyn, our DRL baseline in the main paper.
We evaluated six MPC baselines provided by Mujoco MPC, which include both gradient-based methods (Gradient Descent, iLQG, iLQS) and sampling- based methods (Cross Entropy, Robust Sampling, and Sample Gradient). As shown in Video W18-W23 Table none these methods succeeded in achieving walking with the full-body musculoskeletal model. For non-sampling-bart MPC methods, long planning times are required due to the computational demands of deriving the high-dimensional system dynamics, which impedes real-time decision-man making, For sampling-bai -baed MPC methods, the high- digh--imnnsional action space makes it challenging to sample effective control sequences.
16

---

Published as a conference paper at ICLR 2025
(a)
(b)
Figure 11: Automatic cost function design for improving the walking speed. (a) Optimized control sequences of MPC over human model. (b) Optimized control sequences of MPC over ostrich model.
Methou
MPC
Gradient Descent
iLQG
MPPI
Cross Entropy
Robust Sampling
Sample Gradient
Walk distance
9.50 = 0.19
0.00 + 0.00
0.00 + 0.00
2.05
0.40
1.18 0.23
1.11 + 0.15
1.21 = 0.24
Table I: Walking distance of MPC baselines
D BASELINES
We compare our algorithm with the reinforcement learning algorithms DynSyn. DynSyn adopt SAC as the basic algorithm and use the DRL framework Stable baselines3. We set control frequency to 10 simulation steps, which can significantly increase the sample efficiency of the reinforcement learning algorithm. All the parameters are reported in the original papers, and we use the same parameters for models with similar complexity. Algorithm hyperparameters are summarized in Table 2.
17

---

Published as a conference paper at ICLR 2025
Task
Algorithm
Parameter
Stand
Walk
Learning rate
linear schedule((.001)
Batch size
256
Buffer size
1c6
Warmup steps
100
Discount factor
0.98
Soft update coeff.
2
Train frequency (steps)
I
Gradient steps
4
SAC
Target update interval
Environment number
112
Entropy coeff.
auto
Target entropy
auto
Policy hiddens
[512, 300]
Q hiddens
[512, 300]
Activation
ReLU
Training steps
le7
Control Amplitude
5
Trajectory steps
5e5
DynSyn
Number of groups
100
aD
3c7
kD
5e-9
Table 2: Parameters of SAC and DynSyn
18


Reviews:
[{"review_id": "Yf3oGrrMpj", "reviewer": "Reviewer_4m7j", "summary": "The paper presents a hierarchical model predictive control approach for a  700-dimensional musculoskeletal system, where a modified version of MPPI is used as a higher-level controller to output target postures that are tracked by a lower-level proportional controller. The gains of the lower-level proportional controller are set using a morphology-aware function based on the Jacobian matrices of the current state and the target posture. The method is evaluated on several tasks and the paper reports ablations of the method.", "questions": "- Apart from only being near real-time at execution time? what are other limitations of the method?\n- Is the method robust to perturbations? Can the humanoid keep walking when unknown external forces are applied? \n- Is a different cost function used to obtain a controller for each task? If so, is there a systematic way to unify/merge cost functions to obtain a controller that can perform multiple tasks at the same time? \n- One of the motivations for musculoskeletal systems is its robustness in case of actuator failure. How does the method’s performance degrade in the presence of actuator failures  (when only a subset of tendons can be actuated)? Such evaluation would further highlight the benefits of the presented approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The presented method is simple, can handle the high dimensionality of the action space associated with muscle-actuated systems, and performs well on several challenging tasks, such as walking on rough terrain, walking on slopes, or over stairs. \n- Fast cost-design iteration time to evaluate/generate new controllers compared to Deep RL methods.\n- The paper is well-written overall and the ablation studies justify different parts of the proposed method.", "weaknesses": "- I have concerns that the conference might not be a good fit for the paper. I struggle to classify the presented hierarchical sampling-based approach for control as an area or subfield of machine learning. Especially, considering that MPPI and many of its variants have been traditionally presented in robotics conferences. If the authors can clarify, how the conference is a good fit for the paper or if this is not a concern for any other reviewer, I will not oppose the acceptance of this paper in an exercise of  \" broadening definitions of originality and significance\".\n\n- The paper would benefit from making a clear distinction between the iteration time to create a controller and the time needed to execute a controller at test time. L121 states “Compared to DRL, model predictive control allows for real-time control”.  It is well known that training DRL policies can be time-consuming, however, after the training stage, most DRL policies can be queried/executed in the order of milliseconds thus effectively allowing for real-time control, unlike the presented method which is only suitable for near real-time deployment/execution.\n\n\n- Probably due to the challenging nature of the setting, only a few baselines are provided and the performance of the baselines is not very competitive.  AHAC [1] might provide a stronger baseline for the comparisons.  AHAC has also been tested on a muscle-actuated humanoid task and has better performance than SAC and PPO on such tasks.\n   - [1] Ignat Georgiev et al.  Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EA0bonPfQx", "reviewer": "Reviewer_VP3V", "summary": "## Summary:\nThe paper introduces a control algorithm called Model Predictive Control with Morphology-aware Proportional Control (MPC2) to address\n challenges associated with the real-time control of high-dimensional musculoskeletal systems. The problem is complex due to the inherent complexity of over-actuated nature involving large state and action spaces. The authors proposed a Sampling based MPC with Morphology-aware Proportional Controller for actuator co-ordination (MPC2). They show that the methodology can produce stable motions without any training/data-driven steps, resulting in zero-shot motion control. They used Deep learning based approach as a baseline to compare their method for various tasks such as standing, walking on flat/rough surface, slope, stairs and kicking a ball.   \n\n## Methodology:\nMPC2 combines two main strategies:\n1. High-Level Planning via Model Predictive Control (MPC): This component of MPC2 uses a sampling-based model predictive controller to plan target postures for the system. It operates by forecasting the desired postures over a finite horizon and computing optimal control actions to achieve these postures.\n2. Low-Level Control via Morphology-aware Proportional Control: This controller works at the actuator level, coordinating the actuators to reach the target joint positions determined by the high-level planner. It adjusts the actuators dynamically based on the morphology of the system, ensuring efficient and effective movement control.\n\n## Key Contributions and Findings:\n1. Training-Free Control: Unlike Deep Reinforcement Learning based methods that require extensive training, MPC2 enables zero-shot control, meaning it can control the system effectively without any prior training on specific tasks.\n2. Adaptability to Diverse Tasks: The algorithm was tested on various movement tasks, including standing, walking on different terrains, and sports motion imitation. These experiments demonstrated its capability to adapt to a wide range of activities without the need for manual tuning or reward engineering.\n3. Real-Time Performance: One of the significant advancements of MPC2 is its ability to perform near real-time control. This is due to their Hierarchical control strategy, reducing of the action space for MPC (dz = 37) instead of the full control over all 700 tendons.\n4. Efficient Optimization: The paper shows that MPC2 can optimize its cost function design automatically through Bayesian optimization. This feature significantly reduces the manual effort typically needed in traditional control systems.\n\n## Experimental Validation:\nThe effectiveness of MPC2 was validated through a series of experiments conducted on a simulated 700-actuator musculoskeletal model. The model was tested across various scenarios that require high levels of coordination and control precision. Results from these experiments confirm the robustness and versatility of MPC2, outperforming existing deep reinforcement learning-based methods in terms of speed and adaptability. Their website provides video demonstrations of MPC2’s real-time control capabilities across various movement tasks. These videos effectively illustrate the adaptability and robustness of the proposed algorithm. They show MPC2 handling complex movements like walking on uneven terrain, climbing stairs, and executing sports motions (e.g., kicking a ball) with stable, smooth coordination across multiple actuators. These visual results add strong qualitative evidence to the paper’s claims about MPC2’s ability to perform zero-shot control without training and adapt to different motion tasks with high precision and reliability.\n\n## Conclusion:\nThe development of MPC2 marks a significant step forward in the field of robotics and biomechanics, offering a powerful tool for the high-dimensional control of complex systems. This method holds potential not only for improving robotic system performance but also for advancing human-related applications such as prosthetics and rehabilitation robotics, where adaptive, real-time control is crucial.", "questions": "1. Could the authors provide additional insights into the computational requirements for real-time applications of MPC2?\n2. Could you make the experimental more elaborative towards quantitative results ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper definitely presents some key strengths:\n1. Originality: While breaking down the high-dimensional control/planning problem into multiple stages is a very know concept, the paper presented first of its kind to integrate MPPI with Morphology Aware controller. The paper thus provides a new solution for high dimensional motion control.\n2. Quality: The paper provides a thorough experimental validation in Mujoco simulation. On technical aspect, the use of Bayesian optimization for automated cost function tuning streamlined the traditionally labor-intensive process of reward engineering. \n3. Clarity: The paper is a very well written providing curriculum approach to MPC. The provided code is well structured and documented.\n4. Significance: Though the results were presented only in simulation and qualitatively seemed far from applicable to real applications, the approach presented has significant practical impact on real-time controls.", "weaknesses": "I found several weak points in relation to the experiments and validation of the proposed method (it could be also a result of my background from classical control theory). \n\n1. Lack of quantitative stability metrics: The paper lacks in presenting the performance of their approach on key stability metrics used across the humanoids community like centre of mass polygon support, energy efficiency, etc. Additionally, integrating these metrics into the MPC cost function could enhance the results significantly. (De Viragh, Yvain, et al. \"Trajectory optimization for wheeled-legged quadrupedal robots using linearized zmp constraints.\" IEEE Robotics and Automation Letters 4.2 (2019): 1633-1640.)\n\n2. Experimental results: \n    1. Lacks quantitative discussion in comparison with RL based methods (the stability/energy metrics would have helped here).\n    2. The paper’s comparative analysis primarily uses a DRL-based baseline, but the implementation quality of this baseline appears to be lacking. The RL approach (Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks) should have been compared. \n\n3. Figure 5.a. mentions \"Best Objective Value\" but it isnt discussed in the paper, what that means and why it is significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kTETwhv3x", "reviewer": "Reviewer_SAwp", "summary": "This paper proposes a sampling based hierarchical MPC controller that uses the known model of a simulated high dimensional musculoskeletal humanoid system and achieves good control performance across various tasks without learning a policy representation. This is to be contrasted with deep RL methods that would require a lot of time to train a policy for each task. The hierarchical MPC controller samples target positions that are then used by a morphology aware proportional controller to compute control signals for each DoF. Simulations show that the proposed method works well in various control tasks, where other MPC controllers are claimed to fail invariably.\n\n===== POST-REBUTTAL EDIT =====\n\nI thank the authors for their hard work and the comprehensive rebuttal. The rebuttal has answered some of my concerns and I think the paper could be accepted, it does present a contribution to the simulation/control of high dimensional systems, albeit in a modest way. I raised my score to six as a result. \n\nHere are some more comments based on the rebuttal:\n- The github page with extensive results was quite interesting, I am hoping that the authors will integrate the results with the main paper, as I think the main paper lacks the wide coverage of results that the github page presents.\n- The gait that is learned with MPC^2 also does not seem very natural. Is that the reason why MPC^2 also fails to stabilize after some time I wonder? How would you learn more natural gaits (from a human walking perspective) with e.g. black-box optimization? This could be mentioned in the discussion/conclusions.\n- The paper in the end does not present a learning component. I think the discussion/conclusions should discuss ways to incorporate learning to improve some of the failure modes (gait is not natural, model too complex, MPC not fast enough for real time, more complex scenarios not covered etc.)\n- I am still confused about the 'instant rollout' feature, if I understand instant rollout correctly, every MPC algorithm should incorporate instant rollout at some frequency (maybe different from the control frequency) otherwise it wouldn't be corrective. This means that MPC algorithm (running in parallel to the simulation) should always get current states of the robot from the simulation. However from the paper it seems like that is not the default, any reason why that is so? Is it because in simulation you know the model exactly so you can refrain from doing so?", "questions": "Some questions and minor comments are given below:\n\n- Missing word in Abstract: e.g. 'it suffers from being computationally-intensive ...'\n- Figure 2 is not self explanatory. Also I am confused by the boxes, the 'rollout' box inside big gray box is the access to simulator, which actually replaces the 'reality' of the leftmost box with the skeleton. A more detailed explanation would be appropriate here to ease the introduction.\n- From the introduction or the abstract it is not clear how you acquire the 'model' for the Model Predictive Controller that you use. It should be mentioned that you're considering the simulated environment where you have access to / know the model of the robot. In general DRL does not need a model, so the statements should be modified accordingly (RL methods are more general, although sometimes less sample-efficient etc.)\n- There does not seem to be any 'learning' in the proposed method, hence it is not clear how to use it in uncertain scenarios, where, e.g., we need to learn a model (or a policy, or both).\n- \"thus poses a significant bottleneck on the iteration speed of reward engineering.\" It is not clear up till the sentence what reward engineering is (or what it can be used for) from the introduction, nor the connections of the paper to reward engineering clear.\n- Zakka et al. includes DRL in the title, so may not be the right reference for MPC methods vs. DRL in section 2.2.\n- Are torque-driven humanoids (Meser et al.) less complex than your system? Section 2.2 does not mention the relation. In fact a 'musculoskeletal' system is not defined.\n- notation f is used both for dynamics and for forces (which are part of dynamics, so it's a bit problematic).\n- line 160: the cost function used to update MPPI does not seem to be 'cumulative' but only over the horizon of size H.\n- line 170: NOT always a local approximation: when H is small, the optimized commands can even cause some systems to be unstable. Moreover you claim f_hat is not equal to f (which is typically the case) hence I think the term \"local approximation\" is not precise here.\n- asynchronized -> asynchronous?\n- missing reference for sampling-based MPC methods (there is one reference for an application paper to humanoids)\n- again for MPPI, I think the paper is missing a more general reference that introduces the method (the reference is again an application paper it seems)\n- \\hat{u}^{\\theta} notation conflicts with the previous notation where n was used to refer to rollout number.\n- I failed to get Algorithm 1, although the text leading up to it was fairly clear. Some terms are not explained, like M_pos, R_MP etc., it's not clear how the initial sigma plays a role, etc.\n- The distinction between the terms planning iteration vs. rollout becomes confusing at times, would be nice to define them or illustrate them over the figures.\n- the need to introduce instant rollout is not clear to me. why shouldn't the rollout always start from the current state (hence \\mu_0 = M_pos(s_0) in your notation). M_pos was explained later in line 276 [and in fact I would simply use p_t as the pose rather than introducing a mask notation, which conflicts with the mass matrix M(q)]. In general this seems to be an re-occurring problem in the paper: properties of certain features of the method are mentioned *before* introducing that feature.\n- \"While the original Model Predictive Path Integral (MPPI) can be directly employed as a high-level planner for target positions, we find that it lacks the ability to respond quickly to rapidly changing states, such as when the agent is falling. \" It wasn't clear to me if this is an argument for the low-level proportional controller? If so why not mention it *after* introducing the low level controller?\n- the low level controller seems to be state dependent in eq (11). Hence strictly speaking, it is not a (constant-gain) proportional controller.\n- where is the standard error shown in Figure 4?\n- shouldn't it be max instead of min in eq. 9?\n- it is not clear how eq. (11) resolves the problem of tuning of the gain vector K.\n- the design of the low-level controller seems to be very heuristics-driven: one cannot conclude that it will work for arbitrary morphologies/robots, especially given that the results are only given for a particular robot. This seems to be part of a bigger problem, in that, the paper does not propose any learning components to replace DRL or any other state of the art method. This potential lack of generality is worrying, and should be improved.\n- \"While no previous control methods have demonstrated success in whole-body musculoskeletal systems for these tasks, MPC2 exhibits consistent and stable control performance across various tasks, enabling navigation over different terrain conditions.\" It would be nice to support this claim by comparing against various MPC/control approaches.\n- What are the \"DRL-based methods\" compared against in section 5.2?\n- what are the actual values of the \"weights of the position error terms for different body parts\" that are optimized? is there an interesting trend, such as significant deviation from uniform values? does it correspond to the last figure in the appendix?\n- it's not clear to me what the 'instant rollout' means\n- \"Figure 6(b) shows that MPC2 significantly outperforms both the constant gain and PD control variants.\" How are these variants computed? Can you not tune/learn the parameters of the PD (or PID) controller such that it improves the walking distance significantly in Figure 6b?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper shows that a hierarchical and sampling-based implementation of a model-based predictive controller (MPC) can control in simulation a high-dimensional musculoskeletal system for the first time. This is to be contrasted with DRL methods that take a very long time to optimize policies, moreover they need to typically be re-run for each task. In that sense there's a clear contribution to the simulation literature on how to control such high dimensional systems.", "weaknesses": "I think the paper has some good contributions to control and simulation of high-dimensional systems as mentioned above, yet there are several weaknesses (some of which can be addressed in the rebuttal phase hopefully):\n\n- There is a very extensive literature on MPC algorithms and analysis, yet the paper does not compare the proposed approach to any other MPC algorithm. Would all of the MPC methods proposed in the literature fail for this complex dynamics case? It is not clear from the text. The tradeoffs of using sampling as opposed to non-sampling based methods are also not discussed.\n- I'm not sure if ICLR is the right venue for this paper. Even if we agree that restricting ICLR to papers that 'learn representations' would be too strict, still we're missing a more learning-focus to the paper. This doesn't mean the paper is not well-written or that it is not sound, but I think the paper could benefit more from being targeted to a more controls conference. As of now the learning in the paper is restricted to 'inverting' (without a negative connotation) a known-and-very-complex model. However it would be more interesting for the ML community if the paper would consider not only learning a policy in this particular case, but also extending learning to cover e.g. learning the lower-level policy, learning the higher level policy (if we treat MPPI as a learning method) for different robots, etc.\n\nSee also the comments in the Questions section below for more details and questions. Overall I think the paper needs a significant extension of the experiments and ablations in order to justify its conclusions. The heuristic nature of their design decisions (low-level controller, sampling vs non-sampling, effect of sampling size, effect of horizon size, etc.) need to be discussed in more detail and compared against various alternatives. Experiments should also be presented together more concisely, more comprehensively in a table or in a figure (e.g. replacing half-page illustrations of skeletons).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjSrYWYXwv", "reviewer": "Reviewer_Vffr", "summary": "This paper presents a system for using a sampling-based model predictive controller to control a musculoskeletal character with a large number of actuators. Key components include: (1) reducing the number of variables using only a single posture target as the control signal. (2) A modification to the sampling strategy that also samples around the current pose.", "questions": "Morphology typically means sizes and shapes of the character, so I was originally expecting to see the controller can adapt to different shapes or sizes, but it seems not the case. I suggest changing the word morphology to something more accurate, e.g., configuration.\n\nI don't think it is fair to compare it to a reinforcement learning method when the action space used is different. It would be better if both algorithms used the same action space.\n\nIt seems like instant rollout is only useful for the one very specific scenario discussed. Is it necessary for other scenarios?\n\nIt would be great if videos of the results can be shared.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is easy to understand and is shown to efficiently generate control signals to do various tasks on a high-dimensional simulated humanoid. \n\nThe method allows for the rapid iteration of reward/cost function design.\n\nIt is surprising to me that MPPI works well when the sampling signal is just a fixed repeated pose. It could be an interesting observation to the control community.", "weaknesses": "There are no videos of the result, so it is hard to judge the quality of the synthesized controller.\n\nThere is not much contribution related to machine learning methodology or insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)