HWC-Loco: A Hierarchical Whole-Body Control Approach to Robust Humanoid Locomotion

## Abstract

Abstract Humanoid robots, capable of assuming human roles in various workplaces, have become essential to embodied intelligence. However, as robots with complex physical structures, learning a control model that can operate robustly across diverse environments remains inherently challenging, particularly under the discrepancies between training and deployment environments. In this study, we propose HWC-Loco, a robust whole-body control algorithm tailored for humanoid locomotion tasks. By reformulating policy learning as a robust optimization problem, HWC-Loco explicitly learns to recover from safety-critical scenarios. While prioritizing safety guarantees, overly conservative behavior can compromise the robot’s ability to complete the given tasks. To tackle this challenge, HWC-Loco leverages a hierarchical policy for robust control. This policy can dynamically resolve the trade-off between goal-tracking and safety recovery, guided by human behavior norms and dynamic constraints. To evaluate the performance of HWC-Loco, we conduct extensive comparisons against state-of-the-art humanoid control models, demonstrating HWC-Loco’s superior performance across diverse terrains, robot structures, and locomotion tasks under both simulated and real-world environments. Our project page is available at HWC-Loco .

## 1 Introduction

Figure: Figure 1: An example of recovering from a Hard Kick: The humanoid robot withstands external disturbance by automatically detecting hazardous states and adjusting its motion to regain stability.
Refer to caption: x1.png

Humanoid robots, with a physical structure resembling that of humans, can be seamlessly integrated into humans’ workspace and take on roles in completing different tasks [1].
This capability makes them reliable embodiments of artificial intelligence in various forms [2]. In recent years, with advancements in both hardware capabilities and control algorithms, humanoid robots have become an increasingly significant type of robot with a growing impact on practical applications across various environments, such as factories, homes, and offices [3, 4, 5].

In the development of humanoid control, classic methods commonly rely on model-based optimization [6, 7, 8, 9, 10]. However, these methods require precise and comprehensive modeling of the robot’s structure, kinematics, and dynamics across different environments. In practice, obtaining such data requires significant computational resources or manual effort. These limitations significantly influence the scalability of these approaches. To develop an end-to-end solution with promising generalizability, recent studies [11, 12, 13, 14, 15] adopted Reinforcement Learning (RL) methods by training a neural model to control based on human demonstrations and interactions with the environment.

While these learning-based approaches show potential for effective control policies across various tasks, they are typically trained in simulated environments, which often differ considerably from the deploying environment. To bridge the Simulation-to-Real (Sim2Real) gap, previous works often introduce additional regularization to constrain the robot’s movements or employ domain randomization to account for variations in the robot’s physical properties [16, 17, 18]
. However, excessive regularization can greatly affect the efficiency of control policy, and unstructured randomization often fails to capture safety-critical patterns in real-world applications.

To develop a reliable locomotion policy capable of generalizing from the training to the deployment environment, we propose formulating policy optimization as a robust optimization problem under misspecified environmental dynamics.
However, traditional robust optimization primarily focuses on worst-case control, requiring the policy to ensure safety under the most adverse environmental dynamics within an uncertainty set. Under this context, the learned policy tends to be overly conservative, inducing sub-optimal control performance in terms of finishing the given task.

In this paper, to learn a control policy that can dynamically solve the trade-off between task performance and maintaining safety under different deployment environments, we present Hierarchical Whole-body Control for Robust
Humanoid Locomotion (HWC-Loco).
Figure [1](https://arxiv.org/html/2503.00923v3#S1.F1) illustrates an example of HWC-Loco, where a high-level planning policy detects hazardous states and seamlessly switches from the goal-tracking policy to a safety-recovery policy.

The goal-tracking policy is designed to follow task-specific commands (e.g., moving velocity). Additionally, we ensure that the learned policy can effectively adhere to humanoid behavioral patterns by ensuring its distributional alignment with human motions.
Note that this policy primarily focuses on ensuring task completion performance, without addressing safety issues that may arise from potential mismatches with the deployment environment.

The safety-recovery policy aims at ensuring safety across various deployment environments. To address safety-critical scenarios during deployment, we construct an uncertainty set by designing an extreme-case estimation mechanism. Within these constructed scenarios, we propose a dynamic constraint guided by the Zero Moment Point (ZMP) to ensure stability under various disturbances.

While the goal-tracking and safety-recovery policies are primarily designed to enhance task performance and safety control, respectively, they lack a coordination mechanism to effectively control the trade-off between these objectives. To address this limitation, we propose a high-level planning policy that dynamically selects which policy to activate based on the scenario.

We conduct extensive experiments to evaluate the performance of HWC-Loco from four fundamental perspectives: 1) effectiveness, demonstrated across diverse terrains, 2) robustness, evaluated under varying scales of disturbances, 3) naturalness, exhibited in its ability to imitate human behavioral norms, 4) scalability evaluated in different robot embodiment and motion tracking tasks. These validations are performed in both simulated and real-world settings, demonstrating HWC-Loco as a foundational advancement for achieving reliable humanoid locomotion in safety-critical scenarios.

## 2 Related Work

Learning-based Legged Robot Locomotion. Various learning-based methods have been proposed for legged locomotion, such as quadrupedal locomotion [17, 19, 20, 18, 21, 22, 23] and bipedal locomotion [24, 25, 26, 27]. Despite their success, these approaches cannot be directly applied to humanoid robots because of the complex physical structure and the higher degrees of freedom.
Recent advances in hardware and learning methods (e.g., RL algorithms) have enabled humanoid robots to move through diverse and complex environments.
Some humanoid control policy navigates across diverse terrains using only proprioceptive information [12, 28, 15]. Others enhance performance by incorporating additional sensors, such as vision or LiDAR, to collect detailed environmental data, enabling robots to perform complex tasks like stair climbing and parkour jumps [11, 13, 29]. While these approaches have demonstrated promising locomotion performance, they lack mechanisms to handle safety-critical scenarios that humanoid robots may encounter during real-world deployment.

Whole-body control for humanoid robots.
Learning effective whole-body control remains a central challenge for humanoid robotics. Prior works [30, 31, 32] have demonstrated impressive results in simulated avatars, showcasing the feasibility of learning-based control strategies. More recently, studies such as [33, 14, 34, 35, 36, 37, 38, 39] have enabled realistic robots to imitate complex human-like motions, such as boxing and dancing by leveraging motion priors from human demonstrations. Additionally, previous methods employed multi-stage reward designs to accomplish complex tasks [40, 41]. However, the success of these methods often relies on intricate reward shaping and struggles to generalize across different humanoid robots, particularly when motion priors are unavailable.

## 3 Problem Formulation

Learning Environment.
We formulate the environment of the humanoid locomotion task with a
Partially Observable Markov Decision Process (POMDP) $M=(\mathcal{S},\mathcal{A},\mathcal{O},P_{\mathcal{T}},r,\mu_{0},\gamma)$, where:
1) Within the state space $\mathcal{S}$, a state $s\in\mathcal{S}$ records the environmental information and the robot’s internal states.
2) $\mathcal{A}$ denotes the action space, and action $a\in\mathcal{A}$ denotes the target joint angles that a PD controller uses to actuate the degrees of freedom (DOF).
3) $o\in\mathcal{O}$ denotes an observation, which provides partial information about the state of the agent due to sensory limitations and environmental uncertainty. At a time step $t$, $\boldsymbol{o}_{t}$ includes velocity command $\boldsymbol{v}_{t}^{cmd}\in\mathbb{R}^{3}$ and proprioception $\boldsymbol{o}_{t}^{p}$ that records a humanoid’s internal state. Appendix [A.2](https://arxiv.org/html/2503.00923v3#A1.SS2) introduces the details.
By incorporating the temporal observations
$\boldsymbol{o}_{t}^{H}=[\boldsymbol{o}_{t},\boldsymbol{o}_{t-1},\cdots,
\boldsymbol{o}_{t-H}]$,
the control model can summarize a state as $s_{t}=[\boldsymbol{o}_{t}^{H},\mathbb{P}_{t}]$, where $\mathbb{P}_{t}$ is the privileged information which the robot can’t access in realistic deployment, including base velocity $v_{t}$, terrain height, external disturbance and ZMP features (see Section [4.3](https://arxiv.org/html/2503.00923v3#S4.SS3)).
4) $P_{\mathcal{T}}\in\Delta^{\mathcal{S}}_{\mathcal{S}\times\mathcal{A}}$ denotes the transition function as a mapping from state-action pairs to a distribution of future states.
5) $r$ denotes the reward functions, which typically consist of penalty, regularization, and task rewards. These reward signals significantly influence the optimality of the control policy, for which we provide a detailed introduction in the following.
6) $\mu_{0}\in\Delta^{\mathcal{S}}$ denotes the initial state distribution.
7) $\gamma\in(0,1]$ denotes the discounting factor.
The robot’s policy stops at a terminating state $\tilde{s}$, and the corresponding terminating time is denoted as $T\in(0,\infty)$.

Under this POMDP, our goal is to learn a policy $\pi\in\Delta^{\mathcal{A}}_{\mathcal{S}}$ according to the objective as follows:

$$ $\displaystyle\max_{\pi}\mathcal{J}(\pi,M)=\max_{\pi}\mathbb{E}_{\mu_{0},p_{ \mathcal{T}},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\left[\beta_{\mathfrak{T}} r_{\mathfrak{T}}(s_{t},a_{t})+\beta_{\mathfrak{P}}r_{\mathrm{P}}(s_{t},a_{t})+ \beta_{\mathfrak{R}}r_{\mathfrak{R}}(s_{t},a_{t})\right]\right]$ (1) $$

In the task of humanoid locomotion, the total rewards can be generally represented as a weighted combination of task rewards $r_{\mathfrak{T}}$, penalty rewards $r_{\mathrm{P}}$, and regularization rewards $r_{\mathfrak{R}}$ [14, 34, 11],
where the rewards have distinct semantic meanings:

- •
Task rewards $r_{\mathfrak{T}}$ assess how effectively the agents achieve the goals of the current task, such as tracking velocities [11], managing contacts [41], and producing expressive motions [33]. These task rewards are closely correlated with the optimality of the current control policy, making them an appropriate objective to maximize in this context.
- •
Penalty rewards $r_{\mathrm{P}}$ are used to prevent undesirable events. For example, in locomotion tasks, these penalties are applied when the humanoid falls
or when the robot violates dynamic constraints, such as torque limits and joint position limits. To mitigate these negative outcomes, recent studies [14, 34] have assigned a large weight $\beta_{\mathfrak{P}}$ to these rewards. While these weights inherently function as Lagrange multipliers [42], they are predefined and not subject to optimization. Consequently, a more effective approach would be to model the control problem as a constrained RL problem.
- •
Regularization rewards $r_{\mathfrak{R}}$ aligns the behavior of the humanoid with human preferences on motion styles or safe deployment.
However, accurately specifying these regularizations remains challenging since they are often task-dependent, context-sensitive, and inherently rooted in human experts’ experience [43].
As a result, instead of manually specifying regularizations, we align the robot’s motion with human preferences by imitating human behavior from motion datasets (e.g., AMASS dataset [44]), which contains rich signals about desirable behavior norms.

Constrained RL for Whole-Body Control. Inspired by the aforementioned analysis, we formulate the whole-body control locomotion objective for humanoid robots as follows:

$$ $\displaystyle\max_{\pi}\mathbb{E}_{\mu_{0},p_{\mathcal{T}},\pi}\left[\sum_{t=0 }^{\infty}\gamma^{t}r_{\mathfrak{T}}(s_{t},a_{t})\right]{s.t.}\;\mathcal{D}_{f }(\rho_{M}^{\pi^{E}}\|\rho_{M}^{\pi})\leq\epsilon_{f}\text{ and }\mathbb{E}_{ \tau\sim(\mu_{0},p_{\mathcal{T}},\pi)}\left[\phi(\tau)\right]\geq\epsilon_{\phi}$ (2) $$

where 1) we replace the hard regularization in $r_{\mathfrak{R}}$ with a mimic learning objective $\mathcal{D}_{f}(\rho_{M}^{\pi^{E}}\|\rho_{M}^{\pi})$ that constrains the distributional divergence between the learned policy’s occupancy measure $\rho_{M}^{\pi}$ and the expert policy’s occupancy measure $\rho_{M}^{\pi^{E}}$. In this study we implement $\mathcal{D}_{f}$ by Wasserstein distance (Section [4.1](https://arxiv.org/html/2503.00923v3#S4.SS1)).
2) Instead of relying on the penalty reward $r_{\mathrm{P}}$, we use $\phi$ to capture the feasibility of the current trajectory generated by the policy $\pi$ under the environment $M$.
Within the objective, only the task rewards
$r_{\mathfrak{T}}$ are subject to maximization; other desired characteristics of humanoid robots are captured by constraints learned from human motion datasets. More importantly, this approach effectively eliminates the need for regularization on the humanoid robot’s upper or lower pose, significantly enhancing the learning of whole-body locomotion policies.

Robust Locomotion in Humanoid Robot.
A common approach to learning humanoid locomotion policies involves training in a simulated environment before deploying the policies in a real-world setting. Due to the complexity of real-world environments, there is often a significant discrepancy between the training and deployment environments. In this study, we characterize this discrepancy by the mismatched POMDPs defined as follows:

###### Definition 3.1 .

(Mismatched POMDPs) We define a POMDP $M:=(\mathcal{S},\mathcal{A},P_{\mathcal{T}},\mathcal{O},r,\mu_{0},\gamma)$, to be mismatched with another POMDP $M^{\prime}:=(\mathcal{S},\mathcal{A},P_{\mathcal{T}}^{\prime},\mathcal{O},r,
\mu_{0},\gamma)$ if they differ only in transition functions (i.e., $P_{\mathcal{T}}\neq P_{\mathcal{T}}^{\prime}$). We define a set of mismatched POMDPs as $\boldsymbol{\mathcal{M}}_{m}=\{M^{P_{\mathcal{T}}},M^{P_{\mathcal{T}}^{\prime}
},\dots\}$, where their transition functions differ from each other.

Note that we follow previous studies [45] and primarily focus on the mismatch in transition dynamics during learning and deployment. The uncertainty set of transition function [46] is represented as:

$$ $\displaystyle\mathfrak{P}_{\alpha}^{L}=\{\alpha P_{\mathcal{T}}^{L}+(1-\alpha) \bar{P_{\mathcal{T}}},\;\forall{\bar{P_{\mathcal{T}}}\in\mathfrak{P}}\}$ (3) $$

where $\alpha$ specifies the scale of mismatch, $P_{\mathcal{T}}^{L}$ denotes the transition function in the learning environment and $\mathfrak{P}\subseteq\Delta^{\mathcal{S}}_{\mathcal{S}\times\mathcal{A}}$ represents the set of all candidate transition functions.
While other factors, such as divergence in state-action spaces and initial state distribution $\mu_{0}$, can influence the performance during deployment, we find that the majority of factors affecting Sim2Real performance in humanoid locomotion tasks can be characterized by the mismatch in transition dynamics.
For example, if $\bar{P_{\mathcal{T}}}$ denotes a Gaussian function, the underlying sensor noise can be captured by transitions in $\mathfrak{P}_{\alpha}^{L}$. Additionally, if $\bar{P_{\mathcal{T}}}$ denotes a projection from one spatial state to another, the variations of terrains in the deploying environment can be modeled by transitions in $\mathfrak{P}_{\alpha}^{L}$ [13].

To solve a robust RL problem, previous studies often consider a max-min objective $\max_{\pi}\min_{M\in\boldsymbol{\mathcal{M}}_{m}}\mathcal{J}(\pi,M)$ where $\mathcal{J}(\pi,M)$ denotes a standard RL objective ([1](https://arxiv.org/html/2503.00923v3#S3.E1)). With this objective, while the agent can learn a conservative policy to ensure worst-case control performance, this policy often compromises the control performance, making the humanoid less effective at tracking the given commands. To address this issue, we focus on ensuring a worst-case feasibility constraint, thereby reducing the influence of mismatched POMDP on maximizing task rewards.
Accordingly, we update the CRL objective ([2](https://arxiv.org/html/2503.00923v3#S3.E2)) to the robust humanoid locomotion objective represented as:

$$ $\displaystyle\max_{\pi}\min_{{\color[rgb]{0,0,1}\widehat{P_{\mathcal{T}}}}\in \mathfrak{P}_{\alpha}^{L}}\mathbb{E}_{\mu_{0},{\color[rgb]{1,0,0}P_{\mathcal{T }}^{L}},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{\mathfrak{T}}(s_{t},a_{t}) \right]{s.t.}\;\mathcal{D}_{f}(\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal{T}}^{L} }}}^{\pi^{E}}\|\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}}}^{\pi})\leq \epsilon_{f}\text{ and }\mathbb{E}_{\tau\sim(\mu_{0},{\color[rgb]{0,0,1} \widehat{P_{\mathcal{T}}}},\pi)}\left[\phi(\tau)\right]\geq\epsilon_{\phi}$ $$

where 1) $\mathfrak{P}_{\alpha}^{L}$ represents the set of mismatched dynamics from the learning environment dynamics ${\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}$ (defined in [3](https://arxiv.org/html/2503.00923v3#S3.E3)) and 2) $\rho_{M}^{\pi}(s,a)=(1-\gamma)\pi(a|s)\sum_{t=0}^{\infty}\gamma^{t}p(s_{t}=s|
\pi,M)$ defines the normalized occupancy measure of policy $\pi$ under the environment $M$.
Intuitively, we aim to learn a robust control policy that ensures feasibility across all mismatched transition functions ${\color[rgb]{0,0,1}\widehat{P_{\mathcal{T}}}}\in\mathfrak{P}_{\alpha}^{L}$,
thereby guaranteeing the safety of policy $\pi$. However, for the reward-maximizing and mimic learning objectives, we optimize the policy based on the transition dynamics in the learning environment ${\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}$, rather than concentrating solely on worst-case guarantees.

###### Definition 3.1 .

## 4 Hierarchical Whole-Body Control for Humanoid Robust Locomotion

Figure: Figure 2: Overview of HWC-Loco: The framework consists of two stages: (a) Training goal-tracking policy to effectively enable human-like locomotion across diverse terrains (Section [4.1](https://arxiv.org/html/2503.00923v3#S4.SS1)) and safety recovery policy to recover from safety-ciritical states (i.e., extreme-case) (Section [4.2](https://arxiv.org/html/2503.00923v3#S4.SS2)). (b) Training the high-level planning policy to select between the two pre-trained low-level policies (Section [4.3](https://arxiv.org/html/2503.00923v3#S4.SS3)), thereby ensuring locomotion stability and consistency.
Refer to caption: x2.png

In practice, developing an end-to-end solver for the robust locomotion problem (LABEL:obj:robust-crl) is challenging, as it involves multiple deployment environments and constraints that must be adhered to. To tackle this issue, we propose dividing the objective into two stages: goal-tracking and safety recovery. The goal-tracking policy maximizes rewards while mimicking human behaviors (see Section [4.1](https://arxiv.org/html/2503.00923v3#S4.SS1)). When mismatched environmental dynamics pose a risk to the safety of the control policy, HWC-Loco switches to the safety recovery policy, which manages safety-critical events to ensure the feasibility of humanoid control (see Section [4.2](https://arxiv.org/html/2503.00923v3#S4.SS2)). To dynamically identify the "sweet spot" for policy switching or activation, we introduce a high-level policy that coordinates the low-level policies based on historical observations and the robot’s current status (see Section [4.3](https://arxiv.org/html/2503.00923v3#S4.SS3)). Figure [2](https://arxiv.org/html/2503.00923v3#S4.F2) illustrates the training pipeline of our HWC-Loco.

### 4.1 Learning Goal-Tracking Policy for Efficient Humanoid Locomotion

The goal-tracking policy primarily focuses on tracking the provided command within the learning environment efficiently and naturally. This is achieved by optimizing the following objective function:

$$ $\displaystyle\max_{\pi_{1}}\mathbb{E}_{\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal {T}}^{L}}}}^{\pi_{1}}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{\mathfrak{T}}(s_{t },a_{t})\right]{s.t.}\;\mathcal{D}_{f}(\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal {T}}^{L}}}}^{\pi^{E}}\|\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}}}^{\pi _{1}})\leq\epsilon_{f}$ (5) $$

Note that compared to the complete objective
(LABEL:obj:robust-crl), this goal-tracking objective
focuses only on maximizing performance in the training environments’ dynamics $P_{\mathcal{T}}^{L}$.
Within this objective, the scale of the cumulative task rewards, $\sum_{t=0}^{\infty}\gamma^{t}r_{\mathfrak{T}}$ reflects how effectively the policy $\pi$ tracks locomotion-related commands.
Appendix [A.2](https://arxiv.org/html/2503.00923v3#A1.SS2) specifies the configuration of our task reward.

In addition to the task rewards, a crucial aspect of the goal-tracking objective ([5](https://arxiv.org/html/2503.00923v3#S4.E5)) is evaluating the distance between the occupancy measures of the trained policy and the expert policy
In the following, striving for concise representation, we represent $\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}}}^{\pi^{E}}$ and $\rho_{M^{{\color[rgb]{1,0,0}P_{\mathcal{T}}^{L}}}}^{\pi}$ by $\rho^{\pi^{E}}$ and $\rho^{\pi}$. Driven by information theory, previous works [47] primarily implement $\mathcal{D}_{f}$ using metrics such as Kullback-Leibler (KL) divergence or Jensen-Shannon (JS) divergence. However, these metrics fail when the support of $\rho^{\pi^{E}}$ and $\rho^{\pi}$ has limited or zero overlap. In this study, we consider implement $\mathcal{D}_{f}$ with Wasserstein-1 distance under the Kantorovich-Rubinstein duality [48] such that:

$$ $\displaystyle\mathcal{D}_{f}(\rho^{\pi^{E}}\|\rho^{\pi_{1}})=\sup_{\|f_{d}\|_{ L}\leq 1}\mathbb{E}_{x\sim\rho^{\pi^{E}}}[f_{d}(x)]-\mathbb{E}_{x\sim\rho^{\pi _{1}}}[f_{d}(x)]$ (6) $$

where the bounded Lipschitz-norm $\|f_{d}\|_{L}\leq 1$ ensure the smoothness of functions $f_{d}$. Inspired by adversarial imitation learning [49, 50], an effective and intuitive implementation of $f_{d}$ is a discriminator for measuring whether $x$ is generated by human experts, such that $f_{d}:\mathcal{X}\rightarrow\mathbb{R}$. It can be learned by maximizing the following objective:

$$ $\max_{f_{d}}\;\mathbb{E}_{x\sim\rho^{\pi^{E}}}[f_{d}(x)]-\mathbb{E}_{x\sim\rho ^{\pi_{1}}}[f_{d}(x)]+\beta\cdot\mathbb{E}_{\hat{x}\sim[\rho^{\pi^{E}},\rho^{ \pi_{1}}]}\left[(\|\nabla_{\hat{x}}f_{d}(\hat{x})\|_{2}-1)^{2}\right]$ (7) $$

where weighted by $\beta$, the gradient penalty term enforces the Lipschitz continuity required by the Wasserstein distance (i.e., $\|f_{d}\|_{L}\leq 1$).
During training, $\rho^{\pi^{E}}$ represents the density of expert demonstration dataset $\mathcal{D}^{E}$, which is built from the CMU MoCap dataset [51].
Additionally, we include data about humans standing, walking, and running to learn various human behaviors. We then retarget the motion data into a humanoid robot-compatible format, which we use as the training dataset.
By substituting $\mathcal{D}_{f}$ with equation ([6](https://arxiv.org/html/2503.00923v3#S4.E6)), the goal-tracking objective ([5](https://arxiv.org/html/2503.00923v3#S4.E5)) can be simplified as:

$$ $\max_{\pi_{1}}\mathbb{E}_{\rho^{\pi_{1}}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_ {\mathfrak{T}}(s_{t},a_{t})\right]\quad\text{s.t.}\quad\mathbb{E}_{\rho^{\pi^{ E}}}[f_{d}(s^{d})]-\mathbb{E}_{\rho^{\pi_{1}}}[f_{d}(s^{d})]\leq\epsilon_{f}$ (8) $$

where $s^{d}$ records the necessary state information for the discriminator.
(See details in Appendix [A.5](https://arxiv.org/html/2503.00923v3#A1.SS5)).
Given that this formulation of constrained RL problem has a zero duality gap [52], we can transform it into an unconstrained objective by analyzing the Lagrange dual form of the original constrained RL problem:

$$ $\displaystyle\max_{\pi_{1}}\mathbb{E}_{\rho^{\pi_{1}}}\left[\sum_{t=0}^{\infty }\gamma^{t}\left[r_{\mathfrak{T}}(s_{t},a_{t})-\lambda f_{d}(s^{d})\right]\right]$ (9) $$

where $\lambda$ denotes the optimal Lagrange multiplier. In practical implementation, the policy $\pi$ and discriminator $f_{d}$ can intuitively represent the generator and the discriminator under the adversarial learning framework.
In implementation, we utilize the proximal policy optimization algorithm [53] to update the goal-tracking policy. By alternatively updating ([7](https://arxiv.org/html/2503.00923v3#S4.E7)) and ([9](https://arxiv.org/html/2503.00923v3#S4.E9)), we can develop a goal-tracking policy that effectively maximizes task rewards while closely mimicking the expert’s behavior.

### 4.2 Learning Safety Recovery Policy for Handling Safe-Critical Events

Our safety recovery policy primarily focuses on handling emergency events, thereby recovering the robot from safety-critical situations and preventing failures of locomotion tasks, such as loss of balance or control policy malfunctions. To learn the safety-recover policy, we mainly study the complete objective (LABEL:obj:robust-crl). Similarly, as it is introduced in Section [4.1](https://arxiv.org/html/2503.00923v3#S4.SS1), we implement the divergence metric $\mathcal{D}_{f}$ with Wasserstein-1 distance, deriving the following objective:

$$ $\max_{\pi_{2}}\min_{{\color[rgb]{0,0,1}\widehat{P_{\mathcal{T}}}}\in\mathfrak{ P}_{\alpha}^{L}}\mathbb{E}_{\mu_{0},{\color[rgb]{0,0,1}\widehat{P_{\mathcal{T} }}},\pi_{2}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{\mathfrak{T}}(s_{t},a_{t})- \lambda f_{d}(s_{t},a_{t})\right]\;\;{s.t.}\;\;\mathbb{E}_{\tau\sim(\mu_{0},{ \color[rgb]{0,0,1}\widehat{P_{\mathcal{T}}}},\pi_{2})}\left[\phi(\tau)\right] \geq\epsilon_{\phi}$ (10) $$

where $\epsilon_{\phi}$ is a hyperparameter. By comparing this objective with the goal-tracking objective ([9](https://arxiv.org/html/2503.00923v3#S4.E9)), while both consider task rewards and human mimicking performance, their key differences lie in the incorporation of uncertainty in environmental dynamics and feasibility constraints in policy updates. These additions necessitate the construction of an extreme-case uncertainty set and the modeling of safety constraints by learning the feasibility function $\phi(\cdot)$.

Extreme-case Uncertainty Set.
Real-world deployment has complex terrains, external disturbances, hardware malfunctions, and sensor noise.
Those factors may lead to extreme cases like falling or unexpected behaviors.
To model them in the simulation (i.e., modelling ${\color[rgb]{0,0,1}\widehat{P_{\mathcal{T}}}}$s),
We: 1) apply multi-scale external forces and torques to the humanoid body, 2) introduce random, high-intensity noise to the proprioceptive information and PD gains, 3) resample velocity commands to simulate malicious velocity inputs [54] within the goal-tracking policy, and 4) apply domain randomization to the training environment. Together, these dynamic variations construct the uncertainty set $\mathfrak{P}_{\alpha}^{L}$, under which we learn the safety-recover policy. Appendix [A.7](https://arxiv.org/html/2503.00923v3#A1.SS7) shows our implementation details.

Zero-Moment Point (ZMP) Constraint.
A bipedal robot can be generally modeled by a linear inverted pendulum [55, 56], where ZMP refers to the point where the ground reaction force has no horizontal moment. If the ZMP exits the support polygon (typically the foot’s contact area), the robot will quickly lose balance [57, 58, 59, 60]. Stability is governed by gravity and the Center of Mass (CoM) accelerations, which also account for disturbances like obstacles and slippery surfaces. Under this assumption, we implement the feasibility indicator in the safety recovery objective ([10](https://arxiv.org/html/2503.00923v3#S4.E10)) as follows:

$$ $\displaystyle\phi(s,a)=\|\mathbf{p}_{\text{ZMP}}(s,a)-\mathbf{p_{\text{ac}}}\| _{2}\;\;\text{where}\;\;\mathbf{p}_{\text{ZMP}}(s,a)=\mathbf{p}_{\text{CoM}}(s ,a)-\frac{z_{\text{CoM}}(s,a)}{g}\cdot\ddot{\mathbf{p}}_{\text{CoM}}(s,a)$ (11) $$

where 1) the vector $\mathbf{p}_{\text{CoM}}$ denotes the $x$ and $y$-coordinates of CoM position. 2) $z_{\text{CoM}}$ is the height of the CoM. 3) $g$ represents the acceleration due to gravity. 4) $\ddot{\mathbf{p}}_{\text{CoM}}$ indicates the accelerations of the CoM in the $x$ and $y$-directions. 5) $\mathbf{p}_{\text{ac}}$ represents the center of the support polygon (specific to each robot and known in advance).
Intuitively, $\phi$ provides real-time insights into the robot’s stability and varies in its representation depending on the current support phase. Therefore, the humanoid robot can assess its stability in real time and use whole-body coordination to satisfy the ZMP constraint.

### 4.3 High-Level Planning for Policy Transition

Figure: Figure 3: Proportion of goal-tracking policy.
Refer to caption: x3.png

While the low-level policies can achieve varying levels of optimality,
a fundamental challenge is coordinating these policies to ensure that a humanoid robot can efficiently complete tasks while adhering to safety and robustness requirements. We introduce a high-level planning policy that dynamically determines which policy to activate based on the specific state of humanoid robots.
The task of policy selection has a discrete action space, and we learn this high-level policy $\pi_{0}$ based on the following objective:

$$ $\displaystyle\max_{\pi_{0}}\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\left[ r_{\mathfrak{T}}(s_{t},\bar{a}_{t})-\mathds{1}(\bar{a}_{t-1}\neq\bar{a}_{\text {t}})-\alpha\mathds{1}(s_{t})\right]\right]$ (12) $$

where 1) the policy selection action $\bar{a}_{t}\in\{0,1\}^{2}$ denotes a one-hot vector featuring whether the goal-tracking policy $\pi_{1}$ and safety-recovery policy $\pi_{2}$, 2) $\mathds{1}(\bar{a}_{t-1}\neq\bar{a}_{\text{t}})$ denotes the continuality identifier for discouraging the over-frequent switch of policies, and 3) $\mathds{1}(s_{t})$ denotes the termination identifier for preventing the failing of locomotion tasks. (2) and (3) are both $r_{\mathrm{P}}$ defined in Section [3](https://arxiv.org/html/2503.00923v3#S3). More importantly, within this objective, by adjusting $\alpha$, practitioners can adjust the trade-off safety guarantee and task completion. Training and algorithmic details can be found in Appendix [A.2](https://arxiv.org/html/2503.00923v3#A1.SS2).

We evaluate the average activation proportion of the goal-tracking policy during training.
As shown in Figure [3](https://arxiv.org/html/2503.00923v3#S4.F3), when $\alpha=0$, 20, or 50, the goal-tracking policy $\pi_{1}$ is predominantly selected, with the recovery policy $\pi_{2}$ triggered only in rare dangerous cases. In contrast, when $\alpha=200$, the sparsity of termination rewards induces training instability, resulting in overly conservative behavior.

Practical Deployment. The training of humanoid control policy commonly relies on the privileged information $\mathbb{P}_{t}$, such as external disturbances, terrain dynamics, etc. We follow  [18] and infer $\mathbb{P}_{t}$ by training a VAE-based estimator $P(e_{t},z_{t}|o^{H}_{t})$ where $z_{t}$ captures the contextual embedding of the $\mathbb{P}_{t}$, and $e_{t}$ consists of body velocity and ZMP features. To enhance the representation of $\phi(\cdot)$, we introduce frequency encoding, similar to the approach in [61], to capture the subtle variations in $\phi(\cdot)$ for the policy. Additionally, instead of the zero-mean Gaussian prior, we introduce a learnable prior similar to [62]. Moreover, we include a certain degree of domain randomization in the learning of HWC-Loco policies. The details of domain randomization are in Appendix [A.7](https://arxiv.org/html/2503.00923v3#A1.SS7).

## 5 Experiment

Experiment Settings.
To conduct a comprehensive evaluation, we quantify the control performance of HWC-Loco in simulated (Issac Gym [63]) and realistic environments from the following perspectives:
1) Effectiveness: How effectively does HWC-Loco navigate across diverse terrains?
2) Robustness: How well can HWC-Loco stabilize the humanoid robot under varying levels of disturbance?
3) Naturalness: How well does HWC-Loco imitate human-like movement norms?
4) Scalability: How effectively does HWC-Loco generalize to diverse embodiments and tasks?

Our evaluation metric includes a) Success Rate: The proportion of successful traverse across different scenarios. [64]
b) Goal Tracking performance: The ability to accurately follow velocity commands by maximizing task rewards $r_{\mathfrak{T}}$ detailed in Appendix [A.2](https://arxiv.org/html/2503.00923v3#A1.SS2) [43].
c) Human-Like behavior: Measured as the Wasserstein-1 distance between the robot’s and sampled human motions.

Comparison Methods.
To evaluate the effectiveness of different components in HWC-Loco, we design a comparison method using an ablation approach as follows:
1) HWC-Loco-l sets $\alpha$ to a lower value, thereby reducing the sensitivity of the high-level policy to failure events.
2) Goal-Tracking Policy removes the safety recovery mechanism and relies solely on the goal-tracking policy $\pi_{1}$ (Section [4.1](https://arxiv.org/html/2503.00923v3#S4.SS1)).
3) DreamWaQ-Humanoid further removes the human imitation objective $\mathcal{D}_{f}$ from objective ([5](https://arxiv.org/html/2503.00923v3#S4.E5)), effectively reducing our method to an adaptation of DreamWaQ [18] for humanoid control.
4) To highlight the benefits of our hierarchical policy, we compare a recent advancement of DreamWaQ, called AHL [64], which employs two-phase training for updating the locomotion policy. All policies are trained using three random seeds and evaluated in 1000 distinct environments.

### 5.1 Effectiveness: Locomotion across Diverse Terrains

Effectiveness in Simulated Environments.
We evaluate the policies on challenging terrains, specifically slopes and stairs. To enable a comprehensive comparison, evaluations are performed under both low-speed and high-speed command conditions. Low-speed commands are sampled as $v_{\text{x}}^{c}\sim[0.0,1.0]$, $v_{\text{y}}^{c}\sim[-0.3,0.3]$, and $w_{\text{z}}^{c}\sim[-0.5,0.5]$, while high-speed commands follow $v_{\text{x}}^{c}\sim[1.0,2.0]$, $v_{\text{y}}^{c}\sim[-0.6,0.6]$, and $w_{\text{z}}^{c}\sim[-1.0,1.0]$. Here, $v_{\text{x}}^{c}$ and $v_{\text{y}}^{c}$ denote the commanded linear velocities along the x- and y-axes, respectively, and $w_{\text{z}}^{c}$ denotes the commanded angular velocity around the z-axis. Table [1](https://arxiv.org/html/2503.00923v3#S5.T1) shows that our HWC-Loco achieves the highest success rate across all evaluated scenarios, while maintaining promising goal tracking and human mimic performance. Notably, in the high-speed stair terrain setting, HWC-Loco has a significantly higher success rate than all other policies, although its goal-tracking ability slightly decreases. This indicates that the policy prioritizes stability over aggressive goal pursuit. For example, when given a high-speed command, HWC-Loco enables the policy to adapt dynamically in safety-critical situations (e.g., navigating stairs mid-air) rather than rigidly maintaining high velocity. Comparably, when downplaying the sensitivity to safety-critical events and removing the safety-recovery policy, the success data drops significantly from nearly 85% to around 60% in the testing environment with stairs and high-speed commands.

Effectiveness in Realistic Environments.
Figure [10](https://arxiv.org/html/2503.00923v3#A2.F10) (see Appendix [B.1](https://arxiv.org/html/2503.00923v3#A2.SS1)) demonstrates the robot’s ability to climb 15 cm stairs and 20° slopes. When faced with challenging terrain, the robot prioritizes safety by activating the recovery policy. After regaining stability, it smoothly transitions back to goal tracking, showcasing its adaptive control in dynamic and uncertain environments. Figure [11](https://arxiv.org/html/2503.00923v3#A2.F11) (see Appendix [B.1](https://arxiv.org/html/2503.00923v3#A2.SS1)) further illustrates the robot’s effectiveness in real-world outdoor settings, where it successfully navigates across diverse terrains, including flat ground, grassy surfaces, and slopes.

**Table 1: Locomotion performance in simulated environments. Each evaluation runs for 1200 steps, which is equivalent to 12 seconds of real clock time.**
| Method | Slopes | Stairs |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Success Rate(%) $\uparrow$ | Goal Tracking $\uparrow$ | Human - like $\downarrow$ | Success Rate(%) $\uparrow$ | Goal Tracking $\uparrow$ | Human - like $\downarrow$ |
| (a) Low Speed |  |  |  |  |  |  |
| DreamWaQ | $92.31\pm 0.40$ | $1.19\pm 0.02$ | $3.52\pm 0.05$ | $74.32\pm 1.30$ | $1.19\pm 0.03$ | $3.53\pm 0.02$ |
| AHL | $98.83\pm 0.15$ | $1.21\pm 0.01$ | $3.41\pm 0.04$ | $93.73\pm 0.70$ | $1.21\pm 0.04$ | $3.44\pm 0.01$ |
| Goal-tracking Policy | $99.90\pm 0.04$ | $\boldsymbol{1.31\pm 0.00}$ | $\boldsymbol{3.30\pm 0.05}$ | $96.60\pm 0.14$ | $\boldsymbol{1.31\pm 0.00}$ | $\boldsymbol{3.31\pm 0.01}$ |
| HWC-Loco-l | $\boldsymbol{100.00\pm 0.00}$ | $1.23\pm 0.02$ | $3.32\pm 0.03$ | $99.80\pm 0.08$ | $1.21\pm 0.02$ | $3.31\pm 0.06$ |
| HWC-Loco | $\boldsymbol{100.00\pm 0.00}$ | $1.22\pm 0.02$ | $3.32\pm 0.05$ | $\boldsymbol{99.98\pm 0.01}$ | $1.19\pm 0.03$ | $3.34\pm 0.03$ |
| (b) High Speed |  |  |  |  |  |  |
| DreamWaQ | $90.46\pm 0.43$ | $1.05\pm 0.01$ | $3.63\pm 0.04$ | $60.58\pm 0.64$ | $1.06\pm 0.01$ | $3.66\pm 0.02$ |
| AHL | $97.36\pm 0.23$ | $1.12\pm 0.01$ | $3.76\pm 0.02$ | $67.48\pm 0.77$ | $1.09\pm 0.01$ | $3.68\pm 0.04$ |
| Goal-tracking Policy | $98.51\pm 0.45$ | $\boldsymbol{1.13\pm 0.00}$ | $3.52\pm 0.03$ | $72.60\pm 0.97$ | $\boldsymbol{1.11\pm 0.00}$ | $3.43\pm 0.02$ |
| HWC-Loco-l | $99.95\pm 0.02$ | $1.12\pm 0.02$ | $\boldsymbol{3.43\pm 0.04}$ | $78.92\pm 0.45$ | $1.10\pm 0.01$ | $\boldsymbol{3.39\pm 0.05}$ |
| HWC-Loco | $\boldsymbol{100.0\pm 0.00}$ | $1.12\pm 0.02$ | $3.44\pm 0.09$ | $\boldsymbol{84.34\pm 0.43}$ | $1.07\pm 0.03$ | $3.46\pm 0.03$ |

### 5.2 Robustness: Stable Control under Disturbances

Robustness under Simulated Disturbances. To demonstrate the robustness of HWC-Loco, we conduct extensive disturbance tests in simulation. The robots are commanded to follow randomly sampled velocities within the training distribution while navigating a uniformly mixed terrain comprising flat ground, obstacles, slopes, and stairs. Under these general locomotion settings, we design three types of disturbances as follows: 1) External force/torque disturbances, where random forces and torques (up to 200 N and 200 N$\cdot$m) are applied to each robot link, either at a low frequency of 1 Hz or as constant perturbations lasting 0.5 seconds, following prior setups [65, 66]. 2) Impulse disturbances on the CoM, implemented by directly altering the robot’s CoM velocity [65, 28]. We simulate both low-impulse and high-impulse scenarios, with linear velocity perturbations of 0–1 m/s or 1–2 m/s, and angular velocity perturbations of 0–0.5 rad/s or 0.5–1.0 rad/s, respectively. 3) Payload disturbances, where an additional mass uniformly sampled from either 0 to 5 kg or 0 to 10 kg is added to upper-body links [21, 12], along with internal perturbations from sensor and actuator noise (see Table [11](https://arxiv.org/html/2503.00923v3#A1.T11) in Appendix [A.7](https://arxiv.org/html/2503.00923v3#A1.SS7)), to simulate unmodeled dynamics and varying mass distribution scenarios.

As shown in Table [2](https://arxiv.org/html/2503.00923v3#S5.T2), HWC-Loco consistently achieves the highest success rates across all types of disturbances. Specifically, under constant disturbances, HWC-Loco demonstrates superior performance with a success rate of 75.95%, which is nearly 25% higher than that of DreamWaQ. It validates the effectiveness of HWC-Loco’s recovery mechanism in resisting strong and persistent forces. Under low-impulse disturbances, all methods maintain relatively high success rates, with HWC-Loco again achieving the best performance at 94.84%. When facing more challenging high-impulse disturbances, the performance gap becomes more pronounced. HWC-Loco reaches a success rate of 81.27%, outperforming all baselines by a significant margin. This result indicates that the recovery mechanism effectively mitigates the impact of sudden impulses by enabling the robot to quickly lower its body and wave its arms to regain stability. In the payload test, HWC-Loco also demonstrates the best performance. Although it s not explicitly trained on scenarios such as adding 10 kg to the hands, it maintains a high success rate. This highlights the robustness of HWC-Loco to unseen disturbances.

**Table 2: Robustness of Locomotion across Various Terrains under Different Disturbances**
| Policy | External Force/Torque Disturbances | Impulse Disturbances on CoM | Payload on Upper Body |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Low-freq. $\uparrow$ | Constant $\uparrow$ | Low-imp. $\uparrow$ | High-imp. $\uparrow$ | Low Payload $\uparrow$ | High Payload $\uparrow$ |
| DreamWaQ | $85.92\pm 1.64$ | $51.31\pm 0.97$ | $85.24\pm 0.37$ | $45.34\pm 0.41$ | $67.63\pm 3.10$ | $55.04\pm 2.82$ |
| AHL | $87.15\pm 2.86$ | $60.72\pm 0.36$ | $85.87\pm 1.47$ | $62.94\pm 1.29$ | $79.29\pm 0.65$ | $59.16\pm 1.60$ |
| Goal-tracking Policy | $90.00\pm 2.20$ | $61.20\pm 0.61$ | $88.90\pm 0.60$ | $58.13\pm 0.78$ | $78.34\pm 1.23$ | $61.44\pm 1.00$ |
| HWC-Loco-l | $92.69\pm 0.56$ | $68.60\pm 0.28$ | $92.79\pm 1.75$ | $77.09\pm 0.79$ | $84.11\pm 2.66$ | $63.96\pm 0.09$ |
| HWC-Loco | $\boldsymbol{95.88\pm 0.37}$ | $\boldsymbol{75.95\pm 0.66}$ | $\boldsymbol{94.84\pm 0.54}$ | $\boldsymbol{81.27\pm 0.80}$ | $\boldsymbol{87.43\pm 0.92}$ | $\boldsymbol{69.86\pm 1.00}$ |

Robustness in Realistic Deployment. We deploy HWC-Loco on a real humanoid robot and evaluate its performance under various external force disturbances. Figure [12](https://arxiv.org/html/2503.00923v3#A2.F12) (see Appendix [B.1](https://arxiv.org/html/2503.00923v3#A2.SS1)) shows policy switching and associated changes in roll and pitch angles under continuous perturbations, including pushes, pulls, and kicks.
Upon encountering disturbances, the robot promptly switches to the recovery policy, adjusting its posture and gait to regain stability. Importantly, the controller does not rely solely on recovery mode but dynamically switches between goal-tracking and recovery policies, thereby adapting the action distribution to environmental changes. Figure [13](https://arxiv.org/html/2503.00923v3#A2.F13) (see Appendix [B.1](https://arxiv.org/html/2503.00923v3#A2.SS1)) further demonstrates the robot’s robustness to disturbances in outdoor settings.

### 5.3 Naturalness: Imitating Human-like Movements

This experiment assesses the robot’s ability to imitate human-like locomotion under varying velocity commands. As shown in Figure [4](https://arxiv.org/html/2503.00923v3#S5.F4), the robot remains stationary at 0 m/s, and exhibits natural walking behavior at 1.0 m/s, characterized by straight elbows and coordinated arm swings. At 2.5 m/s, it transitions to running, with longer strides and increasingly bent elbows, closely resembling human running. As the velocity increases, both stride length and step frequency adjust dynamically, mimicking typical human locomotion. In contrast, prior approaches [28, 67, 64, 68] often maintain a fixed step frequency regardless of the commanded velocity. Some even exhibit in-place stepping at 0,m/s [68, 64], resulting in unnatural behavior for tasks requiring a steady stance, such as manipulation.

### 5.4 Scalability: Extending HWC-Loco to New Tasks

Cross Embodiment Validation. We deploy HWC-Loco on the Unitree G1 humanoid robot to evaluate its generalization capability under the same locomotion settings. As shown in Table [3](https://arxiv.org/html/2503.00923v3#S5.T3), G1 outperforms H1 across all metrics. A detailed analysis is provided in Appendix [A.2](https://arxiv.org/html/2503.00923v3#A1.SS2).

Figure: Figure 4: Human-like behavior.
Refer to caption: x4.png

Expressive Motion Tracking
We follow the approach of ExBody[33, 69] and set expressive motion as the goal for HWC-Loco. We train the model following the same pipeline as locomotion tasks. A domain-randomized motion tracking policy serves as the baseline. We evaluate both policies on three representative motions under impulse disturbances. The success rates are reported in Table [4](https://arxiv.org/html/2503.00923v3#S5.T4).

## 6 Limitation

Our approach has three main limitations. First, policy transitions are governed by a learned discrete decision module, while the low-level controllers remain fixed. A jointly optimized hierarchical framework may enable smoother and more adaptive transitions. Second, the humanoid robot used in real-world deployment has only 19 degrees of freedom, which limits whole-body coordination and constrains the expression of complex recovery behaviors. Third, the recovery policy is trained in simulated dynamic scenarios that may not fully reflect the diversity and complexity of real-world disturbances. Incorporating more realistic perturbations, such as those generated by adversarial policies, could further improve recovery ability.

## 7 Conclusion

We introduce HWC-Loco, a hierarchical control framework for humanoid robots that incorporates an embedded safety recovery mechanism. This framework has been validated across various tasks and scenarios, demonstrating exceptional effectiveness, robustness, naturalness, and scalability. Notably, the safety mechanism in HWC-Loco extends beyond locomotion, enabling reliable performance in complex tasks through a dynamic task-safety balance. This ensures robust operation in real-world deployments, positioning HWC-Loco as a foundational solution for safety-critical applications such as industrial automation and assistive robotics.
A promising direction for future research is integrating HWC-Loco with loco-manipulation skills, enabling safety-critical control across a broader range of tasks involving different objects.

## References

- Saeedvand et al. [2019]
Saeed Saeedvand, Masoumeh Jafari, Hadi S. Aghdasi, and Jacky Baltes.
A comprehensive survey on humanoid robot development.
*Knowledge Engineering Review*, 34:e20, 2019.
- Paolo et al. [2024]
Giuseppe Paolo, Jonas Gonzalez-Billandon, and Balázs Kégl.
Position: A call for embodied AI.
In *Forty-first International Conference on Machine Learning, ICML*, 2024.
- Sferrazza et al. [2024]
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel.
Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation.
*arXiv preprint arXiv:2403.10506*, 2024.
- Nasiriany et al. [2024]
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu.
Robocasa: Large-scale simulation of everyday tasks for generalist robots.
*arXiv preprint arXiv:2406.02523*, 2024.
- Tao et al. [2024]
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al.
Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai.
*arXiv preprint arXiv:2410.00425*, 2024.
- Sakagami et al. [2002]
Yoshiaki Sakagami, Ryujin Watanabe, Chiaki Aoyama, Shinichi Matsunaga, Nobuo Higaki, and Kikuo Fujimura.
The intelligent ASIMO: system overview and integration.
In *IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS*, pages 2478–2483. IEEE, 2002.
- Sentis and Khatib [2006]
Luis Sentis and Oussama Khatib.
A whole-body control framework for humanoids operating in human environments.
In *IEEE International Conference on Robotics and Automation, ICRA*, pages 2641–2648. IEEE, 2006.
- Gouaillier et al. [2009]
David Gouaillier, Vincent Hugel, Pierre Blazevic, Chris Kilner, Jérôme Monceaux, Pascal Lafourcade, Brice Marnier, Julien Serre, and Bruno Maisonnier.
Mechatronic design of NAO humanoid.
In *IEEE International Conference on Robotics and Automation, ICRA*, pages 769–774, 2009.
- Radford et al. [2015]
Nicolaus A. Radford, Philip Strawser, Kimberly A. Hambuchen, Joshua S. Mehling, William K. Verdeyen, A. Stuart Donnan, James Holley, Jairo Sanchez, Vienny Nguyen, Lyndon B. Bridgwater, Reginald Berka, Robert O. Ambrose, Mason Myles Markee, N. J. Fraser-Chanpong, Christopher McQuin, John D. Yamokoski, Stephen Hart, Raymond Guo, Adam Parsons, Brian Wightman, Paul Dinh, Barrett Ames, Charles Blakely, Courtney Edmondson, Brett Sommers, Rochelle Rea, Chad Tobler, Heather Bibby, Brice Howard, Lei Niu, Andrew Lee, Michael Conover, Lily Truong, Ryan Reed, David Chesney, Robert Platt Jr., Gwendolyn Johnson, Chien-Liang Fok, Nicholas Paine, Luis Sentis, Eric A. Cousineau, Ryan W. Sinnet, Jordan Lack, Matthew J. Powell, Benjamin Morris, Aaron D. Ames, and Jide Akinyode.
Valkyrie: Nasa’s first bipedal humanoid robot.
*Journal of Field Robotics*, 32(3):397–419, 2015.
- Chignoli et al. [2021]
Matthew Chignoli, Donghyun Kim, Elijah Stanger-Jones, and Sangbae Kim.
The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors.
In *2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids)*, pages 1–8. IEEE, 2021.
- Zhuang et al. [2024]
Ziwen Zhuang, Shenzhe Yao, and Hang Zhao.
Humanoid parkour learning.
In *8th Annual Conference on Robot Learning*, 2024.
- Radosavovic et al. [2024]
Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath.
Real-world humanoid locomotion with reinforcement learning.
*Science Robotics*, 9(89):eadi9579, 2024.
- Long et al. [2024]
Junfeng Long, Junli Ren, Moji Shi, Zirui Wang, Tao Huang, Ping Luo, and Jiangmiao Pang.
Learning humanoid locomotion with perceptive internal model.
*arXiv preprint arXiv:2411.14386*, 2024.
- He et al. [2024a]
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi.
Learning human-to-humanoid real-time whole-body teleoperation.
*arXiv preprint arXiv:2403.04436*, 2024a.
- Gu et al. [2024a]
Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, and Jianyu Chen.
Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning.
*arXiv preprint arXiv:2408.14472*, 2024a.
- Lai et al. [2023]
Hang Lai, Weinan Zhang, Xialin He, Chen Yu, Zheng Tian, Yong Yu, and Jun Wang.
Sim-to-real transfer for quadrupedal locomotion via terrain transformer.
In *2023 IEEE International Conference on Robotics and Automation (ICRA)*, pages 5141–5147. IEEE, 2023.
- Rudin et al. [2022]
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter.
Learning to walk in minutes using massively parallel deep reinforcement learning.
In *Conference on Robot Learning*, pages 91–100. PMLR, 2022.
- Nahrendra et al. [2023]
I Made Aswin Nahrendra, Byeongho Yu, and Hyun Myung.
Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning.
In *2023 IEEE International Conference on Robotics and Automation (ICRA)*, pages 5078–5084. IEEE, 2023.
- Shi et al. [2022]
Haojie Shi, Bo Zhou, Hongsheng Zeng, Fan Wang, Yueqiang Dong, Jiangyong Li, Kang Wang, Hao Tian, and Max Q-H Meng.
Reinforcement learning with evolutionary trajectory generator: A general approach for quadrupedal locomotion.
*IEEE Robotics and Automation Letters*, 7(2):3085–3092, 2022.
- Chen et al. [2023]
Shuxiao Chen, Bike Zhang, Mark W Mueller, Akshara Rai, and Koushil Sreenath.
Learning torque control for quadrupedal locomotion.
In *2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids)*, pages 1–8. IEEE, 2023.
- Margolis and Agrawal [2023]
Gabriel B Margolis and Pulkit Agrawal.
Walk these ways: Tuning robot control for generalization with multiplicity of behavior.
In *Conference on Robot Learning*, pages 22–31. PMLR, 2023.
- Zhuang et al. [2023]
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao.
Robot parkour learning.
*arXiv preprint arXiv:2309.05665*, 2023.
- He et al. [2024b]
Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, and Guanya Shi.
Agile but safe: Learning collision-free high-speed legged locomotion.
*arXiv preprint arXiv:2401.17583*, 2024b.
- Li et al. [2021]
Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath.
Reinforcement learning for robust parameterized locomotion control of bipedal robots.
In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 2811–2817. IEEE, 2021.
- Kumar et al. [2022]
Ashish Kumar, Zhongyu Li, Jun Zeng, Deepak Pathak, Koushil Sreenath, and Jitendra Malik.
Adapting rapid motor adaptation for bipedal robots.
In *2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 1161–1168. IEEE, 2022.
- Li et al. [2023]
Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath.
Robust and versatile bipedal jumping control through reinforcement learning.
*arXiv preprint arXiv:2302.09450*, 2023.
- Duan et al. [2024]
Helei Duan, Bikram Pandit, Mohitvishnu S Gadde, Bart Van Marum, Jeremy Dao, Chanho Kim, and Alan Fern.
Learning vision-based bipedal locomotion for challenging terrain.
In *2024 IEEE International Conference on Robotics and Automation (ICRA)*, pages 56–62. IEEE, 2024.
- Gu et al. [2024b]
Xinyang Gu, Yen-Jen Wang, and Jianyu Chen.
Humanoid-gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer.
*arXiv preprint arXiv:2404.05695*, 2024b.
- Sun et al. [2025]
Wandong Sun, Baoshi Cao, Long Chen, Yongbo Su, Yang Liu, Zongwu Xie, and Hong Liu.
Learning perceptive humanoid locomotion over challenging terrain.
*arXiv preprint arXiv:2503.00692*, 2025.
- Luo et al. [2023a]
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al.
Perpetual humanoid control for real-time simulated avatars.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 10895–10904, 2023a.
- Luo et al. [2024a]
Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Kris Kitani, and Weipeng Xu.
Real-time simulated avatar from head-mounted sensors.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 571–581, 2024a.
- Luo et al. [2024b]
Zhengyi Luo, Jiashun Wang, Kangni Liu, Haotian Zhang, Chen Tessler, Jingbo Wang, Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, et al.
Smplolympics: Sports environments for physically simulated humanoids.
*arXiv preprint arXiv:2407.00187*, 2024b.
- Cheng et al. [2024]
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang.
Expressive whole-body control for humanoid robots.
*arXiv preprint arXiv:2402.16796*, 2024.
- He et al. [2024c]
Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi.
Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning.
*arXiv preprint arXiv:2406.08858*, 2024c.
- Fu et al. [2024]
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn.
Humanplus: Humanoid shadowing and imitation from humans.
In *8th Annual Conference on Robot Learning*, 2024.
- He et al. [2024d]
Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhenjia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, Linxi Fan, and Yuke Zhu.
HOVER: versatile neural whole-body controller for humanoid robots.
*arXiv preprint arXiv:2410.21229*, 2024d.
- He et al. [2025]
Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, et al.
Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills.
*arXiv preprint arXiv:2502.01143*, 2025.
- Ze et al. [2025]
Yanjie Ze, Zixuan Chen, JoÃĢo Pedro AraÃšjo, Zi-ang Cao, Xue Bin Peng, Jiajun Wu, and C Karen Liu.
Twist: Teleoperated whole-body imitation system.
*arXiv preprint arXiv:2505.02833*, 2025.
- Li et al. [2025]
Jialong Li, Xuxin Cheng, Tianshu Huang, Shiqi Yang, Ri-Zhao Qiu, and Xiaolong Wang.
Amo: Adaptive motion optimization for hyper-dexterous humanoid whole-body control.
*arXiv preprint arXiv:2505.03738*, 2025.
- Kim et al. [2024]
Dohyeong Kim, Hyeokjin Kwon, Junseok Kim, Gunmin Lee, and Songhwai Oh.
Stage-wise reward shaping for acrobatic robots: A constrained multi-objective reinforcement learning approach.
*arXiv preprint arXiv:2409.15755*, 2024.
- Zhang et al. [2024a]
Chong Zhang, Wenli Xiao, Tairan He, and Guanya Shi.
Wococo: Learning whole-body humanoid control with sequential contacts.
In *Annual Conference on Robot Learning*, 2024a.
- Gu et al. [2024c]
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll.
A review of safe reinforcement learning: Methods, theories, and applications.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 46(12):11216–11235, 2024c.
- Chen et al. [2024]
Zixuan Chen, Xialin He, Yen-Jen Wang, Qiayuan Liao, Yanjie Ze, Zhongyu Li, S Shankar Sastry, Jiajun Wu, Koushil Sreenath, Saurabh Gupta, et al.
Learning smooth humanoid locomotion through lipschitz-constrained policies.
*arXiv preprint arXiv:2410.11825*, 2024.
- Mahmood et al. [2019]
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.
AMASS: archive of motion capture as surface shapes.
In *IEEE/CVF International Conference on Computer Vision, ICCV*, pages 5441–5450. IEEE, 2019.
- Moos et al. [2022]
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters.
Robust reinforcement learning: A review of foundations and recent advances.
*Machine Learning and Knowledge Extraction*, pages 276–315, 2022.
- Viano et al. [2022]
Luca Viano, Yu-Ting Huang, Parameswaran Kamalaruban, Craig Innes, Subramanian Ramamoorthy, and Adrian Weller.
Robust learning from observation with model misspecification.
In *International Conference on Autonomous Agents and Multiagent Systems*, pages 1337–1345, 2022.
- Hussein et al. [2017]
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne.
Imitation learning: A survey of learning methods.
*ACM Computing Surveys*, 50(2):21:1–21:35, 2017.
- Villani et al. [2009]
Cédric Villani et al.
*Optimal transport: old and new*, volume 338.
Springer, 2009.
- Ho and Ermon [2016]
Jonathan Ho and Stefano Ermon.
Generative adversarial imitation learning.
In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, *Advances in Neural Information Processing Systems, NeurIPS*, pages 4565–4573, 2016.
- Zhang et al. [2020]
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang.
f-gail: Learning f-divergence for generative adversarial imitation learning.
In *Advances in Neural Information Processing Systems, NeurIPS*, 2020.
- [51]
CMU, Cmu graphics lab motion capture database, [http://mocap.cs.cmu.edu/](http://mocap.cs.cmu.edu/).
- Paternain et al. [2019]
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro.
Constrained reinforcement learning has zero duality gap.
In *Advances in Neural Information Processing Systems, NeurIPS*, pages 7553–7563, 2019.
- Schulman et al. [2017]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shi et al. [2024]
Fan Shi, Chong Zhang, Takahiro Miki, Joonho Lee, Marco Hutter, and Stelian Coros.
Rethinking robustness assessment: Adversarial attacks on learning-based quadrupedal locomotion controllers.
*arXiv preprint arXiv:2405.12424*, 2024.
- Morisawa et al. [2006]
Mitsuharu Morisawa, Kensuke Harada, Shuuji Kajita, Kenji Kaneko, Fumio Kanehiro, Kiyoshi Fujiwara, Shinichiro Nakaoka, and Hirohisa Hirukawa.
A biped pattern generation allowing immediate modification of foot placement in real-time.
In *2006 6th IEEE-RAS International Conference on Humanoid Robots*, pages 581–586. IEEE, 2006.
- Harada et al. [2006]
Kensuke Harada, Shuuji Kajita, Kenji Kaneko, and Hirohisa Hirukawa.
An analytical method for real-time gait planning for humanoid robots.
*International Journal of Humanoid Robotics*, 3(01):1–19, 2006.
- Wieber [2006]
Pierre-Brice Wieber.
Trajectory free linear model predictive control for stable walking in the presence of strong perturbations.
In *2006 6th IEEE-RAS International Conference on Humanoid Robots*, pages 137–142. IEEE, 2006.
- Feng et al. [2016]
Siyuan Feng, X Xinjilefu, Christopher G Atkeson, and Joohyung Kim.
Robust dynamic walking using online foot step optimization.
In *2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 5373–5378. IEEE, 2016.
- Scianca et al. [2020]
Nicola Scianca, Daniele De Simone, Leonardo Lanari, and Giuseppe Oriolo.
Mpc for humanoid gait generation: Stability and feasibility.
*IEEE Transactions on Robotics*, 36(4):1171–1188, 2020.
- Xie et al. [2025]
Weiji Xie, Chenjia Bai, Jiyuan Shi, Junkai Yang, Yunfei Ge, Weinan Zhang, and Xuelong Li.
Humanoid whole-body locomotion on narrow terrain via dynamic balance and reinforcement learning.
*arXiv preprint arXiv:2502.17219*, 2025.
- Mildenhall et al. [2021]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis.
*Communications of the ACM*, 65(1):99–106, 2021.
- Luo et al. [2023b]
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu.
Universal humanoid motion representations for physics-based control.
*arXiv preprint arXiv:2310.04582*, 2023b.
- Makoviychuk et al. [2021]
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.
Isaac gym: High performance gpu-based physics simulation for robot learning.
*arXiv preprint arXiv:2108.10470*, 2021.
- Cui et al. [2024]
Wenhao Cui, Shengtao Li, Huaxing Huang, Bangyu Qin, Tianchu Zhang, Liang Zheng, Ziyang Tang, Chenxu Hu, NING Yan, Jiahao Chen, et al.
Adapting humanoid locomotion over challenging terrain via two-phase training.
In *8th Annual Conference on Robot Learning*, 2024.
- Weng et al. [2023]
Bowen Weng, Guillermo A Castillo, Wei Zhang, and Ayonga Hereid.
On the comparability and optimal aggressiveness of the adversarial scenario-based safety testing of robots.
*IEEE Transactions on Robotics*, 39(4):3299–3318, 2023.
- Lee et al. [2020]
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter.
Learning quadrupedal locomotion over challenging terrain.
*Science robotics*, 5(47):eabc5986, 2020.
- Gu et al. [2024d]
Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, and Jianyu Chen.
Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning.
*arXiv preprint arXiv:2408.14472*, 2024d.
- Zhang et al. [2024b]
Qiang Zhang, Peter Cui, David Yan, Jingkai Sun, Yiqun Duan, Gang Han, Wen Zhao, Weining Zhang, Yijie Guo, Arthur Zhang, et al.
Whole-body humanoid robot locomotion with human reference.
In *2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 11225–11231. IEEE, 2024b.
- Ji et al. [2024]
Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, and Xiaolong Wang.
Exbody2: Advanced expressive humanoid whole-body control.
*arXiv preprint arXiv:2412.13196*, 2024.
- Wang et al. [2024]
Zhicheng Wang, Wandi Wei, Ruiqi Yu, Jun Wu, and Qiuguo Zhu.
Toward understanding key estimation in learning robust humanoid locomotion.
*arXiv preprint arXiv:2403.05868*, 2024.

## Appendix A Addtional Details of HWC-Loco

### A.1 Motion Retargeting

Our motion data is derived from the CMU Mocap dataset [51], which exclusively encompasses human locomotion data, featuring a diverse array of walking, jogging, and running styles. In aggregate, we have loaded 318 motion sequences, totaling 3729.18 seconds in duration. Our motion retargeting methodology closely follows the approach outlined in [33]. We have meticulously aligned the humanoid robot’s skeleton with that of humans, incorporating adjustments such as rotation, translation, and scaling. Additionally, we map human joints onto the humanoid robot’s structure. Specifically, the human motion data is adapted to the humanoid robot’s framework by mapping local information. Both the Unitree H1 and G1 robots possess shoulder and hip joints that are functionally equivalent to spherical joints. During the retargeting process, the three hip and shoulder joints are treated as a single spherical joint, corresponding to the human’s spherical joint. For one-dimensional joints such as the elbow and torso, the rotation angle is projected onto the corresponding rotation axis of the joint angles.

**Table 5: Double-DQN Parameters**
| Parameter | Value |
| --- | --- |
| Batch Size | 128 |
| Learning Rate | 1e-4 |
| Gamma | 0.99 |
| Max Grad Norm | 1.0 |
| Replay Buffer Capacity | 2000000 |
| Switch Penalty Coefficient | 0.001 |
| Target Update Frequency | 50 |
| Optimizer | Adam |
| Loss Function | MSELoss |
| Initial Epsilon | 0.1 |
| Minimum Epsilon | 0.001 |
| Epsilon Decay Rate | 0.999 |

### A.2 Implementation Details

Proprioception: For the Unitree H1, proprioception $\boldsymbol{o}_{t}^{p}\in\mathbb{R}^{65}$, which denotes the internal state of the robot, including the base angular velocity $\boldsymbol{w}_{t}\in\mathbb{R}^{3}$, base roll $\boldsymbol{r}_{t}\in\mathbb{R}^{1}$, base pitch $\boldsymbol{p}_{t}\in\mathbb{R}^{1}$, degrees of freedom (DOF) positions $\boldsymbol{q}_{t}\in\mathbb{R}^{19}$, DOF velocities $\dot{\boldsymbol{q}}_{t}\in\mathbb{R}^{19}$, previous actions $\boldsymbol{a}_{t-1}\in\mathbb{R}^{19}$, and projected gravity $\boldsymbol{g}_{t}\in\mathbb{R}^{3}$. The projected gravity refers to the component of gravity expressed in the robot’s local coordinate system. For the Unitree G1, proprioception $\boldsymbol{o}_{t}^{p}\in\mathbb{R}^{77}$, includes base angular velocity $\boldsymbol{w}_{t}\in\mathbb{R}^{3}$, base roll $\boldsymbol{r}_{t}\in\mathbb{R}^{1}$, base pitch $\boldsymbol{p}_{t}\in\mathbb{R}^{1}$, DOF positions $\boldsymbol{q}_{t}\in\mathbb{R}^{23}$, DOF velocities $\dot{\boldsymbol{q}}_{t}\in\mathbb{R}^{23}$, previous actions $\boldsymbol{a}_{t-1}\in\mathbb{R}^{23}$, and projected gravity $\boldsymbol{g}_{t}\in\mathbb{R}^{3}$. As with the H1, the projected gravity denotes the component of gravity in the robot’s local coordinate system.

Action Space: The policy outputs continuous actions $\mathbf{a}_{t}\in\mathbb{R}^{n}$, which are utilized as target positions for a PD controller to compute joint torques. The actions correspond to the robot’s actuated degrees of freedom: specifically, $\mathbf{a}_{t}\in\mathbb{R}^{19}$ (19 DoF) for the Unitree H1 and $\mathbf{a}_{t}\in\mathbb{R}^{23}$ (23 DoF) for the Unitree G1.

Terrain details: Terrains in the training environment simulated by Isaac Gym [63] include flats, obstacles, slopes, and stairs. An example of these terrains is shown in Figure [5](https://arxiv.org/html/2503.00923v3#A1.F5).

Figure: Figure 5: Visualization of different terrains. From left to right, the terrains are flats, obstacles, slopes, and stairs
Refer to caption: x5.png

Low-level Policy Training: For all the low-level policies training, the commands are sampled from ranges: $v_{\text{x}}^{c}\sim[-0.6,2.5]$, $v_{\text{y}}^{c}\sim[-0.6,0.6]$ and $w_{\text{z}}^{c}\sim[-1.0,1.0]$.
During the initial training of the humanoid robot on uneven terrain, the robot often remains stationary due to the high difficulty of navigating complex surfaces. To address this, we introduce a terrain curriculum method [63]. The training terrain consists of various types, including flat planes, rough surfaces, steps, and slopes. As the robot achieves a goal-tracking performance of 70% of the commanded velocity, the terrain difficulty increases. Conversely, if the goal-tracking performance drops below 40% of the commanded velocity, the terrain becomes easier.

Learning for Policy Transition: High-level planning policy is utilized to switch between low-level policies, ensuring compliance with robustness constraints. For the High-level policy, the input is the same set of observations as used by the low-level policies, with the output being a two-dimensional Q-value. During training, two trained low-level policies are loaded and rolled out to generate training data for optimizing the high-level policy.
We train this high-level policy in the simulation environment mentioned detailed in Section [4.2](https://arxiv.org/html/2503.00923v3#S4.SS2), focusing on locomotion goals as the primary task. The objective is to enable the robot to track goal commands across a variety of terrains. The details of the training process is in Algorithm [1](https://arxiv.org/html/2503.00923v3#alg1)

Figure: Algorithm 1 Training for High-level Policy using Double-DQN

Reward Details: To better facilitate efficient humanoid locomotion, our task reward is defined as [28, 67]:

$$ $\displaystyle r_{\mathfrak{T}}$ $\displaystyle=\alpha_{1}\exp\left(-\frac{\|v_{\text{xy}}^{c}-v_{\text{xy}}\|_{ 2}^{2}}{\sigma_{\text{lin}}^{2}}\right)+\alpha_{2}\exp\left(-\frac{\|w_{\text{ z}}^{c}-w_{\text{z}}\|_{2}^{2}}{\sigma_{\text{ang}}^{2}}\right)$ $$

where 1) $v_{\text{xy}}^{c}$ denotes the commanded linear velocity along the $x$ and $y$ axes, while $w_{\text{z}}^{c}$ is the commanded yaw velocity. 2) $v_{\text{xy}}$ and $w_{\text{z}}$ represent the corresponding base velocities of the humanoid robot. 3) $\alpha_{1}$ and $\alpha_{2}$ indicate the hyperparameters that are utilized to adjust the importance of the different velocity terms. 4) $\sigma_{\text{lin}}^{2}$ and $\sigma_{\text{ang}}^{2}$ control the precision of the expected command tracking. Smaller values of $\sigma_{\text{lin}}^{2}$ and $\sigma_{\text{ang}}^{2}$
enhance the precision of command tracking for humanoid robots. However, they may hinder reward acquisition during the initial stages of training.

The recovery policy shares the same task reward formulation as the goal-tracking policy. However, it uses larger values of the $\sigma_{\text{lin}}^{2}$ and $\sigma_{\text{ang}}^{2}$, which implies a greater tolerance for velocity tracking errors. As a result, this reward term serves as a back-tracking reward for the safety recovery mechanism, encouraging it to return to a stable goal-tracking state. To further promote stable posture restoration and enable smooth transitions back to the goal-tracking policy, we introduce an additional stand reward, defined as:

$$ $r_{\text{stand}}=\|\mathbf{q}_{t}-\mathbf{q}_{\text{default}}\|_{2}^{2}.$ $$

where $\mathbf{q}_{\text{default}}$ represents a default standing pose.

For safety realistic deployment, we add some energy and safety reward for the low-level policies training. These rewards are shown in Table.

**Table 7: Safety & Energy Reward**
| Term | Expression | Weight |
| --- | --- | --- |
| DoF position limits | $\mathbf{1}(\mathbf{q}_{t}\notin[\mathbf{q}_{\min},\mathbf{q}_{\max}])$ | $-1e^{2}$ |
| DoF acceleration | $\|\ddot{\mathbf{q}}_{t}\|_{2}^{2}$ | $-1e^{-7}$ |
| DoF velocity | $\|\dot{\mathbf{q}}_{t}\|_{2}^{2}$ | $-5e^{-4}$ |
| Action rate | $\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|_{2}^{2}$ | $-2e^{-3}$ |
| Torque | $\|\tau_{t}\|$ | $-1e^{-5}$ |
| Collision | $\sum_{i\in\text{contact}}\mathbf{1}\{\|F_{i}\|>1.0\}$ | $-10$ |

### A.3 PPO

In order to enhance the understanding of historical observation information, we have made some improvements to the Actor-network. We first process this information using an encoder network to preliminarily extract the features of the observation values at each time step, and then use a merger network to integrate the observation values at different time steps. Finally, we obtain fixed-dimensional feature information, which is then input into the Actor backbone network together with other information.

**Table 8: PPO hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Actor Layer | [512, 256, 128] |
| Critic Layer | [512, 256, 128] |
| Activation Function | ELU |
| Discount Factor | 0.99 |
| GAE Parameter | 0.95 |
| Epochs per Rollout | 5 |
| Minibatch | 4 |
| Entropy Coefficient | 0.005 |
| Value Loss Coefficient ($\alpha_{1}$) | 1.0 |
| Clip Range | 0.2 |
| Learning Rate | 2.e-4 |
| Environment Steps per Rollout | 24 |

### A.4 VAE

The VAE Network consists of an encoder and a decoder. The encoder takes historical observations as input and outputs latent representation and estimation of priviledged information. The decoder takes latent as input and outputs the reconstruction of the next observation. The structure of VAE is shown in Figure [6](https://arxiv.org/html/2503.00923v3#A1.F6)

Figure: Figure 6: VAE Structure
Refer to caption: extracted/6450070/figure/VAE.png

### A.5 Discriminator

The input to the discriminator $s^{d}$ is defined as $[q,v,w,r,p,k]$, where $k$ represents the local positions of key points. Specifically, we define 12 key points, including the hips, shoulders, hands, ankles, knees, and elbows. For the Unitree H1, the total input dimension is 63, while for the Unitree G1, it is 67.

The discriminator’s hyperparameters are shown in Table. [10](https://arxiv.org/html/2503.00923v3#A1.T10).

**Table 10: Discriminator hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Discriminator Hidden Layer Dim | [1024,512] |
| Replay Buffer Size | 500000 |
| Expert Buffer Size | 200000 |
| Expert Fetch Size | 512 |
| Learning Batch Size | 4096 |
| Learning Rate | 2e-5 |
| Gradient Penalty Coefficient | 1.0 |
| lamda | 1.0 |

### A.6 Saliency Analysis of Observations

We adopt an integrated gradients method similar to [70] to assess the importance of different parts of the observation to the policy, as shown in Fig.[7](https://arxiv.org/html/2503.00923v3#A1.F7).

For the recovery policy, the Zero Moment Point (ZMP) encoding proves to be the most critical factor, followed by the estimated velocity. The robot’s current and historical proprioceptive states also contribute substantially to the policy’s decision-making process. In contrast, the environmental latent representation exhibits only a marginal effect. This hierarchy of importance reflects the necessity of strictly adhering to ZMP constraints to maintain dynamic stability and ensure the safety of the robot during recovery.

For the high-level policy, the Zero Moment Point (ZMP) encoding remains the most influential component, indicating that it provides critical information for decision-making and substantially enhances the policy’s awareness of the robot’s stability. This observation implies that the high-level policy prioritizes stability-relevant features over task-irrelevant ones, which is advantageous for promoting generalization across diverse scenarios. Moreover, the environmental latent variable exhibits greater importance compared to that in the recovery policy, suggesting that high-level decisions are more sensitive to environmental factors such as terrain variations.

Figure: Figure 7: Importance of Observations
Refer to caption: x6.png

### A.7 Extreme Initialization

To simulate extreme scenarios that may occur in real-world deployment, we identify three primary causes of such cases: external disturbances, hardware malfunctions, and malicious commands. Additionally, we retain domain randomization to enhance the generalization of physical dynamics within the simulation environment. The specific details are outlined as follows:

To enhance robustness and generalization in simulation, we simulate both extreme deployment scenarios and apply domain randomization. The extreme scenarios include external disturbances, sensor and actuator failures, and adversarial commands, while domain randomization targets physical variability in the environment.

1) External disturbances: We apply external disturbances to every component of the robot. The external forces and torques are applied along the x, y, and z axes, with each direction sampled from the range of -200 N to 200 N for forces and -200 Nm to 200 Nm for torques. These disturbance forces and torques are updated at a frequency of 1 Hz.

2) Hardward magnification : Random, high-intensity noise is introduced to the proprioceptive information and PD gains.
The noise added to the proprioceptive data simulates sensor errors that may occur during real-world deployment. The noise applied to the PD gains simulates fluctuations in motor strength, reflecting potential issues in the robot’s actuators. The parameters for this noise are detailed in Table [11](https://arxiv.org/html/2503.00923v3#A1.T11).

3) Malicious velocity: Velocity commands are resampled from the command range, with updates occurring at a frequency of 5 Hz. Notably, this resampling mechanism is activated once the terrain curriculum exceeds half of its total progression.

4) Domain randomization (for environment variability): Domain randomization is applied to all the training environments. This technique introduces variability in the environment’s parameters, such as friction and gravity. The specifics of the domain randomization are provided in Table [13](https://arxiv.org/html/2503.00923v3#A1.T13).

To improve the generalization of recovery ability, we utilize random initialization with extreme case initialization. The parameters are shown in Table [13](https://arxiv.org/html/2503.00923v3#A1.T13)

**Table 11: Intense Noise Parameters**
| Parameter | Range | Operator |
| --- | --- | --- |
| DOF Position Noise | 0.04 | additive |
| DOF Velocity Noise | 0.30 | additive |
| Angular Velocity Noise | 1.00 | additive |
| IMU Noise | 0.40 | additive |
| Gravity Noise | 0.10 | additive |
| PD Gains | 0.20 | additive |

**Table 12: Domain Randomization Parameters**
| Parameter | Range | Operator |
| --- | --- | --- |
| Friction | [0.6, 2.0] | additive |
| Base Mass | [-1.0, 5.0] | additive |
| Base CoM | [-0.07, 0.07] | additive |
| Motor Strength | [0.8, 1.2] | scaling |
| Action Delay | [0, 5] | - |
| Link Mass | [0.7, 1.3] | scaling |
| Gravity | [-0.1, 0.1] | additive |

Extreme-Case Analysis: We record nearly 60000 normal state data with randomly sampled commands in diverse terrain settings in [8](https://arxiv.org/html/2503.00923v3#A1.F8).
We record nearly 10000 extreme cases in the $\mathfrak{P}_{\alpha}^{L}$. Some of the terminal states are visualized in Fig. [9](https://arxiv.org/html/2503.00923v3#A1.F9). For the extreme states, the distribution of velocity is close to the command ranges. Roll pitch and their velocity are in a common range, which is near zeros. For the normal states, the distribution of linear velocity overlaps with the command velocity range, with a dense cluster around the high-velocity value of 2 m/s. Regarding yaw velocity, most values fall within the commanded range. However, there are instances of high speed, with some even approaching 10 m/s. This may be due to the robot’s extreme behaviors when encountering unseen situations influenced by external forces and random noise. Regarding the roll-pitch velocity, the pitch velocity shows a higher degree of fluctuation. Similarly, in the roll pitch distribution, for pitch, there is a concentrated distribution at a pitch of 1 m/s, which is one of the terminal conditions. These two figures suggest that the robot is tripped by obstacles during its motion and thus terminates, for instance, when it hits steps or gravel during high-speed motion.

Figure: Figure 8: Normal State: Policy rollout’s state distribution in the original learning environment
Refer to caption: extracted/6450070/figure/Velocity_normal2.png

Figure: Figure 9: Extreme State: Policy’s state distribution in the extreme cases
Refer to caption: extracted/6450070/figure/Linear_Velocity.png

### A.8 History Length Experiments

We investigate the impact of observation history length on HWC-Loco’s performance. Setting $H=10$ achieves the best overall trade-off across the three metrics. While increasing $H$ beyond 10 slightly improves the success rate, it leads to a noticeable decline in goal-tracking accuracy and human-likeness. This suggests that excessively long history may hinder the effective use of privileged information, resulting in more conservative and less adaptive behaviors.

**Table 14: Performance comparison with different history lengths ($H$)**
| History Length | Success Rate $\uparrow$ | Goal-tracking $\uparrow$ | Human-like $\downarrow$ |
| --- | --- | --- | --- |
| 1 | $95.13\pm 0.51$ | $1.07\pm 0.00$ | $3.26\pm 0.06$ |
| 10 | $96.73\pm 0.40$ | $\mathbf{1.10\pm 0.00}$ | $\mathbf{3.18\pm 0.01}$ |
| 20 | $96.53\pm 0.40$ | $1.08\pm 0.00$ | $3.18\pm 0.02$ |
| 40 | $\mathbf{97.38\pm 0.33}$ | $1.02\pm 0.00$ | $3.21\pm 0.02$ |

### A.9 Training Cost

All policies are trained on an NVIDIA RTX 4090 GPU with 4,096 parallel environments. Our framework involves training three policies. The goal-tracking policy converges within approximately 10000 iterations (nearly 10 hours), while the recovery policy requires around 8000 iterations to converge (nearly 8 hours). The high-level policy demonstrates faster convergence, requiring approximately 6000 iterations (under 6 hours).

### A.10 Deployment Details

We utilize the Unitree H1 humanoid robot as our deployment platform. This full-sized humanoid robot weighs 47 kg, stands approximately 1.8 meters tall, and features 19 degrees of freedom. The control frequency is set to 100 Hz for both simulation and real-world deployment. The PD gains, characterized by stiffness and damping values, used in the real-world deployment are detailed in Table [15](https://arxiv.org/html/2503.00923v3#A1.T15).

During deployment, inference operates through a hierarchical structure involving both high-level and low-level policies. To meet the strict latency and resource constraints of onboard systems, the policies are optimized for lightweight designs and exported JIT format. The high-level policy model size is below 2MB, and the low-level policies remain under 3MB. Those designs ensures that real-time inference at 100 Hz is feasible on the onboard computer of the Unitree H1 robot.

**Table 15: Stiffness, Damping and Torque Limit**
| Joint Names | Stiffness [N*m/rad] | Damping [N*m*s/rad] | Torque Limit (Nm) |
| --- | --- | --- | --- |
| hip yaw | 200 | 5 | 170 |
| hip roll | 200 | 5 | 170 |
| hip pitch | 200 | 5 | 170 |
| knee | 300 | 6 | 255 |
| ankle | 40 | 2 | 34 |
| torso | 200 | 5 | 170 |
| shoulder | 30 | 1 | 34 |
| elbow | 30 | 1 | 18 |

## Appendix B Experiment

### B.1 H1 Experiments

Effectiveness Tests in the Real World. Figure [10](https://arxiv.org/html/2503.00923v3#A2.F10) demonstrates the robot’s ability to traverse a multi-terrain course consisting of 15 cm steps, 20° slopes, and wooden boards at a speed of 0.3 m/s. Figure [11](https://arxiv.org/html/2503.00923v3#A2.F11) shows the ability to stability in real-world outdoor locomotion, including flat ground, grassy surfaces, and slopes.

Figure: Figure 10: Climb Stairs Test. The blue segments indicate the activation of the goal-tracking policy, while the orange segments correspond to the safety recovery policy.
Refer to caption: extracted/6450070/figure/Parkour.png

Figure: (a) Walking on flat terrain.
Refer to caption: extracted/6450070/figure/Forward_walking.png

Robustness Tests in the Real World. As shown in Figure [12](https://arxiv.org/html/2503.00923v3#A2.F12), we apply various disturbances to Unitree H1 lasting for one minute. Four representative disturbance segments were selected for analysis. The time is measured in control cycles, with each cycle lasting 0.01 seconds. Furthermore, we conduct an outdoor disturbance test by applying pushes, pulls, and kicks to the robot, as shown in Figure [13](https://arxiv.org/html/2503.00923v3#A2.F13). The robot waves its arms and adjusts its gaits to maintain balance. Notably, the high control frequency (100 Hz) ensures smooth transitions between policies, and isolated activations of the recovery policy have minimal impact on overall stability.

Figure: Figure 12: Disturbances in Realistic Deployment: Policy Switching value of 0 corresponds to the goal-tracking policy, while a value of 1 denotes the safety recovery policy.
Refer to caption: extracted/6450070/figure/Disturbance_Test.png

Figure: Figure 13: Robustness in Outdoor Settings: The robot responds to external disturbances in an outdoor environment by waving its arms and adjusting its gaits to maintain balance.
Refer to caption: extracted/6450070/figure/Outdoor_disturbance.png

Case Study on Motion Tracking. We visualize the robot’s performance in tracking a punching motion. During the execution, the robot experiences an impulse impact. In response, it dynamically activates the recovery policy, switching to more conservative and safer actions. Specifically, it lowers its body, adjusts its gait, and stabilizes itself to satisfy ZMP constraints, rather than strictly following the original target motion. As a result, it generates a backtracking motion to maintain balance and ensure robustness.

Figure: Figure 14: Robustness in Motion Tracking: The green dots represent the target keypoints, the blue dots indicate the keypoints corresponding to the DoFs, and the red dot denotes the ZMP.
Refer to caption: extracted/6450070/figure/Motion_recovery.png

### B.2 G1 Experiments

**Table 16: Comparison of Humanoid Configurations**
| Parameter | H1 | G1 |
| --- | --- | --- |
| Height | 178 cm | 127 cm |
| Mass | 47 kg | 35 kg |
| DoFs | 19 | 23 |

G1 outperforms H1 across all three metrics (success rate, goal-tracking accuracy, and human-likeness) primarily due to the following factors:

- •
Size and Mass: As shown in Table [16](https://arxiv.org/html/2503.00923v3#A2.T16), G1’s smaller size and lighter mass contribute to lower inertial loads and reduced torque demands. This facilitates greater stability, especially during dynamic motions, and enables faster corrective actions in response to disturbances.
- •
Degrees of Freedom (DoFs): With 23 DoFs compared to H1’s 19, G1 offers enhanced joint-level flexibility. The additional DoFs, particularly in the upper body and feet, allow for better whole-body coordination. This increased redundancy leads to improved disturbance rejection and more human-like behavior during complex locomotion tasks.

Notably, as illustrated in Figure [15](https://arxiv.org/html/2503.00923v3#A2.F15), G1 is capable of achieving straight-knee walking, which remains a significant challenge for H1 due to its larger size and reduced agility. The additional DoFs in G1 enable finer control of limb trajectories, which is critical for maintaining balance during such biomechanically demanding gaits.

Figure: Figure 15: G1 walk with straight knees
Refer to caption: extracted/6450070/figure/G1_walk.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. the paper compares HWC-Loco against standard baselines. Could the authors provide an ablation study comparing the full HWC-Loco (trained with Robust CRL, Eq. 4) against a version trained with the simpler Constrained RL (Eq. 2) or even a standard RL with a large penalty for constraint violation? This would isolate the benefit of the max-min robust optimization component specifically.

2. The high-level planner is key to the dynamic trade-off. What is the exact trigger mechanism for switching from the goal-tracking policy to the safety-recovery policy? Is it a simple threshold on a stability metric (e.g., ZMP distance from the support polygon), or is it a learned policy itself? Please elaborate on the input features and the decision logic of the planner.

3. The safety-recovery policy enforces ZMP-based constraints. Is this policy trained to be general across all tasks and terrains, or is it specialized? If it is a fixed, model-based controller, please state this clearly. If it is a learned policy, how is its robustness guaranteed, and how does it interact with the goal-tracking policy's learned dynamics?

4. The Sim2Real success is impressive. Were any specific system identification or domain randomization techniques used to tune the simulation parameters to match the real robot before training? If so, please detail these steps, as they are often critical for successful Sim2Real transfer in complex systems like humanoids.
soundness: 3
strengths: The paper presents a novel and well-motivated combination of existing concepts. The formulation of humanoid locomotion as a robust constrained RL problem (Equation 4) is a significant and original contribution, moving beyond simple reward-shaping for safety. The hierarchical architecture, which explicitly and dynamically switches between a task-maximization mode and a safety-guarantee mode, is a practical and elegant solution to the classic robustness-vs-performance trade-off in safety-critical systems.

This work is highly significant for the field of humanoid robotics. Robust and reliable locomotion is a critical bottleneck for real-world deployment. By providing a principled way to integrate safety guarantees (via ZMP-based constraints in the recovery policy) with high-performance task execution (via the goal-tracking policy), HWC-Loco offers a foundational advancement. The demonstrated Sim2Real success and superior robustness under external disturbances suggest a practical and deployable control framework.
summary: This paper, "HWC-LOCO," proposes a novel hierarchical whole-body control (HWC) approach for robust humanoid locomotion. The core contribution is a hierarchical policy designed to dynamically resolve the trade-off between aggressive goal-tracking and conservative safety-recovery, particularly under environmental disturbances and Sim2Real mismatch. The authors frame the policy learning as a robust constrained reinforcement learning (CRL) problem, maximizing task rewards while ensuring worst-case feasibility constraints across an uncertainty set of transition dynamics. The hierarchical structure consists of a high-level planner that switches between a task-oriented goal-tracking policy (trained with a mimic learning objective for natural, human-like motion) and a stability-focused safety-recovery policy (enforcing ZMP-based constraints). The method is evaluated extensively in simulation and on a real-world humanoid platform, demonstrating superior robustness and performance compared to state-of-the-art baselines across various terrains and
weaknesses: 1. While the overall formulation is novel, the concept of a hierarchical controller switching between a task-policy and a recovery-policy is not entirely new in robotics (e.g., in model-based control or even some prior RL works). The paper's novelty rests heavily on the robust constrained RL formulation that trains this hierarchy. The authors should more explicitly discuss and contrast their hierarchical training approach with prior hierarchical execution methods to better highlight the distinction.

2. The robust optimization objective (Eq. 4) depends on the uncertainty set parameter $\alpha$ and the feasibility constraint $\epsilon$. The paper mentions that $\alpha$ specifies the scale of mismatch, but the sensitivity of the final policy's performance and robustness to the choice of $\alpha$ is not thoroughly explored. A poor choice of $\alpha$ could lead to either an overly conservative or insufficiently robust policy. More detailed analysis or guidance on selecting these critical parameters would strengthen the work.

3. Robust RL methods, especially those involving a max-min objective or sampling from an uncertainty set of dynamics, are typically computationally expensive. The paper does not provide sufficient detail on the training time or computational resources required for HWC-Loco compared to the baseline methods. Given the complexity of whole-body humanoid control, this is a crucial practical consideration that should be addressed.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1) What are some of the failure cases and what caused them?
2) What sort of domain randomization was applied in simulation?
soundness: 4
strengths: The paper is well written, structured and easy to understand. The paper tackles a challenging problem and the proposed approach is appropriately motivated and positioned well among related work. Real world robust humanoid locomotion is a challenging task and the results presented in this paper is quite impressive. 

The paper provides sufficient experimental details and extensive validation. Tests across a wide range of conditions such as terrains, task commands and disturbances provide insightful details about the policies performance. Comparisons against relevant baselines and ablation studies shed light on the effectiveness of the proposed approach well.
summary: The authors present a learning based hierarchical control framework for humanoid robots that ensures task completion while maintaining safety requirements. A high level planning policy, trained using double DQN, selects between a goal tracking policy and a safety recovery policy based on robots state and historical observations. The paper presents impressive results in simulation as well as real world experiments, across various tasks and terrains. The extensive evaluation demonstrates superior performance over comparable baselines.
weaknesses: One of the weaknesses I can think of is the complex framework - this involves training two lower level policies separately, then a high level planner , GAN discriminator, and to enable real world deployment a VAE encoder that estimates privileged information that the policy has during training in simulation. These design choices might be hard to reproduce or deploy.

Since the networks are not trained jointly, there could be switching behavior between the two low level policies due to errors in the system. For example, if the VAE estimator has noisy estimates of privileged information.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - how sensitive the tracking/robustness trade off to the imitation multiplier lambda? would the increase of human-like behavior would lead to less recovery agility?
    
- is it possible to distill two policies into a single one to enable smooth transition by teacher-student training?
soundness: 3
strengths: - the hierarchical approach that target for safety and performance trade-off is natural and well-motivated.
    
- the performance of controller is evaluated comprehensively with diverse metrics to show is robustness and naturalness.
summary: The paper proposes HWC-Loco, a hierarchical approach for humanoid locomotion balancing safety and agility. The policy is switching between two policy classes: a goal-tracking policy which imitate human motion and a recovery policy trained with robust objective. Then, a high-level planner is learned to switch between two policies optimizing for overall tracking performance. The results is verified in sim and real world.
weaknesses: - comparison mainly ephasize dreamwaq and hal, where recent strong baselines for humanoid locomotion without hierarchical design like [1] as well as the same style switch controller like [2] is missing.
    
- it would be beneficial if the author could further discuss the sim2real gap identified in the real world deployment and explains with quantitative result on how those sensing and actuation gap hurts the standard policy design and why the hierarchical design can improve on it.
    

[1] AdaMimic: Towards Adaptable Humanoid Control via Adaptive Motion Tracking, Huang et al

[2] Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion, He et al.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Why does the baseline Goal-tracking consistently outperform AHL/DreamWaQ?
2. Some videos on the project website (e.g., Soft and Slippy Terrain) were not visible at my review time. Is this an accessibility issue on my side, or were those videos not yet uploaded?
soundness: 2
strengths: 1. HWC-Loco quantitatively improves task success rates and demonstrates reasonable high-level policy switching to maintain safety.
2: The high-level policy and hierarchical architecture are well-designed, effectively balancing goal-tracking and safety recovery.
3. The paper presents extensive experiments, and the overall structure and clarity of the manuscript are strong.
summary: This paper presents HWC-Loco, a hierarchical whole-body control framework for robust humanoid locomotion. It combines a goal-tracking policy for task performance and human-like motion with a safety policy for handling safety-critical scenarios with a high-level planner that can dynamically switches between them to keep safe and precise. By formulating policy learning as a robust optimization problem under mismatched environmental dynamics, HWC-Loco ensures safety while maintaining efficiency, demonstrating superior performance across diverse terrains, robots, and tasks in both simulation and real-world deployments.
weaknesses: 1. The paper lacks comparison to commonly used baselines such as larger domain randomization or larger domain randomization combined with history-aware policies. While the authors argue that “excessive regularization can greatly affect the efficiency of control policy,” they do not provide evidence that HWC-Loco achieves a better trade-off between efficiency and safety relative to these methods. (If these issues are addressed with stronger evidence, I’d be happy to raise my score.)
2. Performance/Evaluation Concerns: 1) The terrain experiments do not clearly illustrate when or how the high-level policy switching occurs, making it difficult to understand the specific contribution of the safety policy. 2) The terrains and challenge cases used are relatively simple. The paper should include scenarios that highlight the necessity of the hierarchical design—i.e., cases where success cannot be achieved without it or with standard domain randomization alone. Providing qualitative comparisons would help distinguish HWC-Loco from these baselines. 
3. The reliance on ZMP constraints is not that generalizable or robust. ZMP-based control depends heavily on accurate robot dynamics modeling and sensing, which can undermine the reliability (though I agree that integrating it within hierarchical, rather than enforcing hard constraints, is a more flexible design). Moreover, for tasks involving rapid or aerial motions (e.g., jumping) or requiring more agile locomotion, enforcing ZMP-based safety objectives may conflict with task goals, thereby restricting the effectiveness of the safe policy. Finally, its applicability to loco-manipulation under external forces remains limited


Decision: Accept (Poster)