# Latent Conditioned Loco-Manipulation Using Motion Priors

Maciej Sępień, Rafael Kourdis, Constant Roux, Olivier Stasse

► To cite this version:

Maciej Sępień, Rafael Kourdis, Constant Roux, Olivier Stasse. Latent Conditioned Loco-Manipulation Using Motion Priors. 2025 IEEE-RAS 24th International Conference on Humanoid Robots (Humanoids), Sep 2025, Seoul, South Korea. pp.365-372, 10.1109/Humanoids65713.2025.11203023. hal-05242643

HAL Id: hal-05242643

https://hal.science/hal-05242643v1

Submitted on 5 Sep 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
Latent Conditioned Loco-Manipulation Using Motion Priors

Maciej Sępień<fnref n="1" />,<fnref n="2" />, Rafael Kourdis<fnref n="1" />,<fnref n="3" />, Constant Roux<fnref n="1" /> and Olivier Stasse<fnref n="1" />,<fnref n="2" />

<figure><img src="image_1.png" /><figcaption>Fig. 1: Our Solo12 loco-manipulation policy is able to execute and smoothly transition between locomotion and manipulation to reach a specified target.</figcaption></figure>

**Abstract**— Although humanoid and quadruped robots provide a wide range of capabilities, current control methods, such as Deep Reinforcement Learning, focus mainly on single skills. This approach is inefficient for solving more complicated tasks where high-level goals, physical robot limitations and desired motion style might all need to be taken into account. A more effective approach is to first train a multipurpose motion policy that acquires low-level skills through imitation, while providing latent space control over skill execution. Then, this policy can be used to efficiently solve downstream tasks. This method has already been successful for controlling characters in computer graphics. In this work, we apply the approach to humanoid and quadrupedal loco-manipulation by imitating either simple synthetic motions or kinematically retargeted dog motions. We extend the original formulation to handle constraints, ensuring deployment safety, and use a diffusion discriminator for better imitation quality. We verify our methods by performing loco-manipulation in simulation for the H1 humanoid and Solo12 quadruped, as well as deploying policies on Solo12 hardware. Videos and code are available at https://gepetto.github.io/LaCoLoco/

# I. INTRODUCTION

Traditional methods of robot control such as reinforcement learning and trajectory optimization have been successful in enabling legged robots to perform a wide variety of skills such as walking, running or even parkour [1]. These methods work by optimizing a cost metric that encodes information about task success, motion style, constraints and other characteristics of the desired robot motion.

Despite the power and generality of such techniques, designing this metric is a laborious process, commonly known to require extensive trial and error. For example, while defining the cost necessary to achieve single motions is feasible, balancing individual costs to optimize a multipurpose policy is not trivial. Qualities such as the style that motions

should follow can require multiple cost terms that need to be carefully designed, like in [2] [3].

Imitation learning is a promising approach that removes the need for explicit cost functions. Instead, policies are optimized in a data-driven way, by aiming to generate motions that imitate demonstration examples. Instead of encoding a desired behavior through cost functions, one only needs to provide examples to be imitated.

In computer graphics, controlling physically-simulated characters poses a similar problem to robot control. In order to achieve reusable character control policies, [4] introduced Adversarial Skill Embeddings (ASE), a method for imitating motion datasets and generating actions via latent space control. The generative approach used in ASE, enables combining and transitioning smoothly between learned motions. Furthermore, it is possible to simultaneously imitate different motion datasets, regardless of their source. By observing only states instead of state-action pairs [5] [6] the approach can also be applied to sources such as animal motion demonstrations, where the action space is not observable [7].

Motion generation that is controllable through a meaningful latent space instead of working directly in configuration space can be beneficial when solving complex control tasks. A latent space control policy can be interpreted as a decoder from latent to configuration space that ensures the generated motions will be close to the motions it was trained to imitate.

We seek to better utilize the capabilities of legged robots by developing policies that can perform both walking and manipulation skills. In this work, we explore the application of ASE to humanoid and quadrupedal loco-manipulation by first training low-level imitation policies on various motion datasets. Then, we use these policies to solve the problem of reaching points with an end-effector. To facilitate safe hardware deployment, we extend the original method to allow for explicit constraints during training. Additionally, we propose an alternative discriminator to achieve better quality imitation.

<footnote n="1">¹LAAS-CNRS, Université de Toulouse, France, firstname.lastname@laas.fr</footnote>

<footnote n="2">²Artificial and Natural Intelligence Toulouse Institute, France</footnote>

<footnote n="3">³University of Edinburgh, School of Informatics, UK</footnote>
This work proposes the following contributions:

* Development of multi-skill latent conditioned policies for humanoid and quadrupedal loco-manipulation through adaptation of ASE

* Demonstration of successful imitation of synthetic motions from different base controllers, as well as trajectories obtained from dog motion capture data

* Extension of the original ASE method with stochastic constraints and diffusion-based discriminator

* Validation and comparison of methods on the task of end-effector point tracking for simulated H1 and Solo12 robots as well as Solo12 hardware

To the best of our knowledge, this work is the first to successfully apply latent conditioned imitation learning and demonstrate autonomous skill switching between locomotion and manipulation on a quadruped hardware platform.

## II. RELATED WORK

### A. Motion Imitation

Imitation learning is a data-driven approach for designing controllers by aiming to mimic a dataset of expert behaviors. GAIL [5] introduced an imitation learning technique for physics-based control tasks based on Generative Adversarial Networks (GANs) [8] and reinforcement learning. This approach was further extended by AMP [9] to train stylized task-specific policies, by formulating a style reward as imitation. Subsequently, [7] applied AMP to the task of quadrupedal velocity tracking while also imitating a dog. While this method successfully trained hardware controllers that achieve a task in a desired style, these controllers are single-purpose and cannot be reused to achieve a different task than the one originally trained for.

### B. Latent Space Control

To train task-agnostic policies that can perform a diverse set of skills, [10] proposed conditioning a policy on a latent variable and optimizing a mutual information objective. This approach was adapted by [11] to learn a multi-skill quadrupedal locomotion policy conditioned on a discrete latent variable, able to perform different types of gaits. This method is not fully multipurpose, as the controller is trained with a task-specific reward. In ASE [4], a hierarchical control approach with a reusable low-level controller is trained instead. The controller is conditioned on a continuous latent representation, enabling smooth transitions between skills.

### C. Loco-manipulation

The combination of self-mobility (locomotion) and object interaction (manipulation) has been studied for both humanoid robots [12] and quadrupeds. For quadrupeds, [13] proposed a dual-policy approach to perform loco-manipulation using one of the robot's legs. A single-purpose controller was proposed instead by [14], able to perform both behaviors by optimizing a task specific cost. [15] demonstrated the ability to use a similar low-level controller combined with a high-level diffusion policy to execute tasks such as lifting a basket.

For humanoid robots, loco-manipulation capabilities are crucial for enabling human-like environment interaction. To this end, [16] proposed a generalist policy for controlling humanoids that is able to track combinations of kinematic, joint or base-related targets. While this scheme is general enough to allow for a variety of behaviors, we speculate that a meaningful latent space would be a more suitable interface for achieving high-level tasks using a general policy.

### D. Constraints

One key difference between animated character control and robotics is that during robot control, it is of vital importance to take physical limitations of the hardware into account. Limits such as maximum torque or kinematics can be modeled as simulation constraints during policy training, however others such as ground reaction forces cannot. In order to explicitly allow for constraints during reinforcement learning, [17] stochastically terminates future policy rewards on constraint violation, therefore encouraging constraint satisfaction.

## III. PRELIMINARIES

Our proposed hierarchical control approach is based on Adversarial Skill Embeddings (ASE) [4], with two key differences. Firstly, our method supports constraints through stochastic reward termination [17], without requiring difficult-to-tune reward penalties. Taking constraints into account enables us to train safer and better-behaved policies for hardware deployment. Secondly, we use a diffusion discriminator instead of the original GAN-style model to achieve higher quality imitation.

### A. Adversarial Skill Embeddings

The ASE framework [4] consists of two stages: a general-purpose low-level policy controllable by a continuous latent $z$, and a high-level task-specific policy that generates latents to achieve a desired goal.

The low-level policy $\pi(a|s, z)$ provides a distribution over agent actions $a$ that would result in motions imitating a provided dataset $M$. The generated behavior depends on information encoded by the latent skill variable $z$. The policy $\pi$ is trained using reinforcement learning, based on the standard discounted Markov Decision Process (MDP) framework:

$$ \max_{\pi} \mathbb{E}_{p(\tau|\pi)} \left[ \sum_{t=0}^{T-1} \gamma^t r_t \right], \quad (1) $$

where $\gamma$ is the discount factor, $T$ the time horizon, $\tau = \{s_0, a_0, r_0, s_1, \dots, s_{T-1}, a_{T-1}, r_{T-1}, s_T\}$ a trajectory sampled from the policy $\pi$, and $r_t$ the reward at timestep $t$.

For the low-level policy, a combined reward is optimized:

$$ r_t = r_D(s_t, s_{t+1}) + \beta r_E(s_t, s_{t+1}) \quad (2) $$

where $r_D$ is a motion imitation objective, $r_E$ a skill discovery objective, and $\beta$ a weight hyperparameter. We provide more details on these objectives below.

After $\pi$ is trained on a motion dataset $M$, it can be used for downstream tasks. In ASE, a high-level policy is
trained to generate latents z that guide π towards achieving task-specific goals. In the original formulation, an additional motion prior reward is used when training the high-level policy which we omit for simplicity.

1) Imitation objective: The low-level policy imitation term $r_D$ encourages the distributions of state transitions from the policy $d^\pi(s_t, s_{t+1})$ and the dataset $d^M(s_t, s_{t+1})$ to match as closely as possible:

$$r_D = -JS(d^\pi(s_t, s_{t+1}) \| d^M(s_t, s_{t+1})) \quad (3)$$

where JS is the Jensen-Shannon distribution divergence. Maximizing $r_D$ means that the policy will generate motions that imitate the dataset.

Calculating the JS divergence as written in Eq. 3 can be computationally intractable. To combat this issue, ASE [4] uses a GAN-based variational approximation to calculate the distribution distance, introduced in [8]:

$$\min_D \quad -\mathbb{E}_{d^M(s_t, s_{t+1})} [\log(D(s_t, s_{t+1}))] \\ -\mathbb{E}_{d^\pi(s_t, s_{t+1})} [\log(1 - D(s_t, s_{t+1}))] \quad (4)$$

where $D(\cdot, \cdot)$ is a discriminator function that aims to classify whether a state transition was produced by the policy or is part of the dataset.

In ASE, $D(s_t, s_{t+1})$ is calculated directly by a neural network $D_{net}$.

The discriminator network $D_{net}$ and the low-level policy are trained at the same time. During training, the discriminator aims for the generated behaviors to be (from the discriminator's perspective) as if they originated from the motion dataset.

The policy reward term is:

$$r_D = -\log(1 - D_{net}(s_t, s_{t+1})) \quad (5)$$

2) Skill Discovery Objective: Aside from the imitation objective $r_D$, ASE additionally uses a skill discovery term $r_E$ while training the low-level policy. This term maximizes the mutual information $I(\cdot, \cdot)$ between z and state transitions $(s_t, s_{t+1})$ produced by $\pi$:

$$r_E(s_t, s_{t+1}) = I(s_t, s_{t+1}; z|\pi) \\ = \mathcal{H}(s_t, s_{t+1}|\pi) - \mathcal{H}(s_t, s_{t+1}|z, \pi) \quad (6)$$

where $\mathcal{H}(\cdot)$ is the entropy function.

As in [10], optimizing $r_E$ ensures that the distribution of state transitions is specific to that z; in other words, z will control the behavior generated by the low-level policy. However, due to the fact that optimizing this formulation is intractable, the following lower bound is maximized instead:

$$I(s_t, s_{t+1}; z|\pi) \geq \max_q \mathcal{H}(z) \\ + \mathbb{E}_{p(z)} \mathbb{E}_{p(s_t, s_{t+1}|\pi, z)} [\log q(z|s_t, s_{t+1})] \quad (7)$$

where $q(z|s_t, s_{t+1})$ is called the skill encoder. The encoder $q(z|s_t, s_{t+1})$ is modeled as a von Mises-Fisher distribution:

$$q(z|s_t, s_{t+1}) = \frac{1}{Z} \exp(\kappa \mu_q(s_t, s_{t+1})^T z) \quad (8)$$

where $\mu_q$ is predicted by a neural network, $\kappa$ is a scaling factor and Z a normalization constant. In order to achieve the maximization in Eq. 7, the encoder is trained jointly during policy training. The expectations are approximated by using samples of z and state transitions:

$$\max_q \quad \mathbb{E}_{p(z)} \mathbb{E}_{d^\pi(s_t, s_{t+1}|z)} [\kappa \mu_q(s_t, s_{t+1})^T z] \quad (9)$$

In this case, the distribution of z is assumed to be constant, so $H(z)$ does not change and can be omitted. Using the above, the low-level policy skill discovery reward that maximizes $I(s_t, s_{t+1}; z|\pi)$ is therefore:

$$r_E(s_t, s_{t+1}) = \log q(z_t|s_t, s_{t+1}) \quad (10)$$

3) Diversity Term: In addition to optimizing the low-level policy reward, ASE uses a separate diversity objective during optimization that encourages similar latents z to produce similar actions:

$$\mathbb{E}_{d^\pi(s)} \mathbb{E}_{z_1, z_2 \sim p(z)} \left[ \left( \frac{D_{KL}(\pi(\cdot|s, z_1), \pi(\cdot|s, z_2))}{D_z(z_1, z_2)} - 1 \right)^2 \right] \quad (11)$$

where $D_z(z_1, z_2) = 0.5(1 - z_1^T z_2)$ is the cosine distance between latents and $D_{KL}(\cdot, \cdot)$ the KL divergence.

4) Latent Space: The high-level policy in ASE generates unnormalized latents that are then projected to the hypersphere before being passed to the low-level controller. This is so that the high-level policy is able to control the entropy of the distribution of z, allowing for skill specialization.

During training of the low-level policy, z is drawn from a latent space defined as the hypersphere with unit norm:

$$\bar{z} \sim \mathcal{N}(0, I), \quad z = \frac{\bar{z}}{\|\bar{z}\|} \quad (12)$$

## B. Constraints as Terminations

Constraints as Terminations (CaT) [17] introduced a method for enforcing explicit constraints during MDP optimization by extending the original formulation in Eq. 1. CaT encourages constraint satisfaction by including a probability of future reward termination depending on whether the policy violates the constraints:

$$\max_{\pi} \quad \mathbb{E}_{p(\pi|\pi)} \left[ \sum_{t=0}^{T-1} \left( \prod_{t'=0}^{t} \gamma(1 - \delta(s_{t'}, a_{t'})) \right) r_t \right] \quad (13)$$

where $\delta(s, a)$ is a random variable defined as a function of constraint violation $c_i(s, a)$. The i-th constraint is violated at state s with action a, if $c_i(s, a) > 0$.

This formulation probabilistically terminates policy reward if constraints are violated, steering the optimization towards obeying the constraints.
CaT uses an intuitive $\delta$ which increases when constraints are violated more, up to a maximum probability $p_i^{max}$:

$$ \delta = \max_{i \in I} p_i^{max} \cdot \text{clip} \left( \frac{c_i^+}{c_i^{max}}, 0, 1 \right) \quad (14) $$

where $c_i^+ = \max(0, c_i(s, a))$ is the violation of constraint $i$, and $c_i^{max}$ a filtered measure of maximum constraint violation during the last batch of interaction with the environment.

### C. Diffusion-Reward Adversarial Imitation Learning

An alternative to imitation learning approaches such as [5], which use a discriminator neural network trained by Eq. 4, is using diffusion models [18] as a discriminator. This was used and extended by Diffusion-Reward Adversarial Imitation Learning (DRAIL) [6]. The enhanced conditional diffusion discriminator of DRAIL was shown to produce a more robust reward value and exhibit less overfitting to expert demonstrations than the GAN-based method.

Instead of the original state-action formulation, we modify DRAIL slightly to work on state transitions. The DRAIL discriminator aims to classify whether a transition was part of an expert demonstration or generated by the policy. This is achieved in a single denoising step, by calculating the diffusion loss:

$$ \mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l) = \mathbb{E}_{t_d \sim T_d} \left[ \| \epsilon_{\phi}(s_t, s_{t+1}, \epsilon, t_d | l) - \epsilon \|^2 \right] \quad (15) $$

where $(s_t, s_{t+1})$ is a state transition, $\epsilon \sim \mathcal{N}(0, I)$ the applied noise, $\epsilon_\phi$ the predicted noise, $t_d$ the diffusion timestep and $l \in \{l^M, l^\pi\}$ a binary condition label where $l^M$ denotes a transition from the dataset and $l^\pi$ one from the policy.

$\mathcal{L}_{\text{diff}}$ is used to measure how well the distribution of policy state transitions matches the motion dataset by comparing noise predictions. For example, if a state transition was sampled from the dataset, the predicted noise for label $l^M$ would match the actual noise, setting $\mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l^M)$ close to 0. $\mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l^\pi)$ would in this case be large. The reverse applies to transitions generated by the policy.

In order to obtain a reward signal from $\mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l)$, a sigmoid classifier is constructed:

$$ D_{\text{diffusion}}(s_t, s_{t+1}) = \sigma \left( \mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l^\pi) - \mathcal{L}_{\text{diff}}(s_t, s_{t+1}, l^M) \right) \quad (16) $$

where $\sigma(\cdot)$ is the sigmoid function. As in the GAN-style formulation [5], the low-level policy reward is defined as:

$$ r_D(s_t, s_{t+1}) = - \log(1 - D_{\text{diffusion}}(s_t, s_{t+1})) \quad (17) $$

where $D(s_t, s_{t+1})$ is provided in this case by the diffusion model. To calculate the expectation in Eq. 15, multiple $t_d$ are sampled during training. The $\epsilon_\phi$ network is trained directly using the discriminator loss.

<figure><img src="image_2.png" /><figcaption>Fig. 3: H1 policy transitioning from walking to reaching</figcaption></figure>

## IV. METHOD

By leveraging unstructured motion datasets, our method aims to learn reusable robot policies that can perform and switch between multiple skills. We base our work on the hierarchical framework of Adversarial Skill Embeddings (ASE) with two key modifications:

* We use the diffusion discriminator introduced by DRAIL [6] instead of the GAN-based discriminator in ASE [4]. We have qualitatively observed that the diffusion discriminator generally results in higher-quality motions that are closer to the demonstration dataset.<fnref n="1" />

* We allow for constraints to be explicitly specified during low-level policy training by using the Constraints as Terminations (CaT) [17] method. This is crucial for successful deployment of our method to hardware, as the policy can be trained to respect limits which are difficult to enforce during simulation. We show that limiting ground reaction forces during imitation results in more gentle policies and smoother motions. We also demonstrate that enforcing constraints on the low-level policy is enough; motions generated by a task-specific high-level policy still respect constraints set during low-level policy training.

These modifications proved key in enabling our method to generate high-quality, safe, multi-skill policies that are applicable to robot hardware.

## V. APPLICATION TO HUMANOID AND QUADRUPEDAL LOCO-MANIPULATION

We demonstrate the effectiveness of our method in imitating a variety of motion datasets and learning reusable skills, by focusing on humanoid and quadrupedal loco-manipulation. We select the loco-manipulation task in order to investigate whether a policy trained with our method can learn to perform and switch smoothly between the distinct motions required when locomoting and reaching for a point in space. Furthermore, we would like to investigate whether the latent space learned by training the low-level policy allows for precise robot control.

<footnote n="1">¹Please refer to the supplementary video for a comparison of motions generated by each method.</footnote>
We formulate the loco-manipulation task similarly to [14]. The task reward to be optimized is a function of end-effector distance from a point in 3D space:

$$r_g = \exp(-10\|x_{eef} - x_d\|) \qquad (18)$$

where $x_{eef}$ is the current end-effector position and $x_d$ the position of the destination point.

For our humanoid experiments, we test using the Unitree H1 robot in simulation and select the right hand as the end-effector. For our quadruped experiments, we use the Solo12 open-source robot [19] in simulation and real life. The front right foot is selected as the end-effector.

## VI. EXPERIMENTAL SETUP

To evaluate the behavior of our method, examine the generated motions and verify its applicability to hardware, we train policies for the loco-manipulation task on various motion datasets. For H1, we compile a synthetic dataset of separate walking and reaching-in-place motions. These correspond to the two fundamental skills required for loco-manipulation. We also generate a similar dataset for Solo12.

To further demonstrate the generality of our method, we investigate whether the low-level policy can be also trained to imitate physically-infeasible motions. We use a dataset of dog motions collected using a motion capture system [20], that we kinematically retarget to the morphology of Solo12 without regard to dynamic feasibility.

For all experiments, we perform an ablation comparing the base ASE [4] method, ASE with the DRAIL [6] discriminator, as well as both versions trained using CaT [17] to limit ground reaction forces. We examine the end-effector tracking accuracy of each method, the amount of time that constraints are in violation and how many times the robot fell while executing the task.

### A. Motion Datasets

1) **Synthetic Datasets:** For both H1 and Solo12, we generate separate locomotion and reach-in-place motions for low-level policy training. For H1, we obtain walking trajectories by training the example velocity tracking policy in IsaacLab [21] and randomly sampling velocity commands. We obtain reaching trajectories by randomly sampling close position commands and training a simple policy that optimizes end-effector position tracking. We follow a similar procedure for the Solo12 dataset. We generate walking motions using the policy in [17] and reaching motions using the pedipulation policy in [14].

In both cases, we sample targets for the reaching motions that are close enough to the robot to be reachable without walking. Overall, we collect 120 examples of walking and 20 examples of reaching, each lasting 10 seconds.

2) **Dog Motion Dataset:** We generate a dataset of dog motions for the Solo12 by kinematically retargeting motion capture (mocap) data from [20]. For each trajectory in the source mocap data, we formulate an optimization problem per timestep to find a robot configuration that minimizes the

<figure><img src="image_3.png" /><figcaption>Fig. 4: Solo12 policy learns to extract and switch between skills encountered in dog motion dataset</figcaption></figure>

distance of selected robot frames from the equivalent dog keypoints:

$$\min_{q_t} \sum_{k=1}^{K} \|p_{fk}^{k}(q_t) - \hat{p}_{t}^{k}\|^2 + \alpha \left\| q_{t}^{\text{joints}} - q_{\text{neutral}}^{\text{joints}} \right\|^2 \quad (19)$$

where $q_t$ is the configuration of the robot at timestep $t$, K the number of keypoints to be matched, $p_{fk}^k(q_t)$ the world position of the k-th frame at configuration $q_t$, $\hat{p}_t^k$ the equivalent position of frame $k$ at time $t$ in the mocap motion and $\alpha = 0.005$ a weight factor. The second regularization term encourages the joint positions $q_t^{\text{joints}}$ to not deviate too much from a default joint configuration $q_{\text{neutral}}^{\text{joints}}$.

We calculate $p_{fk}^k(q_t)$ for each frame using forward kinematics. For our experiments, we select the foot and shoulder frames to be matched by the optimization, eight in total. This is adequate to obtain motions that are close to the source set. We optimize using Ipopt [22] by warm-starting the next timestep with the output configuration of the previous timestep.

### B. Observations

Drawing inspiration from [7], our discriminator has access to a subset of the full robot state (observation space) to use as features when producing the imitation reward. For both Solo12 and H1, we construct this subset using the following components: base height, base linear and angular velocities, gravity vector projection to the base frame, joint positions and velocities. The observation space used for the low-level policy consists of the same state components, with the addition of the last generated policy action.

### C. Training

We use the Isaac Lab [21] simulator to train policies using Proximal Policy Optimization (PPO) [23]. For all methods, we simulate 4096 environments in parallel. To facilitate easier transfer of controllers from simulation to hardware, we add noise to policy observations and randomize frictions and masses during training.

1) **Low-level Policy:** For all methods, the frequency of the low-level policy is 50 Hz. The policy is trained for 7000 epochs and uses a 7-dimensional continuous latent space.

For methods using CaT, we limit the maximum ground reaction force per leg to ensure robot movements are gentle on the hardware. For H1, we set this limit to 1.5 times
<table><caption>TABLE I: Simulation results for H1 and Solo12 (↓) (best policy among 5 seeds)</caption><thead><tr><th>Experiment</th><th>Method</th><th>End-Effector<br>Mean Error ± Std [cm]</th><th>Contact Force<br>Violation [%]</th><th>Falls [%]</th></tr></thead><tbody><tr><td rowspan="4">H1 Synthetic</td><td>ASE</td><td>14.43 ± 8.18</td><td>0.7189</td><td>0.0244</td></tr><tr><td>ASE DRAIL</td><td>7.46 ± 3.81</td><td>1.4394</td><td>0.0000</td></tr><tr><td>ASE CaT</td><td>11.51 ± 6.11</td><td>0.0198</td><td>0.3418</td></tr><tr><td>ASE DRAIL CaT</td><td>10.24 ± 5.76</td><td>0.3249</td><td>0.0000</td></tr><tr><td rowspan="4">Solo12 Synthetic</td><td>ASE</td><td>3.45 ± 1.44</td><td>0.6632</td><td>0.0488</td></tr><tr><td>ASE DRAIL</td><td>2.72 ± 3.89</td><td>0.8725</td><td>0.1709</td></tr><tr><td>ASE CaT</td><td>3.47 ± 7.35</td><td>0.0426</td><td>0.0000</td></tr><tr><td>ASE DRAIL CaT</td><td>2.51 ± 2.74</td><td>0.0892</td><td>0.0488</td></tr><tr><td rowspan="4">Solo12 Dog Imitation</td><td>ASE</td><td>19.98 ± 12.18</td><td>1.6981</td><td>0.0000</td></tr><tr><td>ASE DRAIL</td><td>9.74 ± 16.36</td><td>0.7926</td><td>0.8301</td></tr><tr><td>ASE CaT</td><td>26.64 ± 14.08</td><td>0.1433</td><td>0.7812</td></tr><tr><td>ASE DRAIL CaT</td><td>11.85 ± 15.95</td><td>0.1632</td><td>0.7568</td></tr></tbody></table>

the weight of the robot. For Solo12 we use 25 N which is approximately the weight of the robot. To encourage skill acquisition, we set $p_i^{max} = 0$ at the beginning of training. Towards the end, $p_i^{max}$ is increased linearly to 0.1 for H1 and 0.2 for Solo12, to ensure adherence to constraints.

2) *High-level Policy:* For all experiments, we train the high-level policy using the end-effector tracking cost of Eq. 18 for 2000 epochs. The frequency of the policy is 10 Hz.

To train the Solo12 controller, we sample destination points on a cylinder of radius 1.5 m at height in the range (0.16, 0.42) m. To ensure that the robot would have to combine both locomotion and reaching to achieve the goal, we randomly initialize its position near the center of the circle.

For H1, we always initialize the robot at the origin. We randomly sample destination points in the rectangle $x: (0.3, 4.0)$, $y: (-1.0, 1.0)$, $z: (1.0, 1.4)$ m.

## VII. RESULTS

In all loco-manipulation experiments, we compute the end-effector position error by using the last episode step. We train all four methods on each dataset using 5 seeds and report the best policy, picked based on performance and fall count. For the Solo12 hardware experiments we deploy the best controllers as measured in simulation.

### A. Simulation

In simulation, all methods successfully train policies that perform and switch between skills to achieve the loco-manipulation task, as seen in Figures 3 and 4. Surprisingly, this is also the case for methods imitating the retargeted dog motion dataset.

The reaching performance of all methods that train on the synthetic datasets is similar, as can be seen from Table I. In these experiments, methods using the DRAIL discriminator yielded only slightly better errors. In experiments that train on the dog motion dataset, DRAIL results in a more significant improvement.

All methods that use CaT during training to limit ground reaction forces show a reduced constraint violation time. The

fall count over 4096 episodes is generally small and does not seem to follow an obvious trend.

Training-wise, all Solo12 policies demonstrate some switching from locomotion to reaching. For H1 however, DRAIL methods are more unstable to train, despite the increased accuracy achieved in the best case. We observed that in most cases, policies trained with DRAIL for H1 did not learn to switch motions for the entire episode.

Qualitatively, we notice that methods trained on the synthetic datasets generally behave similarly. The difference between methods is significantly more pronounced when training on the dog motion dataset. In this case, the original ASE method tends to generate motions that are more harsh and oscillatory. Adding CaT results in visibly more graceful motions. The DRAIL discriminator has the largest impact on motion quality. DRAIL results in significantly smoother movement that resembles the original dataset better, as well as stable reaching. Please refer to the supplementary video for a visual comparison of these policies.

### B. Hardware

In addition to simulation tests, we use a real-life Solo12 robot to evaluate all four policies trained using the synthetic motion dataset. The results are presented in Table II. We collect 20 measurements for each method, starting from random positions and orientations. The end-effector target position is set to $\hat{p} = (0, 0, h)$, where $h$ is a random height sampled from the range (0.16, 0.42) m for each trial but shared across all methods. We use a motion capture system to track the position and linear velocity of the robot.

<table><caption>TABLE II: Hardware results for Solo12 Synthetic (↓) (best policy in simulation among 5 seeds)</caption><thead><tr><th>Method</th><th>End-Effector<br>Mean Error ± Std [cm]</th><th>Falls [%]</th></tr></thead><tbody><tr><td>ASE</td><td>7.25 ± 3.37</td><td>0.00</td></tr><tr><td>ASE DRAIL</td><td>5.64 ± 5.52</td><td>5.00</td></tr><tr><td>ASE CaT</td><td>9.03 ± 6.19</td><td>0.00</td></tr><tr><td>ASE DRAIL CaT</td><td>5.62 ± 2.76</td><td>0.00</td></tr></tbody></table>

All policies are successful in combining locomotion and reaching to achieve the loco-manipulation task. Accuracy is
similar for all, with DRAIL achieving only slightly better
results.

The qualitative difference in policy behavior however, is
more significant. ASE without CaT generates aggressive
locomotion that jumps when reaching the target, leading
to excessive stress on the hardware. With our Solo12, we
observe that these motions are likely to trigger a hardware
issue, where motor drivers are sometimes unable to track
high accelerations. When this occurs, we repeat the experi-
ment as the robot shuts down.

Methods trained with DRAIL generate more gentle lo-
comotion trajectories on the hardware, which resemble the
motion dataset more closely. Furthermore, as expected, poli-
cies trained with CaT are generally even less aggressive and
more stable. No policy resulted in significant loss of balance,
with only ASE DRAIL falling in 1 out of 20 trials.

We did not manage to successfully deploy any of the
policies trained on the dog motion dataset. Unfortunately,
no policy was able to balance long enough when walking to
achieve the task. We hypothesize that the gait style in the
dog dataset is more difficult to learn and execute, since the
feet are close together when walking. This results in a small
support polygon, making balance more demanding. Given
that all methods were successful in simulation but not in
real life, model mismatch might be a potential explanation
why these policies did not work.

VIII. DISCUSSION AND FUTURE WORK

In this work, we demonstrate the application of hierarchi-
cal latent conditioned control in robotics. We show that by
training on unlabeled motion datasets, a single controller can
extract reusable skills that can be used to achieve high-level
tasks on hardware. This approach does not require expensive
tuning of cost functions, or annotations of the source dataset.

Our experiments prove that providing a latent space in-
terface to a task-specific controller is adequate to enable
precise robot control. We observe that policies select and
switch between distinct locomotion and reaching behaviors
in order to achieve a task in both simulation and hardware.

In addition, we adapt the original ASE method to use
a diffusion discriminator, as well as support constraints for
safer hardware deployment. We explore the quantitative and
qualitative differences of these methods and demonstrate
successful imitation of motions from different sources. We
finally show that these methods work on real hardware, and
perform ablation experiments to examine the impact of our
proposed improvements.

While our approach is promising, this is still preliminary
work. We consider the number of imitated motions and
high-level tasks we trained to be limited. Furthermore, the
applicability of our H1 policies on hardware is yet to be
validated. Other open problems include extending the method
to locomotion on rough terrain and integrating perception
information to the high-level policy.

ACKNOWLEDGEMENTS

We would like to thank Thomas Flayols and Sabrina Evans
for their insightful comments and suggestions.

This research was supported by ANITI IA Cluster project 23-IACL-0002 and ANR-19-PI3A-0004. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD011016368 made by GENCI.

REFERENCES

[1] C. Tang et al. "Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes". In: *Annual Review of Control, Robotics, and Autonomous Systems* 8. Volume 8, 2025 (2025), pp. 153–188. ISSN: 2573-5144.

[2] N. Rudin et al. “Learning to walk in minutes using massively parallel deep reinforcement learning”. In: *Conference on Robot Learning*. 2022.

[3] Z. Li et al. “Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control”. In: *The International Journal of Robotics Research* 44.5 (2025), pp. 840–888.

[4] X. B. Peng et al. “ASE: large-scale reusable adversarial skill embeddings for physically simulated characters”. In: *ACM Trans. Graph*. 41.4 (July 2022).

[5] J. Ho and S. Ermon. “Generative adversarial imitation learning”. In: *Proceedings of the 30th International Conference on Neural Information Processing Systems*. NIPS’16. 2016, pp. 4572–4580.

[6] C.-M. Lai et al. “Diffusion-Reward Adversarial Imitation Learning”. In: *Advances in Neural Information Processing Systems*. Vol. 37. 2024, pp. 9545–9548.

[7] A. Escontrela et al. “Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions”. In: *2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*. 2022, pp. 25–32.

[8] I. Goodfellow et al. “Generative adversarial networks”. In: *Commun. ACM* 63.11 (Oct. 2020), pp. 139–144.

[9] X. B. Peng et al. “AMP: adversarial motion priors for stylized physics-based character control”. In: *ACM Trans. Graph*. 40.4 (July 2021).

[10] B. Eysenbach et al. “Diversity is all you need: Learning skills without a reward function”. In: *International Conference on Learning Representations* (2019).

[11] C. Li et al. “Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions”. In: *2023 IEEE International Conference on Robotics and Automation (ICRA)*. 2023, pp. 2944–2950.

[12] Z. Gu et al. “Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning”. In: *arXiv preprint arXiv:2501.02116* (2025).

[13] X. Cheng, A. Kumar, and D. Pathak. “Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion”. In: *2023 IEEE International Conference on Robotics and Automation (ICRA)*. 2023, pp. 5106–5112.
[14] P. Arm et al. "Pedipulate: Enabling Manipulation Skills using a Quadruped Robot's Leg". In: 2024 IEEE International Conference on Robotics and Automation (ICRA) (2024), pp. 5717-5723.

[15] Z. He et al. "Learning Visual Quadrupedal Loco-Manipulation from Demonstrations". In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2024, pp. 9102-9109.

[16] T. He et al. "HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots". In: arXiv preprint arXiv:2410.21229 (2024).

[17] E. Chane-Sane et al. "CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning". In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2024, pp. 13303-13310.

[18] J. Ho, A. Jain, and P. Abbeel. "Denoising diffusion probabilistic models". In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS '20. 2020.

[19] F. Grimminger et al. "An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research". In: IEEE Robotics and Automation Letters 5.2 (2020), pp. 3650-3657.

[20] H. Zhang et al. "Mode-adaptive neural networks for quadruped motion control". In: ACM Trans. Graph. 37.4 (July 2018).

[21] M. Mittal et al. "Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments". In: IEEE Robotics and Automation Letters 8.6 (2023), pp. 3740-3747.

[22] A. Wächter and L. T. Biegler. "On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming". In: Mathematical Programming 106 (2006), pp. 25-57.

[23] J. Schulman et al. "Proximal Policy Optimization Algorithms". In: arXiv preprint arXiv:1707.06347 (July 2017).