Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning

## Abstract

Abstract Representation learning and unsupervised skill discovery can allow robots to acquire diverse and reusable behaviors without the need for task-specific rewards.
In this work, we use unsupervised reinforcement learning to learn a latent representation by maximizing the mutual information between skills and states subject to a distance constraint. Our method improves upon prior constrained skill discovery methods by replacing the latent transition maximization with a norm-matching objective. This not only results in a much a richer state space coverage compared to baseline methods, but allows the robot to learn more stable and easily controllable locomotive behaviors. We successfully deploy the learned policy on a real ANYmal quadruped robot and demonstrate that the robot can accurately reach arbitrary points of the Cartesian state space in a zero-shot manner, using only an intrinsic skill discovery and standard regularization rewards.

## 1 Introduction

Learning dynamic motions for legged robots typically requires a combination of complex reward engineering, imitation learning and curriculum learning. This can be limiting in two ways: firstly, reward engineering can be a tedious and time-consuming process, which does not scale well to learning a large range of behaviors; secondly, it constrains the problem through the injection of strong inductive bias. For example, in locomotion it is common to have a reward for moving directly towards the goal — behavior, which can be very sub-optimal if there are any obstructions between the robot and the target.
In contrast, skill discovery, or options learning, is a subset of representation learning that presents a promising solution to this problem. It encourages the agent to autonomously explore its environment and learn diverse behaviors with an intrinsic motivation reward (Salge et al., ; Choi et al., ). The main advantage of this approach is that the agent can learn general behaviors that not only do not require as much handcrafting and reward engineering but can also be easily reused for downstream tasks.

Skills in a latent representation are typically learned by maximizing an information theoretic objective, such as the mutual information between skills and state trajectories (usually with a discriminator) (Eysenbach et al., ; Choi et al., ), or diversity in terms of expected features and state distributions under different skills (Zahavy et al., ; Laskin et al., ; Cheng et al., ). However, one of the main limitations of prior methods is that they define mutual information through the Kullback-Leibler (KL) divergence, which does not consider the degree to which different behaviors are distinguishable (Park et al., ). To learn “useful” behaviors —for example state-traversing locomotive behaviors— it is common to constrain the encoder to focus only on certain parts of the observation space, which also injects a strong inductive bias into the problem. Prior work (Park et al., ; ) has proposed maximizing the latent transitions in the skill space under a Lipschitz constraint. While such methods can learn skills spanning a great part of the state-space without additional inductive bias, the maximization objective leads to only acquiring overly aggressive high-velocity motions, which cannot be deployed on most robotics hardware.

In this work, we explore the application of skill discovery via representation learning to the domain of quadrupedal locomotion. We introduce an enhanced constrained skill-learning objective that facilitates the robot’s acquisition of an extensive repertoire of locomotion skills. Our work builds upon prior methods for Lipschitz-constrained latent transition maximization Park et al. (; ). However, unlike these works, we explicitly avoid always maximizing the latent transition magnitude. The strength of our proposed method is that it learns more stable and controllable skills that cover a wider range of the state space, as shown in Fig. 1.
By employing a skill-conditioned policy, our approach enables the robot to navigate the environment at varying velocities. Our method achieves this without using task-related rewards, instead, it relies solely on intrinsic mutual information rewards and extrinsic regularization rewards.
Although this skill-conditioned policy facilitates accelerated learning of downstream tasks, we demonstrate that, even without additional training, the policy is sufficiently effective to achieve precise zero-shot goal tracking.
Our contributions are as follows:

- •
We introduce an unsupervised skill discovery approach for pre-training quadruped robots, enabling the acquisition of diverse locomotion skills using only skill discovery and regularization rewards.
- •
We propose a skill matching objective for constrained skill discovery, enabling the agent to learn a broader range of skills and reliably cover a larger portion of the state space compared to the baseline methods.
- •
Our method conditions the trained policy on a desired latent space transition, achieving accurate zero-shot goal-tracking on the ANYmal quadruped robot in the real world without the need for additional training.

Figure: Figure 1: We learn to map skill-conditioned state transitions to a latent space $\mathbb{Z}^{2}$ through an encoder $\phi(\cdot)$. By training a skill-conditioned policy together with the encoder, we learn a large range of locomotive skills on quadruped robots. Prior methods (Park et al., ; ) always maximize the latent transitions, leading to only learning less stable high-velocity motions regardless of the skill magnitude. In contrast, our proposed method learns a wider distribution of behaviors, which we can control by varying the magnitude of the sampled skills.
Refer to caption: x1.png

## 2 Related Work

Legged Locomotion: Reinforcement learning has shown great results in recent years when applied to legged locomotion, both for blind (Hwangbo et al., ; Lee et al., ; Kumar et al., ; Margolis et al., ; Yu et al., ; Gangapurwala et al., ) and for perceptive locomotion (Miki et al., ; Loquercio et al., ; Yang et al., ; Gangapurwala et al., ). The standard approach is to frame the learning problem as a velocity-following task, where the robot learns to track user-specified velocity commands under noise and various disturbances. One disadvantage of this method is that the policy often converges to a single gait behavior, unless additional rewards (Fu et al., ) or neural network structures (Yang et al., ) are included.
Recently, more works have explored learning navigation and locomotion, resulting in a goal-conditioned policy that can reach various points in the Cartesian state space while allowing the robot to adapt its velocity depending on the environment. Due to the sparsity of the task rewards, many works implement additional dense exploratory rewards (Rudin et al., ; Zhang et al., ; Cheng et al., ), or task-specific curricula (Li et al., ; Atanassov et al., ). Cheng et al. (), for example, use a simpler velocity-based reward but requires pre-specified oracle waypoints for the robot to follow as it learns to navigate more challenging environments. Other parkour works (Zhuang et al., ; Hoeller et al., ) require learning several manually defined skills, each with its own set of task-based rewards that require separate design and tuning.
To summarize, legged locomotion by reinforcement learning needs heavy tuning of task-specific rewards, which requires domain knowledge and human efforts.

Unsupervised Skill Discovery: To address the challenges of sparse reward problems that require strong and structured exploration, many works have proposed encouraging curious and exploratory behavior through intrinsic motivation. There are many methods for representing intrinsic motivation in reinforcement learning agents, which can be broadly grouped into three information-theoretic objectives - maximizing expected information gain, novelty, and empowerment (Salge et al., ) (also referred to as options learning and skill discovery) (Aubret et al., ). Of these three methods, skill discovery is particularly attractive as it promises to encourage exploration throughout training and to learn a reusable representation or set of behaviors for downstream tasks.

Many works commonly use the mutual information (MI) between skills and states, $I(z;s)$, as the optimization objective for learning more diverse behaviors (Choi et al., ). The MI can be expressed in two ways - the first being in terms of the skill-conditioned state entropy and the entropy of the states, i.e. $I({\mathbf{z}};{\mathbf{s}})=H({\mathbf{s}})-H({\mathbf{s}}|{\mathbf{z}})$ (Liu & Abbeel, ; Sharma et al., ; ; Laskin et al., ); or alternatively
as $I({\mathbf{z}};{\mathbf{s}})=H({\mathbf{z}})-H({\mathbf{z}}|{\mathbf{s}})$, where the skill entropy $H({\mathbf{z}})$ depends on the choice of distribution $z\sim p({\mathbf{z}})$, typically chosen as uniform and is therefore constant. As maximizing $H(z|s)$ directly is intractable, a variational approximation is used with the following lower bound:

$$ $\displaystyle I({\mathbf{z}};{\mathbf{s}})$ $\displaystyle=H({\mathbf{z}})-H({\mathbf{z}}|{\mathbf{s}})$ (1) $\displaystyle=-\mathbb{E}_{\mathbf{z}}[\log p({\mathbf{z}})]+\mathbb{E}_{{ \mathbf{s}},{\mathbf{z}}}[\log p({\mathbf{z}}|{\mathbf{s}})]$ (2) $\displaystyle\geq-\mathbb{E}_{\mathbf{z}}[\log p({\mathbf{z}})]+\mathbb{E}_{{ \mathbf{s}},{\mathbf{z}}}[\log q_{\phi}({\mathbf{z}}|{\mathbf{s}})]$ (3) $\displaystyle\geq c+\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[\log q_{\phi}({ \mathbf{z}}|{\mathbf{s}})],$ (4) $$

where $q_{\phi}(z|s)$ is a variational approximation of the conditional pdf $p(z|s)$. This can be approximated with a discriminator for both discrete (Gregor et al., ; Eysenbach et al., ; Achiam et al., ) and continuous (Choi et al., ; Hansen et al., ) skill spaces, and used as the reward function for the agent $\pi(a|s,z)$ in an RL setting. This works as a cooperative game, where both the agent’s and discriminator’s objectives are to maximize the lower bound. In VALOR (Achiam et al., ) the complete episodic trajectory is used to distinguish states, but it is more common to use combinations of current, next, initial or final states to simplify the problem (Gregor et al., ; Park et al., ).
In DISDAIN (Strouse et al., ) the reward is augmented to use the disagreement between an ensemble of discriminators, which helps address the problem of the discriminator making incorrect predictions on novel states it hasn’t encountered. The same intrinsic motivation formulation is also used in CASSI (Li et al., ) in combination with adversarial motion imitation (Peng et al., ) to learn discrete diverse skills from a motion dataset. Similarly, ASE (Peng et al., ) uses an encoder-based representation to learn continuous high-dimensional skills given a motion dataset.
One major disadvantage of the common formulations is the lack of exploration incentive (Laskin et al., ). In fact, as long as the discriminator can distinguish between skills there is no benefit to exploring more diverse states (Park et al., ). With a large enough discriminator, even a small difference in the states could lead to accurate skill predictions. This causes many unsupervised skill discovery methods to produce distinguishable skill-conditioned trajectories that cover a reasonably small part of the state-space. To tackle this, for locomotion it is common to add a high level of task inductive bias into the problem, for example by limiting the discriminator input to the XY position of the body (Eysenbach et al., ; Sharma et al., ). Alternatively, giving structure the problem by imitating a reference motion dataset (Peng et al., ; Li et al., ) or an optimal pretrained policy (Zahavy et al., ; Cheng et al., ) can also significantly help with converging to more diverse behaviors.
Recently, Park et al. () proposed an alternative formulation to align the transitions in a learned latent space with the corresponding skills and maximize their norm. Crucially, they then add a constraint on the norm of these latent transitions being a lower bound on a chosen distance metric, such as the Euclidean (LSD) (Park et al., ) or the temporal distance (METRA) (Park et al., ) between a transition pair in the original state space. This constraint ensures that skills that cover large latent distances also cover large state space distances under the chosen metric, leading to much larger coverage and more useful skills.

Unsupervised skill discovery has mainly been studied in relation to standard problems (such as the MuJoCo robotic environments) in the domain of reinforcement learning. Few skill discovery methods have been applied to robotics and tested in the real world. Sharma et al. () applied an off-policy version of DADS (Sharma et al., ) for real-world reinforcement learning on a quadruped robot. Recently, Cheng et al. () applied DOMiNO (Zahavy et al., ) to goal-conditioned quadrupedal locomotion, where the discrete skills are used to produce diverse trajectories that reach the same goal point. However, their approach requires an optimal pretrained policy, that is already capable of reaching the goal.

## 3 Methodology

Figure: Figure 2: Learning scheme for our proposed approach. The encoder $\phi(\cdot)$ maps state transitions into a latent space optimized to match the skills (sampled from a predefined distribution $p(z)$), as shown by the MSE loss. An intrinsic reward is given to the agent based on the loss magnitude and an extrinsic reward from the environment, which encourages smooth behaviors.
Refer to caption: x2.png

Our approach builds upon the method proposed in (Park et al., ; ), where we learn an encoder $\phi$ that maps state transition pairs to a latent space. As shown in Fig. 2, we sample a 2-D continuous skill vector and optimize both the encoder and the skill-conditioned policy $\pi({\mathbf{a}}|{\mathbf{s}},{\mathbf{z}})$ with the common objective of minimizing the prediction error between latent embedding and the sampled skill.

### 3.1 Skill Discovery

In (Park et al., ; ) the authors show that by parameterizing the discriminator in (4) with a normal distribution with mean $\phi({\mathbf{s}})$ and unit variance, the mutual information objective can be reformulated as follows:

$$ $\displaystyle I({\mathbf{z}};{\mathbf{s}})$ $\displaystyle\geq\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[\log q_{\phi}({ \textnormal{z}}|{\textnormal{s}})]+{\textnormal{c}}=-\frac{1}{2}\mathbb{E}_{{ \mathbf{s}},{\mathbf{z}}}[||\phi({\textnormal{s}})-{\mathbf{z}}||^{2}]+{ \textnormal{c}}$ (5) $$

The initial ${\mathbf{s}}_{0}$ and final states ${\mathbf{s}}_{T}$ of the episode are used as input, and $\phi(\cdot)$ is referred to as a latent space encoder with the following simplified decomposition used as the loss:

$$ $\displaystyle L$ $\displaystyle=-\frac{1}{2}\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[||(\phi({ \mathbf{s}}_{T})-\phi({\mathbf{s}}_{0}))-{\mathbf{z}}||^{2}]+{\textnormal{c}}$ (6) $\displaystyle=\underbrace{\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[(\phi({ \mathbf{s}}_{T})-\phi({\mathbf{s}}_{0}))^{T}(\phi({\mathbf{s}}_{T})-\phi({ \mathbf{s}}_{0}))]}_{\text{L2 Regularization}}-\underbrace{\frac{1}{2}\mathbb{ E}_{{\mathbf{s}},{\mathbf{z}}}[(\phi({\mathbf{s}}_{T})-\phi({\mathbf{s}}_{0})) ^{T}{\mathbf{z}}]}_{\text{Directional Alignment}}+\underbrace{\mathbb{E}_{{ \mathbf{s}},{\mathbf{z}}}[{\mathbf{z}}^{T}{\mathbf{z}}]+{\textnormal{c}}}_{ \text{Constants}}$ (7) $\displaystyle\geq-\frac{1}{2}\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[(\phi({ \mathbf{s}}_{T})-\phi({\mathbf{s}}_{0}))^{T}{\mathbf{z}}]=L^{LSD},$ (8) $$

where the squared skill term $\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[{\mathbf{z}}^{T}{\mathbf{z}}]$ can be ignored as it is constant (given a fixed skill sampling distribution). The effect of this simplified decomposition, which ignores the squared $\phi$ regularization term, is that the latent space transition $\phi({\mathbf{s}}_{T})-\phi({\mathbf{s}}_{0})$ should be aligned with the direction of the skill vector ${\mathbf{z}}$, and maximized. By combining this optimization objective with a constraint on the latent space transition norm of the form $\forall{\mathbf{s}}_{T},{\mathbf{s}}_{0}\in\mathcal{S},\quad||(\phi({\mathbf{s
}}_{T})-\phi({\mathbf{s}}_{0}))||\leq d({\mathbf{s}}_{T},{\mathbf{s}}_{0})$, their method prioritizes learning skills which maximize the corresponding distance (according to the chosen metric $d$) between initial and final states.

The drawback of the original LSD and METRA formulations is that the agent will always maximize the norm of the latent space transition (and thus the distance under the metric $d$). In a locomotion scenario, with Euclidean distance as the metric as in (Park et al., ), this would result in the agent accelerating as fast as possible in a different direction on the XY plane. This means that while the policy would cover a large part of the state space, most of these states would be transient - i.e. to reach the distant states, the agent has to pass through the near states, but it will not stay there. Staying within the locomotion scenario, this would translate to all skills having a large mean linear velocity, with no low velocity skills. This poses a significant problem if we want to deploy these motions on robot systems. In addition, these slower skills might have been necessary for downstream tasks, causing the performance to deteriorate. Therefore, we propose an alternative objective function while maintaining the same distance metric constraint as in (Park et al., ; ). Instead of only optimizing for the alignment between skills and latent trajectories, we also include norm matching. If we use a Mean Squared Error loss, we can recover the original more general formulation in (6), which can be seen as a continuous regressive discriminator form of DIAYN (Eysenbach et al., ; Choi et al., ). This results in the following objective:

$$ $J^{\mathrm{Ours}}=-\frac{1}{2}\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[||(\phi({ \mathbf{s}}_{T})-\phi({\mathbf{s}}_{0}))-{\mathbf{z}}||^{2}],\quad s.t.\quad|| \phi({\mathbf{s}}_{T})-\phi({\mathbf{s}}_{0})||\leq||{\mathbf{s}}_{T}-{\mathbf {s}}_{0}||$ (9) $$

To make the objective suitable for solving as an RL problem where we want to optimize the objective from a batch of environmental transitions, we define the per-step loss similarly to (Park et al., ):

$$ $J^{\mathrm{Ours}}_{t}=-\frac{1}{2}\mathbb{E}_{{\mathbf{s}},{\mathbf{z}}}[||N( \phi({\mathbf{s}}_{t+1})-\phi({\mathbf{s}}_{t}))-{\mathbf{z}}||^{2}],\quad s.t .\quad||\phi({\mathbf{s}}_{t+1})-\phi({\mathbf{s}}_{t})||\leq||{\mathbf{s}}_{t +1}-{\mathbf{s}}_{t}||,$ (10) $$

where ${\mathbf{s}}_{t}$ and ${\mathbf{s}}_{t+1}$ are a pair of consecutive states, and N is the number of steps in the episode. While this telescopic sum is not equal to the original formulation, as in (Park et al., ) (due to the use of the squared norm), we show in the Supplementary Materials that it is an upper bound.

To then encourage the skill-conditioned policy $\pi({\mathbf{a}}|{\mathbf{s}},{\mathbf{z}})$ to produce diverse motions, we give it a reward based on how accurate the discriminator is:

$$ $r(t)=\frac{1}{1+\sigma e},\quad\text{where}\quad e=||N(\phi({\mathbf{s}}_{t+1} )-\phi({\mathbf{s}}_{t}))-{\mathbf{z}}||^{2},$ (11) $$

where $\sigma$ is a scaling term to account for the magnitude scale of the loss.
An alternative way to interpret this objective is that $\phi(\cdot)$ is an encoder that compresses states into a latent space, and our objective pushes the agent to traverse this latent space in the same direction and magnitude as the skill it is conditioned on. By constraining this latent space transition with the state space Euclidean norm, we can ensure that we cover a distance of at least that magnitude in the original state space. Unfortunately, this upper bound does not guarantee that the real traveled distance will be of similar magnitude to the latent one. In fact, all skills could traverse a very large real space distance and still satisfy the constraint in (9). However, in practice, this was not an issue as the agent would prefer stationary behavior over highly dynamic motions due to its extrinsic regularization rewards.

We train the encoder $\phi(\cdot)$ using supervised learning using the objective in (10) and the RL policy using the intrinsic reward in (11), on state pairs from the environment and the skills they were produced under. This can be seen as a cooperative game, where the agent tries to produce skill-conditioned trajectories that are more easily distinguishable by the discriminator. Since the L2 loss can be more sensitive to large-scale values, we use the Smooth L1 loss instead for training the encoder $\phi(\cdot)$.

### 3.2 Reinforcement Learning Setup

Observations, actions and control framework. For the policy observations, we use a concatenated array of current joint positions and velocities, base linear and angular velocities, base quaternion and the previous actions.
While some previous skill discovery works (Eysenbach et al., ; Sharma et al., ) require constraining the discriminator input based on the task (e.g. to only the XY base position for locomotion tasks), this is not the case for our method, similarly to (Park et al., ). For this reason, we use the full observations, i.e. the base linear $\mathbf{v}_{b}$ and angular $\bm{\omega}_{b}$ velocities, base quaternion $\mathbf{\bar{q}}_{b}$, joint positions $\mathbf{q}_{j}$ and velocities $\mathbf{\dot{q}}_{j}$, and the base position $\mathbf{p}_{b}$.
The action space consists of desired joint positions for the 12 joints on the ANYmal robot relative to a nominal standing configuration. These are updated at $50\text{\,}\mathrm{Hz}\text{/}$ and summed up with the nominal joint positions and passed at $400\text{\,}\mathrm{Hz}\text{/}$ to low-level higher frequency PD controllers.

Reward design. To slightly bias the skill discovery towards feasible motions that can be applied to the real robot without damaging the hardware, we provide several extrinsic rewards commonly used in legged locomotion. These are energy conservation and smoothness terms, together with feet air time (to prevent high-frequency stepping), flat orientation and nominal base height rewards, and unwanted contact penalties. We note that these extrinsic rewards are only necessary to make the motions smoother and more aesthetically pleasant. In the supplementary material we conduct an ablation with and without those extrinsic regularization rewards, and show that they are not crucial for learning the diverse behaviors. In comparison to quadruped locomotion acquired by pure reinforcement learning, we use far fewer reward terms since we don’t need any task-specific rewards.

## 4 Experimental Results

We demonstrate the ability of our approach to discover a broad range of skills which are used to locomote to desired goal poses. These learned skills are used to achieve zero-shot goal-tracking in the real world. We compare our method with two state-of-the-art baselines:
(i) Lipschitz-constrained skill discovery (LSD) (Park et al., ) with the objective shown in (8) and a Euclidean distance constraint;
(ii) Metric-aware abstraction (METRA) (Park et al., ) with the same objective as LSD and a temporal distance constraint.
As in the baselines, we choose a 2D skill space, and therefore a 2D latent space for the encoder $\phi(\cdot)$, and sample the skills uniformly from within a circle of radius $||z_{\mathrm{max}}||=50$. We choose this value because for a 300 step episode it results in a per-step latent transition of $\sim 0.17$, which is the maximum magnitude achieved by LSD on the same task. For the rest of the experimental setup, please refer to the supplementary material.

### 4.1 State space coverage

We compare our approach to LSD and METRA in terms of the mean base velocity magnitude across the skill space, as illustrated in Fig. 3.
Both LSD and METRA exhibit a very narrow distributional range of base velocities, with almost all skills exhibiting high velocities. In contrast, our method can achieve a much more uniform spread, which enables the learning of both low- and high-speed locomotive behaviors.

Figure: Figure 3: Density distribution of the mean (across the episode) base velocity of 1000 trajectories with uniformly sampled skills, grouped into equally spaced bins in the range $0\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}3\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$. We show the results for the baseline LSD (in orange), METRA (in blue), and ours (in green). A broad distribution is a result of a larger skill space.
Refer to caption: x3.png

### 4.2 Skill discovery comparison with baseline methods.

In this section, we compare the performance in terms of XY state coverage with existing skill discovery methods - namely METRA (Park et al., ), LSD (Park et al., ), CASSI (Li et al., ), ASE (Peng et al., ) and DOMiNiC (Cheng et al., ). These methods have been shown to learn impressive diverse skills on both quadruped robots and physically simulated characters. For a fair comparison, we evaluate purely the skill discovery parts of the methods without the motion imitation dataset and objective (or the optimal pretrained policy in the case of DOMiNiC).

Figure: Figure 4: Comparison of XY base position trajectory (in meters) between ours, METRA, LSD, ASE, CASSI and DOMiNiC. To better illustrate the magnitudes of difference in the performance, we show the results with a fixed x- and y-axis scale across all algorithms. The colors indicate different skills (whether discrete or continuous). For ours, we show the performance when sampling skills with both the maximum magnitude, and with varying magnitudes in the first two plots, respectively.
Refer to caption: x4.png

As can be seen from Fig. 4 both LSD and METRA learn a simpler latent representation where the same latent (and real) transitions are achieved regardless of the magnitude of the sampled skill - and the magnitude of these transitions is always explicitly maximized. On the other hand, our method produces both low- and high magnitude transitions depending on the magnitude of the conditioned skill.
Two-dimensional ASE results in behaviors that do not traverse the XY state space, but only affect the joint positions of the robot. By constraining the encoder input to only the base XY position (ASE 2D XYO), or increasing the dimensionality of the skill space, the robot learns to locomote across the XY plane. However, compared to our method, ASE exhibits much lower state space coverage. The reason for this is that the minimization of the encoder loss only ensures diverse behaviors, but not the degree to which they differ. Similar observations were noted in (Park et al., ) when comparing to other methods like DIAYN and DADS. Even when constraining the encoder to only look at the XY position, it converges to small movements that are distinguishable but not useful for traversing the state space. We note that ASE uses almost the same objective as LSD and METRA — but with a von Mises-Fischer distribution rather than a Gaussian, a slightly modified loss ($\mathcal{L}=\phi(s,s^{\prime})^{T}z$ rather than $\mathcal{L}=(\phi(s^{\prime})-\phi(s))^{T}z$), and crucially, without the constraint on the latent distance seen in Eq. 10. We believe the lack of maximization of the latent transitions and lack of constraint between the real states and the latent ones results in the subpar performance of ASE compared to ours, LSD and METRA.
Similarly, when using the full observation as input to the discriminator, CASSI mostly relies on the joint position as a distinguishing feature for different skills. If the input is constrained to the XY position (CASSI 6 skills XYO), different skills result in small XY state space traversal. However, as in ASE, since there is no incentive to maximize this traversal, each skill only covers a small ($<1$m) part of the space. Our version of DOMiNiC does not normalize the successor features and thus the objective can maximize them to learn more diverse skills. With either the full observation or just the XY position as discriminator inputs (the latter not shown here for conciseness), DOMiNiC is not able to traverse the XY state space. However, when constraining the input to the base linear velocity only, the method can discover locomotive behaviors. The state space coverage is smaller compared to ours and the skills are discrete, which limits their potential scalability and usefulness for downstream tasks.

### 4.3 Zero-Shot Goal-Tracking

Figure: Figure 5: After training, we can plan in the learned latent representation to condition the policy to reach desired states. We encode the current state ${\mathbf{s}}$ and desired state ${\mathbf{s}}_{\mathrm{des}}$ into the latent space, and use that as the conditioning skill for the policy.
Refer to caption: x5.png

Figure: Figure 6: Goal-tracking comparison between Ours (top) and the baseline LSD (bottom). As our skill space can exhibit a larger range of velocities, it can accurately reduce its velocity, and reach and stay at the target. The color gradient on the trajectories indicates the change in skill direction.
Refer to caption: extracted/5913191/initial_submission/images/comparison_photos_updated.png

Figure: Figure 7: Position tracking performance and corresponding body position profile in hardware experiments. The robot is able to accurately track two successive target poses.
Refer to caption: x6.png

Similar to (Park et al., ), our method can also achieve zero-shot goal-tracking after training the skill-conditioned policy. As the latent space transition is aligned with its corresponding skill, we can invert the problem and select a skill based on the desired latent space transition, as ${\mathbf{z}}_{\mathrm{des}}=\phi({\mathbf{s}}_{\mathrm{des}})-\phi({\mathbf{s}
}_{t})$, where ${\mathbf{s}}_{\mathrm{des}}$ and $s_{t}$ are the desired final state and the current state, respectively. We set ${\mathbf{s}}_{\mathrm{des}}={\mathbf{s}}$ for all states except the base position, which is set to the desired goal. By conditioning the policy on the desired skill $\pi({\mathbf{s}}_{t},{\mathbf{z}}_{\mathrm{des}})$, the robot will produce a motion that reaches $\phi({\mathbf{s}}_{\mathrm{des}})$ from $\phi({\mathbf{s}}_{t})$ in $t_{ep}$ seconds, which is the duration of the episode throughout training. However, in order to ensure that the robot can stop at the desired state, we update the direction and magnitude of the desired skill ${\mathbf{z}}_{\mathrm{des}}$ at every step (in a closed loop). In contrast, if we keep ${\mathbf{z}}_{\mathrm{des}}$ constant and do not update it, the robot would move in the direction of the goal with a constant velocity, cross the goal and keep moving along the same direction. As shown in Fig. 6, the former results in highly accurate goal-tracking behavior without additional training. In comparison, while the baseline method moves in the direction of the goal, most of its skill space is dedicated to high velocity motions, causing the robot to repeatedly overshoot the target.
We further validated our approach on the real ANYmal robot, as shown in Fig. 7, where the robot has to track two consecutive targets. As can be seen, the robot can accurately reach and stop at the desired goal.

### 4.4 Goal reaching in higher dimensional (3-D) skill spaces

The encoder learns to compress its input into the latent space in a way that preserves both the magnitude and the direction of the corresponding skills. In this case, this results in mapping the base linear velocities and the base position and ignoring the remaining dimensions of the input states. The reason for this is that the encoder has a limited latent space dimensionality and those are the dimensions that can achieve the largest transition magnitudes.
An interesting hypothesis is that by increasing the latent dimensionality, the encoder should use a larger part of its input state space, and subsequently be able to perform zero-shot goal tracking on the rest of its state space. To investigate this, we trained our approach with a 3-dimensional latent (and skill) space. In Fig. 8 we condition the policy on tracking different base positions with an arbitrary desired yaw. As can be seen in the figure, the agent can successfully move to the desired point with the required heading simultaneously, which was not possible with 2-dimensional skills.

Figure: Figure 8: Tracking the desired position with desired orientation (shown by the direction of the red arrow), starting from the origin with a 3-D latent space encoder.
Refer to caption: extracted/5913191/initial_submission/images/3d_orientation.png

## 5 Conclusion

In this work, we presented a novel approach that uses unsupervised skill discovery to learn robust locomotion skills. We showed that these cover a much larger part of the state space when contrasted to baseline methods. We introduced an augmented mutual information objective function, so that an encoder learns to map input states to a latent space and predict both the direction and magnitude of the sampled skills. By combining this with a constraint on the Euclidean distance between states, as in previous work, our method resulted in a much richer state space coverage. While these skills can be used for various downstream tasks, in this work, we showed how the robot can accurately reach desired Cartesian positions in a zero-shot manner. Our method achieves this without any task-relevant rewards, unlike other works which require carefully engineered dense exploratory rewards. Compared to the baselines (Park et al., ; ) we achieve a much more uniform skill-conditioned state distribution. In contrast with other intrinsic motivation works that require an imitation dataset (Peng et al., ; Li et al., ) or a pretrained optimal policy (Zahavy et al., ; Cheng et al., ), our method can learn a diverse set of motions with very little prior structure.

Limitations and future work.
One limitation of our approach is the use of Euclidean distance as the distance metric. As we only take the norm of the transitions, large oscillatory motions (such as the joint position change) can contribute to the per-step constraint satisfaction, without being used by the encoder to distinguish the states. While this was not a problem for learning diverse locomotion skills, we will investigate other distance metrics as future work.

In this work we used up to 3-dimensional continuous skill spaces, which allow for control of both position and orientation of the robot. However, since our formulation is inherently more challenging to learn than the LSD and METRA baselines, as we learn both magnitude and direction, the encoder is more difficult to train for larger skill dimensions. A possible explanation is that the encoder input does not contain enough information to predict high dimensional skills. For example, some states like the pitch and roll cannot be changed much without resulting in the robot falling; other states like those describing the joint and base motion are co-dependent and intertwined for complex articulated systems like quadruped robots. Nevertheless, in future work, we will investigate whether more complex model structures can be leveraged to train accurately for even higher continuous skill dimensions.

An interesting research direction would be to evaluate the performance at more downstream tasks than just goal-tracking. Exploring the application of unsupervised skill discovery methods to loco-manipulation tasks is an exciting possibility. Prior work has shown that such methods can learn useful skills in robot manipulator setups, and investigating the kind of behaviors the agent acquires for simultaneous locomotion and manipulation is a promising research direction.

## Appendix A Appendix

### A.1 Per step objective loss

$$ $\displaystyle\text{min}\ \mathcal{L}$ $\displaystyle=||(\phi(s_{T})-\phi(s_{0}))-z||^{2}$ (12) $\displaystyle=||\sum_{t=0}^{T}(\phi(s_{t+1})-\phi(s_{t})-\frac{z}{T})||^{2}$ (13) $\displaystyle\leq\sum_{t=0}^{T}||(\phi(s_{t+1})-\phi(s_{t})-\frac{z}{T})||^{2},$ (14) $$

where T is the number of steps in the episode and (14) follows from the triangle inequality. Since the RHS in (14) is an upper bound to the minimization objective, we can minimize the per step loss at every step of the RL algorithm and ensure the true loss is also minimized.

### A.2 Training Setup

For the encoder, we use a simple 3-layer [256,128,64] MLP with ReLU activations, and for the policy, we use a 3-layer [512,256,128] MLP with ELU activations. We train both models for 100k environmental steps in the Isaac Sim simulator using the Orbit framework (Makoviychuk et al., ; Mittal et al., ). For the policy we use Proximal Policy Optimization (PPO) (Schulman et al., ) with hyperparameters reported in Tab. 1. The encoder is trained in a supervised manner, using Smooth L1 loss and the Adam optimizer, as shown in Tab. 2.

### A.3 Hardware setup

To perform zero-shot goal tracking, we require the base position as an observation ot the encoder. On the hardware, we use the manufacturer’s ANYmal robot state estimator which directly provides estimates of the base position in the world frame, relative to the initial position of the robot.

**Table 1: PPO hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| PPO clip ratio | 0.2 |
| Value clip ratio | 0.2 |
| Entropy Loss scale | 0.1 |
| Value loss scale | 1.0 |
| Optimizer | Adam |
| Learning rate | Adaptive |
| KL divergence threshold | 8.e-3 |
| Discount Factor $\gamma$ | 0.99 |
| GAE Lambda | 0.95 |
| Rollouts per update | 24 |
| Learning epochs | 15 |
| Mini-batches | 4 |

**Table 2: Skill discovery hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| Lagrange multiplier | 30. |
| Lagrange multiplier learning rate | 1.e-4 |
| Lagrange multiplier slack | 1.e-6 |
| Optimizer | Adam |
| Encoder learning rate | 5.e-3 |
| Loss | Smooth L1 Loss |
| Loss $\beta$ | 5.0 |
| Rollouts per update | 24 |
| Learning epochs | 15 |
| Mini-batches | 4 |

### A.4 Reward Terms

Table 3 shows the extrinsic rewards we use in our problem (based on those used in (Rudin et al., )).

**Table 3: Extrinsic reward terms and weights. All of the scales are multiplied by the time step size $\Delta_{t}$.**
| Name | Objective | Weight |
| --- | --- | --- |
| Collisions | $\bm{1}_{\textit{body/thigh/shank collision}}$ | 30.$\Delta_{t}$ |
| Feet air time | $(\mathbf{t}_{air}-0.5)$ | 10.$\Delta_{t}$ |
| Joint vel limits | $\begin{cases}-||\dot{\mathbf{q}}-\dot{\mathbf{q}}_{\text{lim}}||&\text{if }<br>\dot{\mathbf{q}}\notin[-\dot{\mathbf{q}}_{lim},\dot{\mathbf{q}}_{lim}]\\<br>0&\text{otherwise}\end{cases}$ | -10.$\Delta_{t}$ |
| Joint acceleration | $-||\ddot{\mathbf{q}}||^{2}$ | -2.5e-7$\Delta_{t}$ |
| Action rate | $-||a_{t}-a_{t-1}||^{2}$ | -5.e-2$\Delta_{t}$ |
| Energy | $-||\dot{\mathbf{q}}\bm{\tau}||$ | -1.e-3$\Delta_{t}$ |
| Nominal joint position | $-||\mathbf{q}-\mathbf{q}_{nom}||^{2}$ | -2.$\Delta_{t}$ |
| Orientation | $-||\mathbf{g}||^{2}$ | -10.$\Delta_{t}$ |
| Base height | $-||p_{z}-0.6||^{2}$ | -5.$\Delta_{t}$ |

### A.5 Additional comparisons with baseline LSD and METRA methods

In Fig. 9 we sample skills uniformly from a 2-D circle and linearly vary the magnitude, resulting in the pattern shown on the left (first column). We compare the latent space encodings (second column) and XY trajectories (third and fourth columns) between the baseline METRA (top), LSD (middle), and Ours (bottom). The baseline methods (LSD and METRA) only optimize for the alignment of latent transitions and skills and always maximize their magnitude. This results in the latent and XY state transitions being maximized for all skills, regardless of the magnitude of the original skill. On the real robot, this in turn translates to very high base linear velocities for every skill. On the other hand, our proposed method preserves the magnitude information in the latent space and enables the learning of behaviors with varying linear velocities, as can be seen from the bottom plots in Fig. 9.

Figure: Figure 9: Comparison between the baseline METRA (top rows), LSD (middle rows), and Ours (bottom rows).
Refer to caption: x7.png

In Fig. 10 we show the evolution of the base position and base linear velocities for ours and one of the baseline methods for a zero-shot goal tracking task. As can be seen, the baseline overshoots the desired target several times and eventually falls over. In comparison, our approach slows down as it approaches the goal, stops and maintains its position.

In Fig. 11 we compare the mean (across all parallel environments) and the maximum traveled distance in an episode throughout training. Due to the additional complexity of predicting the magnitude, our method takes about 15k steps longer to converge. However, as can be seen from the left plot, our method has a lower mean traveled distance while maintaining the same maximum value as the baseline approaches, showing that it has learned a broader range of skills.

Figure: Figure 10: Comparison between base position (in the world frame) and base linear velocity (in the local frame) between the baseline METRA (in red) and Ours (in green) in simulation. The desired final position is shown by the blue line.
Refer to caption: x8.png

Figure: Figure 11: The mean (left) and max (right) traveled distance during an episode, as a function of the number of environmental steps throughout training. Ours (in green) can reach the same maximum distance, while keeping the distribution (and thus the range of skills) much broader.
Refer to caption: x9.png

### A.6 Intrinsic and extrinsic reward ablations

The regularization rewards that we propose are necessary for a successful hardware deployment. However, they can add a degree of structure to the problem, which makes the task of skill discovery easier. To better evaluate the effect of our proposed algorithm (rather than that of the extrinsic regularization rewards), we compare several ablations:

- •
Baseline - Our proposed method.
- •
Variant 1 - Intrinsic rewards only.
- •
Variant 2 - Intrinsic rewards with body-contact termination.
- •
Variant 3 - Intrinsic rewards with body-contact termination and contact avoidance extrinsic rewards.

As can be seen from Fig. 12, Variant 1 which uses only the intrinsic rewards and no termination conditions has the lowest performing behavior. Qualitatively, the behavior consists of the robot falling down and rolling around a bit, resulting in the small maximum traveled distance. Just the addition of a termination condition on body-ground contacts (Variant 2) enables the robot to learn a good locomotive policy. For highly articulated robots like the ANYmal, after falling down it is very challenging to either move or recover to the nominal pose. Without any external bias, if the robot converges to this local optimum it becomes very hard for it to explore enough so that it learns to stand up and then locomote. Finally, by rewarding a lack of hip and shank contacts, as in Variant 3, we can get a much better policy with a similar loss to our proposed Baseline. The main differences between Variant 3 and Baseline is that the former is much less robust and stable, and covers a smaller distance in each episode.
From this ablation it can be seen that while the extrinsic rewards can help give some helpful structure to the problem without constraining too strictly, they are not crucial for discovering locomotive skills (as shown by Variant 2). While Variant 1 did not perform well, we believe that the main reason is due to the ”prone” local minimum, from which the policy cannot easily escape. This agrees with prior work (Park et al., ) where an ”upright base” bonus is used on the Humanoid environment to get good behaviors. METRA (Park et al., ) does not use such a bonus, and the policy converges to a rolling behavior — which is not possible on the ANYmal-C due to its morphology, rather than locomotion.

Figure: Figure 12: Ablation study between Ours (blue), intrinsic rewards only (orange), intrinsic rewards with termination on body contacts (green), and intrinsic rewards with termination on body contacts and extrinsic reward for avoiding hip/shank collisions (red).
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Equation 10, it seems there may be a misuse of notation. Is \( N \) a scalar rather than a function? If so, is this simply a multiplication?
2. How does this approach handle practical environments with obstacles?
3. What is the rationale behind restricting the latent space to a 2D vector? Could you provide some intuition on when this restriction might be beneficial and when it might need to be extended?
4. In Figure 3, the density distribution of the proposed method appears similar to a uniform random distribution. Why is this desirable? Does it indicate greater coverage of the state space?
soundness: 2
strengths: - The figures are well-illustrated and nicely done. I especially like Figures 1 & 4, which show a representation of skills in the 2D space.
- Real robot experiments using an ANYmal quadruped
summary: This paper extends the Lipschitz-constrained Skill Discovery (LSD) framework by Park et al. (2022) to enable the learning of stable and controllable locomotion skills with broader state-space coverage. The proposed approach learns a skill-conditioned policy, where each skill is represented by a 2D continuous vector. This method allows robots to navigate at varying velocities without relying on task-specific rewards, instead using intrinsic mutual information and extrinsic regularization rewards. The authors further demonstrate that the policy can achieve effective zero-shot goal-tracking performance in quadruped locomotion tasks involving the ANYmal robot.
weaknesses: 1. The contributions are unclear. Could the authors clarify how this work improves upon LSD (Park et al., 2022)? Specifically, could you present a side-by-side comparison of the two objectives, highlighting what has been added or removed, and explain the rationale behind these changes?
2. It appears that this approach may not perform well in environments with obstacles. Based on the illustration of the sampled skills, it seems that the quadruped is capable of reaching any 2D location on a plane, but only in the absence of obstacles.
3. The authors overlook many relevant works related to skill discovery, including references [1-7]. Note that these works may use different terminology, such as 'options,' to achieve similar objectives.
4. The term 'skill' is not clearly defined. What is the duration of a skill? Does a skill have a termination condition? What constitutes a skill?
5. While I appreciate the demonstration of results on a real robot, it would also be valuable to see results on simulated benchmarks to better illustrate the efficacy of the proposed approach compared to LSD and Metra.

**References**
1. Durugkar, I., Tec, M., Niekum, S., & Stone, P. (2021). Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34, 8622-8636.
2. Cho, D., Lee, S., & Kim, H. J. (2023). Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.11741.
3. Li, C., Song, D., and Tao, D. Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations, 2022.
4. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181–211, 1999
5. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017
6. Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with a deliberation cost. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
7. Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In AAMAS (1), pp. 299–306. Citeseer, 2008.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: I understand the problem the norm-matching objective is trying to solve. Did existing works try other approaches to solve the same problem and fail? Why is norm-matching interesting / novel? To me, it seemed like the most obvious solution.
soundness: 4
strengths: - Based on the results, the norm-matching objective allows the robot to cover a broader range of states compared to previous approaches.
- It eliminates the need for task-specific rewards by using intrinsic motivation through skill discovery.
- The paper includes successful experiments with an actual robot.
summary: This paper presents a method for constrained skill discovery applied to quadrupedal robot locomotion using unsupervised reinforcement learning. The proposed approach builds upon previous works that use mutual information between skills and states. In contrast to existing works, this paper introduces a norm-matching objective so as to achieve broader state space coverage with stable, controllable movements. This enables a quadruped robot to reach arbitrary points in its environment without requiring task-specific rewards. The proposed method was evaluated on the ANYmal robot.
weaknesses: - The novelty of the paper is questionable. It is obvious from the results that the norm-matching objective is indeed helpful, but this is just a simple regularization applied on top of existing methods. It is also the straightforward / obvious solution to the problem the paper talks about. Thus, it may not attract much interest.
- The paper is sometimes very difficult to follow. To me, this is a major concern that is sufficient for my rejection rating, because I believe the paper needs a major rewrite which is beyond a rebuttal. Starting from the second section, it uses a lot of notation and assumes some prior knowledge about skill discovery methods. This makes the paper inaccessible for people who are not working on this topic. A proper notation definition is missing, e.g., the variable z is used for skills starting from Section II but they are not given a formal definition as a mathematical object in the paper.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - The derivation in Equations 6-7 of the paper contains errors; the coefficients after expanding the squared difference formula are incorrect. What is the purpose of comparing the magnitudes of \( L \) and \( L^{LSD} \)? Compared to \( L \), \( L^{LSD} \) is a simplified result. Does choosing the unsimplified \( L \) as the optimization function make the optimization more complex?
- Figure 7 shows that the ANYmal robot can accurately track velocity commands in the x and y directions. What specific performance criteria were used to evaluate tracking in this experiment? (e.g., The experiment can provide quantitative metrics, such as the mean squared error of position or velocity over time, and compare the performance with controllers that include velocity tracking rewards or controllers from previous methods.)
- In the final 30 seconds of the supplementary video, the test results on the ANYmal robot are demonstrated. However, this section lacks textual and verbal explanations, making it unclear what the paper intends to convey. (e.g., The unannotated parts of the video could include annotations displaying the target position or velocity, or show comparative content.)

[1] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitzconstrained Unsupervised Skill Discovery. October 2021.
[2] Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. October 2023.
soundness: 3
strengths: 1. The paper is clearly written, with Sections 2 and 3 providing a well-articulated motivation and a strong connection to prior work. Both the method and empirical studies are explained with considerable detail.

2. The proposed approach allows for skill control in terms of both magnitude and direction, enabling the robot to navigate the environment at different speeds.

3. The experimental results are impressive, particularly the position-tracking comparisons. Compared to previous methods, this approach appears to achieve more accurate position and velocity tracking without relying on task-specific domain knowledge.
summary: This paper proposes a novel unsupervised reinforcement learning method for skill discovery, which improves upon previous constrained skill discovery methods (LSD[1], METRA[2]) by introducing a norm-matching objective. This norm-matching objective replaces latent transition maximization, achieving the following goals: (1) enabling the agent to learn a broader range of skills and reliably cover a larger portion of the state space; (2) addressing the issue of overly aggressive, high-speed movements resulting from the previous maximization objective, which are challenging to deploy on most robotic hardware. Experiments demonstrate that, compared to baseline methods, this approach allows for more accurate target-tracking in quadruped robots and enables zero-shot deployment on actual quadruped hardware.
weaknesses: - Previous methods [1][2] conducted comparative experiments on multiple benchmark tasks (e.g., Ant, HalfCheetah) and validated their effectiveness across various downstream tasks. This paper, however, only conducts experiments on the ANYmal quadruped robot, focusing primarily on velocity tracking. (If this paper aims to demonstrate the broad applicability of the proposed method, comparison experiments should be conducted on datasets like Ant and HalfCheetah. If the focus is on quadruped robots, it is necessary to compare the method with other controllers designed for quadruped robots.)

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: - Regarding weakness 1: if LSD fails to regularize the norm of the latent space transition, why not simply add it back or normalize it to a unit vector maintaining only directional information? Alternatively, why not consider selecting state space such that it encodes \phi(s_t, s_{t+1}) or \phi(s_{t+1}-s_t) where you process the directional information before encoding?
- It would be helpful to demonstrate the applicability of the proposed method with experiments on more complex settings other than the single simplified euclidian space coverage presented in the paper.
- For a fair comparison with ASE and CASSI, one should select encoder input with the same features as in the proposed method. As the authors pointed out, if joint information is considered, the diversity on the XY plane coverage will deteriorate. And the encoder input should consider states of consecutive steps to match the usage in the proposed method, such as (pos_xy, pos_xy', ...).
soundness: 3
strengths: - The paper is clearly presented, with abundant ablation studies and various baseline methods. The analysis in the experiment part explains well how the introduced constraint can help improve the state space coverage intuitively.
- The experimental setup is thorough, with comparisons against state-of-the-art methods like Lipschitz-constrained skill discovery (LSD) and Metric-aware abstraction (METRA). The paper presents extensive performance evaluations in both simulated and real environments using the ANYmal robot, showing robust results in real-world conditions.
summary: The paper proposes a method for unsupervised reinforcement learning to enable quadruped robots to autonomously learn diverse locomotion skills. The key idea is to maximize the mutual information between the robot's skills and states while constraining the distance covered, replacing the previous latent transition maximization with a norm-matching objective. This allows for richer state space coverage and more controllable and stable locomotive behaviors than prior approaches. The method demonstrates successful deployment on the ANYmal quadruped robot, where the robot can accurately perform zero-shot goal tracking without any task-specific rewards, relying solely on intrinsic skill discovery.
weaknesses: - The proposed method additionally imposes a norm matching constraint on the original LSD formulation to regularize the agent from maximizing the latent space transition. In Eq.9, using L2 norm on the original state space makes sense in the simplified setting where only euclidian distance is considered and XY plan coverage is focused on. It may fail to generalize to more complex state space to diversity skills in to justify the choice of this bound. Think of diversifying in a joint space; it is not reasonable to impose such a bound with the difference between two consecutive joint positions since similar joint states can represent completely different robot gaits.
- Limited experiments on a simplified setting with euclidian space coverage. It would be super helpful and make the proposed method much stronger to show results in style/gait diversification on quadruped, or simulated settings such as maze navigation, where the state space euclidian distance does not reflect the adjacency in the latent space.

Meta-review: The AC read the paper, reviewers and discussions. As suggested by the general opinions from reviewers, the quality of this paper is below the acceptance threshold.
The paper investigates unsupervised RL for constrained skill discovery, with applications to quadrupedal robot locomotion control. It incrementally improves results based on previous works e.g. Lipschitz-constrained Skill Discovery (LSD). It takes advantage of mutual information between skills and states. A norm-matching objective is proposed to achieve broader state space.
As mentioned by reviewers: previous methods such as LSD conducted comparative experiments on multiple tasks in simulation. While this paper only report experiments on velocity tracking on the ANYmal quadruped robot, falling in the gulf between algorithmic and experimental papers.

Decision: Reject