Language Guided Skill Discovery

## Abstract

Abstract Skill discovery methods enable agents to learn diverse emergent behaviors without explicit rewards. To make learned skills useful for unknown downstream tasks, obtaining a semantically diverse repertoire of skills is essential.
While some approaches introduce a discriminator to distinguish skills and others aim to increase state coverage, no existing work directly addresses the “semantic diversity” of skills. We hypothesize that leveraging the semantic knowledge of large language models (LLMs) can lead us to improve semantic diversity of resulting behaviors.
In this sense, we introduce Language Guided Skill Discovery (LGSD), a skill discovery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. The prompts serve as a means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace.
We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in discovering more diverse skills compared to five existing skill discovery methods in robot-arm manipulation environments. Lastly, LGSD provides a simple way of utilizing learned skills via natural language.

## 1 Introduction

One of the key capabilities of intelligent agents is to autonomously learn useful skills applicable to downstream tasks without task-specific objectives.
Consider how human infants acquire manipulation skills through simple play with toys, which can later be applied to a broader set of tasks, such as using a spoon or holding a bottle. Our research aims to emulate this process of skill learning. Prior works in unsupervised skill discovery suggest that maximizing diversity of behaviors can be a way to develop such skills [Gregor et al., , Eysenbach et al., , Kwon, ]. These approaches operate under the assumption that acquiring a wide range of diverse behaviors may lead to the development of useful skills for downstream tasks. Building on this premise, studies like Sharma et al. [], Hansen et al. [], Liu and Abbeel [], Laskin et al. [] have associated behaviors with randomly sampled vectors by maximizing Mutual Information between them to acquire diverse skills.
Additionally, there are other approaches aimed at increasing coverage of the state space [Liu and Abbeel, , Yarats et al., , Park et al., ],
or exploration-based strategies that leverage errors in predictive models to enhance learning [Burda et al., , Pathak et al., , Park et al., ].

However, we contend that these measures are proxies for what we term “semantic diversity”, and existing proxies objects do not necessarily reflect the semantic diversity of a repertoire of skills. For instance, in high degrees of freedom (DOF) systems, such as robot-arm manipulation tasks, unstructured movements of each DOF might exhibit high diversity in terms of state coverage or distinctiveness from a neural discriminator, but nonetheless lack meaningful semantic diversity. To more accurately measure semantic diversity, we propose the use of Large Language Models (LLMs) as they are well-suited for understanding and assessing the semantic significance of different behaviors in a way that better aligns with human judgment.

Figure: Figure 1: We proposed LGSD which can discover a semantically distinctive set of skills. We display four sample skills acquired from a single training run. Red dots indicate the path of the end-effector, while blue dots represent the trajectory of the object moved. Two skills moved the green cube to the right, while others moved the red cube to the top.
Refer to caption: extracted/5648683/img/two_cube_skills.png

In this work, we present Language Guided Skill Discovery (LGSD), a skill discovery method that utilizes the guidance of LLMs to learn semantically diverse skills. LGSD aims to fulfill three desiderata for skill discovery. (i) Firstly, we want to discover skills that have diverse semantic meanings. We use LLMs to generate descriptions for each of agent states. Based on these descriptions, we then measure the semantic difference between states using language-distance, and train skills to maximize this distance. Since LLMs are equipped with a semantic understanding of the world, this provides a way to maximizing semantic differences between skills. Fig. 1 illustrates skills learned from LGSD, showcasing the semantic diversity of resultant behaviors. (ii) Secondly, we seek to restrict the search space for skills into a desired semantic subspace.
Intuitively, restricting the skill space is akin to enforcing an infant to grab toys only with their hands, and not with their mouth or feet. We implement this desideratum by utilizing language prompts. Humans provide prompts to LLMs, and the generated descriptions are then constrained by these prompts. This will allow our method to ignore differences outside the intended semantic space.
(iii) Lastly, we aim for the learned skills to be easy to use. When the skill space is continuous, merely selecting the best skills for target tasks becomes non-trivial. LGSD supports a zero-shot language instruction-following capability by training separate network $\psi$ for skill inference. Humans can provide a natural language description of the desired state, and then LGSD can infer which skill should be used to reach that goal state.

The primary contributions of this work are as follows. 1) We propose a skill discovery framework that enables the discovery of semantically diverse skills by utilizing language guidance from LLMs. To the best of our knowledge, this is the first work to incorporate semantic diversity into skill discovery. 2) We present a method of constraining the skill search space to a user-defined semantic space using language prompts. This allows users to control the resulting skills by simply specifying different prompts. We emphasize that, prior to training, existing methods only offer limited control over the skills to be learned.
3) We provide a theoretical proof of how language-distance, a proxy for semantic distance, can be employed as a valid pseudometric. 4) We propose a method that enables an agent to reach a goal state specified by natural language, facilitating the convenient use of learned skills. 5) We demonstrate that our method can train semantically diverse skills. Through extensive experiments, we show that LGSD outperforms five different skill discovery baseline methods on both locomotion and manipulation tasks, in terms of both diversity and sample efficiency.

## 2 Related work

### 2.1 Unsupervised skill discovery

#### Mutual information based approach

Common approaches to associate skills with corresponding behaviors are based on maximizing the mutual information (MI) between states $S$ and skills $Z$, or $\mathcal{I}(S;Z)$ [Gregor et al., , Eysenbach et al., , Sharma et al., , Hansen et al., , Liu and Abbeel, , Laskin et al., ]. MI can be viewed as a dependency measure that quantifies the association between two random variables. To maximize this quantity, a popular early formulation utilizes the decomposition $\mathcal{I}(S;Z)=\mathcal{H}(Z)-\mathcal{H}(Z|S)$, where $\mathcal{H}[\cdot]$ refers to Shannon entropy. Since directly computing $\mathcal{H}(Z|S)$ is intractable, Eysenbach et al. [] introduce a variational posterior $q_{\phi}(z|s)$ to optimize the lower bound. Similarly, CIC [Laskin et al., ] maximizes $\mathcal{I}((s,s^{\prime});Z)$, promoting distinct state transitions across different skills. However, a significant limitation of using MI as the objective arises from the use of Kullback–Leibler (KL) divergence to define MI:

$$ $\mathcal{I}(S;Z)\stackrel{{\scriptstyle\text{def}}}{{=}}D_{KL}(P(S,Z)||P(S)P(Z )).$ $$

Maximization of MI using KL divergence often results in skills with less distinctive behaviors. This is because KL divergence is fully maximized when two densities have no overlap, posing a problem as there is no further incentive to distinguish the densities beyond this point. This is problematic because having no overlap between two densities does not necessarily mean two skills are noticeably distinctive. In practice, this becomes a severe shortcoming of DIAYN [Eysenbach et al., ], where its objective can be fully optimized as long as the discriminator $q_{\phi}$ can distinguish them perfectly. Neural networks can easily distinguish minor numerical differences, so the skills learned with this objective often have only these minute differences.

#### Distance maximization approach

To overcome these shortcomings, recent skill discovery works [He et al., , Park et al., , , ] propose using the Wasserstein Dependency Measure (WDM,  Ozair et al. []):

$$ $\mathcal{I_{W}}(S;Z)\stackrel{{\scriptstyle\text{def}}}{{=}}\mathcal{W}(P(S,Z) ||P(S)P(Z))$ $$

WDM is defined using the Earth-Mover (EM) distance and retains the advantage of always providing an incentive to maximize the distance between distributions, even when they are already distinguishable.

An essential characteristic of $\mathcal{I_{W}}$ is that it must be defined within a metric space.
Therefore, the choice of metric to measure the difference between skills comes to the gist of the algorithm since it ultimately governs the resulting behavior of each skills. LSD [Park et al., ] employs the Euclidean distance in state space as its metric. Maximizing $\mathcal{I_{W}}$ under this metric encourages agents to visit states that are are far apart in terms of Euclidean distances.
Conversely, CSD [Park et al., ] points out the problem of using Euclidean distance as the metric. They argue that maximizing Euclidean distance tends to focus on "easy" changes in the state space, such as moving the robot arm itself rather than the target object.
Thus, CSD proposed to maximize "controllability-aware distance" instead. This metric measures the negative log likelihood of a transition dynamics $-\log{p(s^{\prime}|s)}$. Using this metric means the algorithm favors visiting rare transitions. LGSD adopts a similar approach but differentiates itself by employing a language-based distance metric within the WDM framework, to maximize the semantic difference.

### 2.2 Using language as guidance for behavior learning

Recent work has studied using semantic knowledge in large pre-trained models for control tasks in several ways. A popular approach is to use the common-sense reasoning capabilities of LLMs to produce high-level plans over known, existing skills through the interface of natural language [Sharma et al., , Ahn et al., , Singh et al., , Huang et al., , Ha et al., ] or generated programs [Liang et al., , Zeng et al., , Vemprala et al., ]. In addition, LLMs have also been shown to be effective in directly producing low-level actions that can be executed by an agent [Wang et al., , Kwon et al., , Driess et al., , Brohan et al., ]. Our work does not use LLMs to generate *actions* at any level of abstraction but to provide a meaningful distance measure to guide learning towards discovering semantically meaningful skills.

Several works have used LLMs to guide agents to learn semantically meaningful behavior by using them to design reward functions for either a given desired task [Yu et al., , Ma et al., ]. This showcases LLMs’ capacity to understand specific physical embodiments assorting it with semantic knowledge to reason about given tasks. We note that these works aim to solve a single specific task, whereas our method aims to learn a repertoire of diverse behaviors without explicit supervision.

The work most related to ours is Du et al. [], which employs an LLM to guide exploration. Their approach uses the LLM to suggest plausible set of goals for the agent, maximizing the cosine similarity between the language embeddings of the goals and the current state. While they focus on using an LLM to reason with a low-dimensional, discrete set of high-level actions in complex environments like Crafter and Housekeep, our work focus on discovering meaningful behaviors in complex, continuous control spaces where actions (e.g. joint targets) are not as readily interpretable. Furthermore, their reward function, based on language similarity, lacks theoretical support as it is not a valid metric. In contrast, our method effectively handles the language-distance measure and provides a theoretical framework for its use. We elaborate on this in Section 4.3.

## 3 Preliminaries

Unsupervised skill discovery can be formalized within the framework of a reward-free Markov Decision Process (MDP), denoted as $\mathcal{M}\equiv\{\mathcal{S},\mathcal{A},\mathcal{P}\}$. Here, $\mathcal{S}$ represents the state space, $\mathcal{A}$ denotes the action space, and $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,\infty)$ defines the transition dynamics. During training, a latent vector $z$ is sampled from a prior distribution $z\sim p(z)$ at the start of each episode and remains fixed throughout the episode. This vector informs the policy $\pi(a|s,z)$, guiding the generation of corresponding rollouts through $\pi$. The training procedure aims to foster a dependency of the latent vector $z$ on its resultant behavior, allowing it to function effectively as a skill. It is referred to as the skill because being conditioned on different skill $z$ aims to behave differently by encouraging agents to visit distinct distributions of states $s$, transitions $(s,s^{\prime})$, or trajectories $\tau$. A common approach to training skills involves defining intrinsic rewards $r^{\text{int}}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ and optimizing the accumulation of discounted intrinsic rewards with the following objective: $\mathcal{J}=\mathbb{E_{\pi,\mathcal{P}}}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}r^
{\text{int}}_{t}\Big{]}$. Various off-the-shelf reinforcement learning (RL) algorithms can be employed to optimize $\pi$ to maximize this objective. Notably, this process does not utilize any task-related explicit rewards.

## 4 Language Guided Skill Discovery

Figure: Figure 2: Overview of how LGSD works. Given a prompt, the LLM generates the description for each state. We then measure the difference between these descriptions and denote it as $d_{\text{lang}}$. Based on $d_{\text{lang}}$, we constrain the latent space by enforcing the 1-Lipschitz condition on $\phi$. Then the agent is encouraged to visit states that make the vector $\phi(s^{\prime})-\phi(s)$ aligns well with a randomly sampled vector $z$ from an isotropic Gaussian prior. This makes the agent explore the latent space in diverse directions depending on the sampled $z$.
Refer to caption: extracted/5648683/img/method_overall.png

In this section, we begin by providing an overview of our algorithm, followed by in-depth explanations of the prompting, language distance measurement, and how we can utilize the acquired skills using language prompts.

### 4.1 Algorithm

Our goal is for an agent to learn a corpus of skills that visits semantically different states. Discovering semantically distinct states requires 1) a way of measuring the semantic difference between states, and 2) a means of maximizing this difference measure.
To evaluate the semantic differences between different behaviors, we leverage the power of LLMs. We will consider two states as substantially distinct if the language descriptions produced by LLMs are semantically different. As illustrated in Fig. 2, an LLM is queried to produce a natural language description for each state. This description links the state to its semantic meaning. We then measure the difference between these language descriptions. To measure the difference between two language descriptions, we leverage a pre-trained natural language embedding model, Sentence-Transformer [Reimers and Gurevych, ], to map the language descriptions into fixed-dimension real-valued vectors. This allows us to measure the difference between these vectors using cosine similarity. We refer to this difference measure as language-distance and denote it as $d_{\text{lang}}$.

Given a measure of the difference between language descriptions $d_{\text{lang}}$, which acts as a proxy for the semantic differences between states, the next step is to incentivize an agent to maximize this distance in order to discover diverse skills. We utilize existing distance maximization skill discovery algorithms, discussed in Section 2.1. Intuitively, we aim for agents to spread out from the origin in the latent space, which is constrained under $d_{\text{lang}}$. This constraint ensures that spreading out in the latent space leads to maximizing sum of $d_{\text{lang}}$ along the path in the state space. More specifically, we train a representation function $\phi:\mathcal{S}\to Z$ that maps state space $\mathcal{S}$ onto a latent space $Z$. Here, we enforce that $\phi$ is 1-Lipschitz continuous under $d_{\text{lang}}$, meaning that $\forall(s,s^{\prime})\in\mathcal{S},\ \|\phi(s^{\prime})-\phi(s)\|\leq d_{
\text{lang}}(s,s^{\prime})$. Thus, the Euclidean distance between $s$ and $s^{\prime}$ in latent space is always less than or equal to $d_{\text{lang}}(s,s^{\prime})$. Hence, simply maximizing the Euclidean distance in the latent space results in maximizing $d_{\text{lang}}$ in the state space.

Spreading out in the latent space can be achieved by aligning the vector $\phi(s^{\prime})-\phi(s)$ with a latent skill vector $z$ sampled from a Gaussian distribution $z\sim\mathcal{N}(0,I)$.
Because the vector $z$ is sampled isotropically, aligning $\phi(s^{\prime})-\phi(s)$ with $z$ fosters a diversity of behaviors depending on the sampled $z$. We use the reward $r(s,z,s^{\prime})=(\phi(s^{\prime})-\phi(s))^{T}z$ and maximize the cumulative reward to achieve this objective. Here, Park et al. [] showed that maximizing the cumulative reward $r(s,z,s^{\prime})$ with a 1-Lipschitz continuous $\phi$ is equivalent to maximizing the WDM between the terminal state $S_{T}$ and $Z$:

$$ $\displaystyle\mathop{\rm argmax}_{\pi,\phi}\ {\mathcal{I_{W}}(S_{T};Z)}$ $\displaystyle\approx\mathop{\rm argmax}_{\pi,\phi}{\mathbb{E}_{\pi,z\sim p(Z)} \left[\sum_{t=0}^{T-1}(\phi(s_{t+1})-\phi(s_{t}))^{T}z\right]}$ $\displaystyle\ \ \ \ \ \ \ \ s.t.\ \forall(s,s^{\prime})\in\mathcal{S},\ \| \phi(s^{\prime})-\phi(s)\|\leq d_{\text{lang}}(s,s^{\prime})$ (1) $$

We provide the proof in Appendix A.
Therefore, the meaning of maximizing traveled language-distance is equivalent to maximizing the WDM between $s_{T}$ and $z$ under language-distance. Any off-the-shelf model-free RL algorithms can then be used to maximize the sum of rewards. We used PPO [Schulman et al., ] for as our primary RL algorithm because of its stable convergence. To ensure $\phi$ is 1-Lipschitz continuous under a distance metric $d$, we use dual gradient descent with a with a Lagrange multiplier $\lambda$ [Boyd and Vandenberghe, ].

### 4.2 Constraining the skill space via prompts

Figure: Figure 3: By prompting the LLM to generate different type of descriptions depending on the state, LGSD can adapt its focus during training.
Refer to caption: extracted/5648683/img/cube_eef_distance.png

The potential space for skills is vast, yet only a narrow region possesses semantic meanings. We want to explore only within a subspace which has inherent semantic meanings. To achieve this desiderata, we use language prompts to constrain the entire skill space into the desired semantic subspace.

Fig. 3 shows a manipulator environment that aims to move an object. In this environment, we can consider skills enabling object interactions as semantically meaningful while considering unstructured robot-arm movements not.
Our language-based distance allows us to capture such delicate semantics of skills.
For instance, we prompted an LLM to describe the distance $d_{1}$ between the robot arm and the object unless the object is moved already.
Therefore, the scene description remains the same unless there is a change in the robot-object distance $d_{1}$.
Because $d_{\text{lang}}$ becomes positive only when the description of a state is changed, our distance maximization algorithm encourages an agent to visit states that can change descriptions, which makes a robot arm to approach the object.

Moreover, we highlight that we can induce the LLM to not only focus on a single semantic aspect of the scene but also to adapt its focus onto different semantic aspects throughout the training.
For instance, when the robot arm finally reaches the cube, the LLM is then asked to describe the object’s moved distance $d_{2}$. This additional prompt shifts the focus of the semantics to the object’s traveled distance and any other changes in the states are ignored by the LLM. This leads to the robot arm pushing the object to its maximum extent. Note that existing skill discovery algorithms have limited capability to constrain its focus by feeding subset of observations [Eysenbach et al., , Park et al., ]. In contrast, LGSD enables not only focusing on semantic dimensions but also adapting its focus across different semantics during training.

### 4.3 Language distance as a valid metric

In Section 4.1, we stressed that $d_{\text{lang}}$ establishes a criterion for determining the distance between any two points in the latent space. Therefore, maintaining a coherent criterion is crucial for building the latent space. In this sense, $d_{\text{lang}}$ should be a valid distance metric. In this section, we present how $d_{\text{lang}}$ can be employed as a valid (pseudo-) metric, despite not satisfying the triangle inequality.

We begin with defining $d_{\text{lang}}$ formally. Let us denote a user prompt as $l_{\text{prompt}}$, and a rule-based annotator as $g:\mathcal{S}\to l_{\text{state}}$. The annotator $g$ is a pre-defined function that translates state vector into corresponding natural language sentence without any addition or loss of information. Then, we denote the output description of an LLM for state $s$ as $l_{\text{desc}}(s)=LLM(\cdot|l_{\text{prompt}},l_{\text{state}})$. We configure the LLM to be deterministic with zero temperature during generation, thus treating it as a function. Next, we employ a pre-trained language embedding model $f_{\text{embd}}:l_{\text{desc}}\to\mathbb{R}^{N}$. Using this setup, we can now define the language distance $d_{\text{lang}}$ as follows:

$$ $\forall(s,s^{\prime})\in\mathcal{S},\ d_{\text{lang}}(s,s^{\prime})\stackrel{{ \scriptstyle\text{def}}}{{=}}1-\frac{f_{\text{embd}}(l_{\text{desc}}(s))^{T} \cdot f_{\text{embd}}(l_{\text{desc}}(s^{\prime}))}{\lVert f_{\text{embd}}(l_{ \text{desc}}(s))\rVert\lVert f_{\text{embd}}(l_{\text{desc}}(s^{\prime})) \rVert}.$ (2) $$

However, to be considered as a pseudometric, a function $d:\mathcal{S}\times\mathcal{S}\to\mathbb{R}$ must satisfy three conditions: $\forall x,y,z\in\mathcal{S}$,
(i) $d(x,x)=0$, (ii) $d(x,y)=d(y,x)$, and (iii) $d(x,z)\leq d(x,y)+d(y,z)$. In our case, $d_{\text{lang}}$ satisfies conditions (i) and (ii), but does not meet the condition (iii), the triangle inequality.

To overcome this problem, we propose to impose the constraint in Eq. 1, only for adjacent pairs of states $\forall(s,s^{\prime})\in\mathcal{S}_{\text{adj}}$, where two states $s$ and $s^{\prime}$ are considered adjacent if $s^{\prime}$ can be reached from $s$ within a single state transition.
Therefore, every encountered transition sample of $(s,s^{\prime})$ during training is adjacent. We claim that this modification to Eq. 1 will implicitly induce a valid pseudometric $\tilde{d}$ for all state pairs $\forall(s,s^{\prime})\in\mathcal{S}$.

Claim 1. For any function $d_{\text{lang}}:\mathcal{S}\times\mathcal{S}\to\mathbb{R}_{+}$, there exists a valid pseudometric $\tilde{d}:\mathcal{S}\times\mathcal{S}\to\mathbb{R}_{+}$ that imposing the Eq. 1 on adjacent pairs of states lead to imposing the same equation on every possible state pairs under $\tilde{d}$, i.e.,

$$ $\forall x,y\in S_{adj},\ \|\phi(x)-\phi(y)\|\leq d_{\text{lang}}(x,y)\implies \forall x,y\in S,\ \|\phi(x)-\phi(y)\|\leq\tilde{d}(x,y).$ $$

We provide a proof of the claim in Appendix B. Applying this modification, we came to the final objective function as follows:

$$ $\mathcal{J_{\text{LGSD}}}=\mathbb{E}_{\pi,z\sim p(Z)}\left[\sum_{t=0}^{T-1}( \phi(s_{t+1})-\phi(s_{t}))^{T}z\right],\ \ s.t.\ \forall(s,s^{\prime})\in \mathcal{S_{\text{adj}}},\ \|\phi(s^{\prime})-\phi(s)\|\leq d_{\text{lang}}(s, s^{\prime}).$ $$

Consequently, this objective encourages agents to explore diverse and distant states in latent space, resulting in a greater language-distance traveled along the trajectory within the state space.

### 4.4 Utilizing learned skills using natural language

After training is complete, we need a way of selecting specific $z$ values to utilize the behaviors associated with each skill to solve concrete tasks. However, we lack knowledge about which $z$ leads to the desired behavior. We can sweep over the skill space at fixed intervals [Laskin et al., ], or sample a number of skills and evaluate each with hundreds of rollout trajectories [Park et al., ]. To overcome these challenges,
Park et al. [, ] have suggested using $z=(\phi(g)-\phi(s))/|\phi(g)-\phi(s)|_{2}$ for continuous skills, when we have Lipschitz continuous $\phi$. Yet, in many cases, we cannot fully specify the desired goal state. For instance, consider an agent using a robotic arm to move a cube. The state includes the end-effector’s rotation and position, as well as the cube’s coordinates. While you can specify the cube’s position, the required rotation and position of the end-effector to achieve that position may not be known.

To address this challenge of goal state specification, we propose training a separate network $\psi:f_{\text{embd}}(l_{\text{desc}})\to Z$ to infer the skill. Intuitively, $\psi$ is a neural network designed to infer which $z$ was used to reach the state that yields the given description. If we train $\psi$ alongside the goal-conditioned policy $\pi$, then we can use $\psi$ to select $z$, which is then fed into $\pi$. In this way, the agent can reach the desired state as described in natural language in a zero-shot manner. The full algorithm, including the training of $\psi$, is presented in Appendix D.

## 5 Experiments

In this section, we evaluate our proposed LGSD by conducting a series of experiments on continuous control environments, encompassing both locomotion and manipulation setups. We aim to answer three questions:
(1) Can prompting constrain the skill space into a desired semantic subspace?
(2) Can language guidance lead to obtaining more diverse skills compared to unsupervised skill discovery baselines?
(3) Can we employ learned skills using natural language?

#### Experimental setup

We trained our algorithm and baselines using Isaac Gym [Makoviychuk et al., ], a high-throughput GPU-based physics simulator. For the language model, we employed gpt-4-turbo-2024-04-09[Achiam et al., ]. We set the temperature parameter of the language model to 0 to get a consistent, low-variance measure of $d_{\text{lang}}$. To reduce the number of unique queries, we discretized and cached the input and output of these queries and reused them during training. We provide the exact prompts used for each experiments in Appendix F.

### 5.1 Constraining the skill space into a desired semantic subspace

We first explored how LGSD constrains the skill space into a desired semantic subspace using prompts. To visually present this concept, we designed experiments where agents visit mutually exclusive states depending on the prompts.
We tested this idea in two environments, Ant and FrankaCube. In the Ant environment, we required the agent to traverse only half of the plane. For instance, we allowed an agent to visit any location in the northern part of the plane but restricted it from entering the southern area. We repeated these experiments for each cardinal direction: North, South, East, and West (NSEW). Similarly, in the FrankaCube environment, we intended for the robot arm to move the object to only half of the plane for each of the NSEW directions starting from the origin. We trained four agents, each with its own unique prompt, encouraging the agent to explore only within the intended semantic subspace.

Figure: Figure 4: Trajectories of different skills trained with different prompts. For the Ant (top row), we recorded the base’s $x,y$ coordinates. For the Franka robot-arm agent (bottom row), we recorded the $x,y$ coordinates of the object on the table.
Refer to caption: extracted/5648683/img/ant_cube_skills4.png

Fig. 4 (and Fig. 9 in Appendix) shows the discovered skills trained with each prompt. For each plot, we sampled $z\sim\mathcal{N}(0,\mathbb{I})$ 150 times independently and then generated trajectories using a skill-conditioned policy. As shown in Fig. 4, LGSD effectively constrained the skill space into the desired subspace, especially in the Ant environment. For the Franka agent, the robot arm was able to move the object into the intended region, but it could not fully cover all skills within the constrained subspace. We believe this limitation is due to the inherent nature of the exploration problem.
In the Ant environment, the dynamics of the agent is invariant to horizontal translation, which allows effective control at any location. On the other hand, moving the cube along a long trajectory often requires changes in pushing strategies with different contacts and force profiles.

### 5.2 Language guidance aids discovering diverse skills

Figure: Figure 5: Initial state of robot arm.
Refer to caption: extracted/5648683/img/franka_initial_far.png

To observe the effect of language guidance, we evaluate the diversity measure of the skills learned using LGSD against unsupervised skill discovery baselines. For the baselines, we take two classical mutual information-based skill discovery algorithms: DIAYN [Eysenbach et al., ] and DADS [Sharma et al., ], as well as three state-of-the-art distance maximization based approaches: LSD [Park et al., ], CSD [Park et al., ], and METRA [Park et al., ].
Although METRA was originally designed for pixel-based input tasks, it can also handle vector-based inputs. Therefore, we used vector-based state inputs for all methods to facilitate direct comparison. For all methods, we utilized no human prior knowledge in selecting parts of the state dimension to induce desired behaviors; instead, we passed full observations directly to the skill discovery algorithms. Detailed information about the observations is provided in Appendix E.1.

Figure: (a) Object moved distance
Refer to caption: extracted/5648683/img/result_cube_avg_dist.png

Figure: Figure 7: Paths of objects moved on a horizontal table by LGSD and five baseline algorithms, illustrated using the best-performing checkpoint from four runs for each algorithm.
Refer to caption: extracted/5648683/img/baseline_trajectories.png

We set up a challenging manipulation task as illustrated in Fig. 5. Here, we expect to discover skills that can manipulate the object toward diverse directions. The challenge arises from the initial distance between the robot arm and the object, which is set to the maximum possible value. We believe this task can confirm the validity of the two-stage prompt suggested in section 4.2. After training, we randomly sampled $200$ skills and measured both the average moved distance of the object and the state coverage that counts the total number of visited horizontal grids, each sized 1cm x 1cm.

We present the results in Fig. 6 with two diversity criteria. We trained LGSD and all baseline methods with four different seeds, reporting the mean and standard deviation of the results. Fig. 6 suggests that LGSD discovers more diverse states compared to all baseline algorithms, confirming that language guidance from an LLM helps in discovering both diverse and meaningful skills. Notably, all distance-maximization methods (CSD, METRA, LSD) could manipulate the object while MI-based methods could not. We also present the qualitative results of the learned behaviors by each method in Fig. 7, where LGSD was able to manipulate objects to cover more grids. Additionally, we provide visualizations and an analysis of the agent’s movements in the latent space in Appendix C.

### 5.3 Utilizing learned skills via language

Figure: Figure 8: Users can specify goals in natural language. Blue dots represent the desired position of the object, while red dots indicate the object’s trajectory.
Refer to caption: extracted/5648683/img/goal_following_franka4.png

Lastly, we demonstrate how we can design an agent that can execute learned skills to follow a natural language instruction. As discussed in Section 4.4, our idea is to utilize $\psi(z|f_{\text{embed}}(l_{\text{desc}}))$ to infer a skill to reach the goal state.
For instance, our instruction can be a description of desired state,"The object is located at [0.3,0.2]", which does not contain information regarding robot-arm’s position or orientation.
Nevertheless, the learned $\psi$ allows an agent to translate the given language instruction into a skill to reach the goal state.

Fig. 8 illustrates scenarios that a robot-arm moves the object to the six different goal positions based on language description. This experiment showcased that the agent can understand the desired goal state with the help of $\psi$, execute the skill accurately, and successfully move the cube to the target positions in a zero-shot manner. We also provide the results of the zero-shot goal-following agent in the locomotion domain in Appendix, Fig. 10.

## 6 Conclusion

Our work introduced Language Guided Skill Discovery (LGSD), a novel framework for skill discovery that leverages the semantic understanding capabilities of large language models (LLMs) to guide the learning of semantically diverse skills. By incorporating language as a tool for both constraining the skill space and measuring semantic diversity, LGSD offers a novel approach to learning diverse skills. We have demonstrated through various experiments that LGSD not only constrains skills within semantically meaningful subspaces but also enhances the diversity and applicability of the skills learned.

Despite these advancements, there are areas for further exploration and improvement. The approach could benefit from a more nuanced understanding of the scene, potentially by incorporating vision-language models (VLMs). We conjecture that some semantics can be more easily captured through vision. VLMs would effectively help to utilize these dimensions. Furthermore, we suggest that incorporating trajectory-level semantic differences, instead of state-level, could be an interesting future direction. We could provide entire trajectories to LLMs/VLMs and query them to associate them with semantic meanings.

Overall, we believe our work marks the beginning of a series of efforts that capture semantic diversity between skills. We hope this work facilitates further research endeavors in discovering semantically meaningful skills with the aid of external knowledge sources, including LLMs.

## Appendix A Proof of maximizing Wasserstein Dependency Measure

We begin with the alternative form of Wasserstein distance using Kantorovich-Rubenstein duality following Villani et al. [], to make $\mathcal{I_{W}}$ tractable.

$$ $\mathcal{I_{W}}(S;Z)=\sup_{f\in\mathcal{L}_{S\times Z}}\mathbb{E}_{p(S,Z)}[f(S ,Z)]-\mathbb{E}_{p(S)p(Z)}[f(S,Z)]$ $$

where $\mathcal{L}_{S\times Z}$ is a set of all 1-Lipschitz function defined in ${S\times Z}\to\mathbb{R}$ under the distance metric $d$. For the remainder, we follow the proof outlined in Park et al. [], but we provide a simplified version.

We first parameterize $f(S,Z)=\phi(S)^{T}\psi(Z)$ using $\phi:\mathcal{S}\to\mathbb{R}^{D}$ and $\psi:\mathcal{Z}\to\mathbb{R}^{D}$, which are 1-Lipschitz constrained functions. This is safe decomposition because the expressive power of $\phi(S)^{T}\psi(Z)$ is universal. Please refer to Park et al. [] for more detail.

$$ $\mathcal{I_{W}}(S;Z)\approx\sup_{\|\phi\|_{L}\leq 1,\|\psi\|_{L}\leq 1}\mathbb {E}_{p(S,Z)}[\phi(S)^{T}\psi(Z)]-\mathbb{E}_{p(S)}[\phi(S)]\mathbb{E}_{p(Z)}[ \psi(Z)]$ $$

Then we use $Z$ instead of $\psi(Z)$ to earn simplicity at the cost of expressiveness. Also we choose the prior distribution $p(Z)$ to have zero-mean. This makes the second term $\mathbb{E}_{p(S)}[\phi(S)]\mathbb{E}_{p(Z)}[Z]$ to 0. Finally, similar to VIC, we measure WDM between the final state $S_{T}$ and the skill, insteaf of all states.

$$ $\mathcal{I_{W}}(S_{T};Z)\approx\sup_{\|\phi\|_{L}\leq 1}\mathbb{E}_{p(S,Z)}[ \phi(S_{T})^{T}Z]$ $$

Next, we utilize $\mathbb{E}_{p(S,Z)}[\phi(S_{0})^{T}Z]=\mathbb{E}_{p(S)}[\phi(S_{0})]\mathbb{E}
_{p(Z)}[Z]=0$ because the distribution of the initial state $S_{0}$ and $p(Z)$ are mutually independent, and $p(Z)$ has a zero-mean. So subtracting it from the right-hand side doesn’t change the equation. Now we have the final objective:

$$ $\displaystyle\mathcal{I_{W}}(S_{T};Z)$ $\displaystyle\approx\sup_{\|\phi\|_{L}\leq 1}\mathbb{E}_{p(S,Z)}[\phi(S_{T})^{ T}Z]-\mathbb{E}_{p(S,Z)}[\phi(S_{0})^{T}Z]$ $\displaystyle\approx\sup_{\|\phi\|_{L}\leq 1}\mathbb{E}_{p(\tau,z)}\left[\sum_ {t=0}^{T-1}(\phi(S_{t+1})-\phi(S_{t}))^{T}Z\right].$ $$

Therefore, we can use RL with the reward function $r(s,z,s^{\prime})=(\phi(s^{\prime})-\phi(s))^{T}z$ to maximize $\mathcal{I_{W}}(S_{T};Z)$.

## Appendix B Proof of Claim 1

Our goal is to prove the following two claims.

- (i)
$\forall x,y\in S_{adj},\ \|\phi(x)-\phi(y)\|\leq d_{\text{lang}}(x,y)\implies
\forall x,y\in S,\ \|\phi(x)-\phi(y)\|\leq\tilde{d}(x,y)$
- (ii)
$\tilde{d}(x,y)$ is a valid pseudometric.

We follow the proof of CSD [Park et al., ] similarly, but it has significant differences which will be covered in later of this section.

#### Proof of (i)

We begin with introducing $P(x,y)$ as the set of all finite path from state $x$ to state $y$, for $\forall x,y\in\mathcal{S}$. Here, we only consider existing path $p=(s_{0}=x,s_{1},...,s_{T-1},s_{T}=y)\in P(x,y)$. The term existing indicates that for every consecutive state pairs $(s_{i},s_{i+1})$ in $p$, $s_{i+1}$ should be reachable from $s_{i}$. More concretely,

$$ $\forall(s_{i},s_{i+1})\in p,\ \exists a\in\mathcal{A},s.t.,\mathcal{P}(s_{i},a ,s_{i+1})>0$ $$

where, $\mathcal{P}$ is a transition probability. Then, for a path $p$, we denote the sum of the language distance along the path $p$ as $D_{\text{lang}}(p)$, i.e., $D_{\text{lang}}(p)\stackrel{{\scriptstyle\text{def}}}{{=}}\sum_{t=0}^{T-1}d_{
\text{lang}}(s_{t},s_{t+1})$. Now we can define $\tilde{d}(x,y)$ as follows:

$$ $\tilde{d}(x,y)\stackrel{{\scriptstyle\text{def}}}{{=}}\left\{\begin{array}[]{ ll}\inf_{p\in P(x,y)}D_{\text{lang}}(p)&\text{if }x\neq y\\ 0&\text{if }x=y.\end{array}\right.$ $$

Now we are ready to move on to the main proof. We assume that $\forall x,y\in S_{adj},\ \|\phi(x)-\phi(y)\|\leq d_{\text{lang}}(x,y)$. Also, note that for all $x,y\in\mathcal{S}$, both $d_{\text{lang}}(x,x)=0$ and $d_{\text{lang}}(x,y)=d_{\text{lang}}(y,x)$ holds. We denote the optimal path as $p^{*}=(x,s_{1}^{*},s_{2}^{*},...,s_{T-1}^{*},y)\in P(x,y)$ which satisfy $\tilde{d}(x,y)=D_{\text{lang}}(p^{*})$. Then,

$$ $\displaystyle\forall x,y\in\mathcal{S},\ \lVert\phi(x)-\phi(y)\rVert$ $\displaystyle=\|\phi(x)-\phi(s_{1}^{*})+\phi(s_{1}^{*})-\phi(s_{2}^{*})+...+ \phi(s_{T-1}^{*})-\phi(y)\|$ $\displaystyle\leq\|\phi(x)-\phi(s_{1}^{*})\|+\|\phi(s_{1}^{*})-\phi(s_{2}^{*}) \|+...+\|\phi(s_{T-1}^{*})-\phi(y)\|$ $\displaystyle\leq d_{\text{lang}}(x,s_{1}^{*})+d_{\text{lang}}(s_{1}^{*},s_{2} ^{*})+...+d_{\text{lang}}(s_{T-1}^{*},y)=\tilde{d}(x,y).$ $$

This concludes the proof of (i).

#### Proof of (ii)

To show that $\tilde{d}$ is a valid pseudometric, we need to prove following three conditions: $\forall x,y,z\in\mathcal{S}$, $\tilde{d}(x,x)=0$, $\tilde{d}(x,y)=\tilde{d}(y,x)$, and $\tilde{d}(x,y)\leq\tilde{d}(x,z)+\tilde{d}(z,y)$.

At first, given that $\forall{x}\in\mathcal{S}$, $d_{\text{lang}}(x,x)=0$ , $\tilde{d}(x,x)=0$ holds trivially.

Secondly, given that $\forall x,y\in\mathcal{S}$,
$d_{\text{lang}}(x,y)=d_{\text{lang}}(y,x)$
,$\tilde{d}(x,y)=\tilde{d}(y,x)$ also holds trivially.

For the triangle inequality of $\tilde{d}$,

$$ $\displaystyle\forall x,y,z\in\mathcal{S},\ \tilde{d}(x,y)$ $\displaystyle=\inf_{p\in P(x,y)}D_{s}(p)$ $\displaystyle\leq\inf_{p_{1}\in P(x,z),p_{2}\in P(z,y)}D_{s}(p_{1})+D_{s}(p_{2})$ $\displaystyle=\inf_{p_{1}\in P(x,z)}D_{s}(p_{1})+\inf_{p_{2}\in P(z,y)}D_{s}(p _{2})$ $\displaystyle=\tilde{d}(x,z)+\tilde{d}(z,y).$ $$

This completes the proof of (ii).

#### Difference from CSD

The primary distinction is that we focus exclusively on the existing path $p\in P(x,y)$. Due to this specific focus, $\tilde{d}$ no longer serves as a lower bound for $d_{\text{lang}}$. We concentrate on the existing path because our real optimization data originates solely from this path. Consequently, we aim to align our theory more closely with the actual training scenario.

Another notable difference is that we update only with the adjacent pairs of states. Thanks to our claim 1., we still can induce a valid pseudo-metric across all pairs of states within the state space. This approach aims to bridge the gap between our theoretical model and the training data used for model updates.

## Appendix C Latent space visualization

Figure: Figure 11: Each sub-figure depicts a trajectory drawn from a policy conditioned on a randomly sampled skill. The left figure displays the actual paths of both the end-effector (blue) and the object (red) in the real world. The right figure illustrates the corresponding trajectory in latent space, generated using the learned $\phi$ function. A black arrow indicates the skill vector used to produce the trajectory. The numbers indicate transition step in the episode.
Refer to caption: extracted/5648683/img/latent_space/latent_space1.png

As discussed in Section 4.1, LGSD encourages the agent to distribute evenly across the latent space, which is regulated by the language distance, $d_{\text{lang}}$. Fig. 11 demonstrates that the agent’s trajectory in the latent space aligns well with the sampled skill vector. Since the skill vector is sampled isotropically, agent effectively spread out in the latent space. Additionally, it is noteworthy that the distance between two consecutive points in the latent space increases especially when the cube is moved. On the other hand, when the cube is stationary, there is no further extension in the latent space as can be seen in the Fig. 11 (a). This indicates that our language distance $d_{\text{lang}}$ serves as a meaningful proxy for semantic distance.

## Appendix D Full algorithm of LGSD

Figure: Algorithm 1 Language Guided Skill Discovery

## Appendix E Implementation details

### E.1 Observations

**Table 1: Ant Environment Observations**
| Name | Description | Dimension |
| --- | --- | --- |
| Base position | x,y,z position of the robot’s base | 3 |
| Base velocity | velocity of robot’s base in x,y,z direction | 3 |
| Base angvel | angular velocity of robot’s base | 3 |
| Base rotation | Yaw, Pitch, Roll of robot’s base | 3 |
| DOF position | Current angle of each DOF | 8 |
| DOF velocity | Angular velocity of each DOF | 8 |
| Previous action | Action executed in previous step | 8 |
| Sum |  | 36 |

**Table 2: Franka Manipulator Environment Observations**
| Name | Description | Dimension |
| --- | --- | --- |
| Cube pos | x,y,z coordinate of the target cube | 3 |
| Cube quat | Rotation of the target cube | 4 |
| EEF pos | x,y,z coordinate of robot arm’s end-effector | 3 |
| EEF quat | Rotation of Robot arm’s end-effector | 4 |
| Gripper q | Gripper width | 2 |
| Sum |  | 16 |

### E.2 Hyperparameters

**Table 3: Hyperparameters of LGSD**
| Name | Value |
| --- | --- |
| Learning rate | 0.0001 |
| Optimizer | Adam(Kingma and Ba []) |
| Minibatch size | 16384 |
| Horizon length | 32 |
| PPO clip threshold | 0.2 |
| PPO number of epochs | 5 |
| GAE $\lambda$ (Schulman et al. []) | 0.95 |
| Discount factor $\gamma$ | 0.99 |
| Entropy coefficient | 0.0001 |
| Initial Lagrange coefficient $\lambda$ | 300 |
| Dim. of skill $z$ | 2(Ant), 3(Franka) |
| Policy network $\pi$ | MLP with [256, 256, 128], |
| Activaion of $\pi$ | ELU(Clevert et al. []) |
| Representation function $\phi$ | MLP with<br>[256, 256, 128] |
| Activaion of $\phi$ | ReLU |
| Skill inference network $\psi$ | MLP with<br>[256, 256, 128] |
| Activaion of $\psi$ | ReLU |

## Appendix F Prompts used for each of the experiments

In this section, we share the prompts used for each of the experiments, and the examplary outputs from the LLM. One key detail is that we prompted LLM to finalize its sentence with a keyword of "Description:", so that we can parse the final description out of whole generated sentences.

### F.1 Prompts for enforcing Ant agent to move North

#### Input

You will describe a given robot’s state where the robot is moving on a plane. The state of the robot is as follows:

Robot’s x, y, z coordinates: [6.5, -0.5, -0.3]
Robot’s rotation: [0.9770, 0.1942, 0.0884, 0.0012]

If the y-coordinate is less than 0, we consider the area as southern. Conversely, if the y-coordinate of the robot is greater than or equal to 0, we consider the area as northern.

If the robot is in the northern area relative to the origin, provide the robot’s [x, y] position. Please conclude your answer with the final description in the format: ’Description: The robot’s x, y position is ___’.

If the robot is in the southern area relative to the origin, just state that ’Description:The robot is located in the southern area.’

#### Output

Since the y-coordinate of the robot is -0.5, which is less than 0, the robot is located in the southern area.

Description: The robot is located in the southern area.

### F.2 Prompts for enforcing Franka agent to push objects toward North

#### Input

You will describe a scene where a robot arm is approaching an object. The state of the robot arm is as follows:

Robot’s end effector’s x, y, z coordinates: [6.5, -0.5, -0.3], Object’s x, y, z coordinates: [0, 1.0, 0.2], Origin’s x, y, z coordinates: [0, 0, 0]

If the object’s y coordinate is less than 0, tell me ’Description: The object is located in southern area.’

If the object’s y coordinate greater than or equal to 0, tell me the object’s x, y coordinates. You can use any introductory words, but please conclude your answer with the final description in the format: ’Description: The object’s x, y position is ___.’

If the object’s coordinates match the origin’s, tell me the distance between the robot’s end effector and the object. Again, conclude with the final description in the format: ’Description: the distance is ___ units.’

#### Output1 - when object is moved

In this scene, the object’s y coordinate is 1.0, which is greater than 0. Thus, the x and y coordinates of the object are 0 and 1.0 respectively.

Description: The object’s x, y position is 0, 1.0.

#### Output2 - When object is not moved

In this scene, we observe a robot’s end effector positioned at coordinates [6.5, -0.5, -0.3] aiming to approach an object. The object itself is found at the coordinates [0, 0, 0], which are identical to the origin’s coordinates.

To analyze this setup, we calculate the distance between the robot’s end effector and the object. Using the Euclidean distance formula:

$$ $\text{Distance}=\sqrt{(x_{2}-x_{1})^{2}+(y_{2}-y_{1})^{2}+(z_{2}-z_{1})^{2}}$ $$

$$ $=\sqrt{(0-6.5)^{2}+(0-(-0.5))^{2}+(0-(-0.3))^{2}}$ $$

$$ $=\sqrt{6.5^{2}+0.5^{2}+0.3^{2}}$ $$

$$ $=\sqrt{42.25+0.25+0.09}$ $$

$$ $=\sqrt{42.59}$ $$

$$ $\approx 6.53\text{ units}$ $$

Description: The distance is 6.53 units.

### F.3 Prompts for Franka agent manipulates object in general

#### Input

You will describe a scene where a robot arm is approaching an object. The state of the robot arm is as follows:

Robot’s end effector’s x, y, z coordinates: [6.5, -0.5, -0.3], Object’s x, y, z coordinates: [0, 1.0, 0.2], Origin’s x, y, z coordinates: [0, 0, 0]

If the object’s coordinates differ from the origin’s, tell me the object’s x, y coordinates. You can use any introductory words, but please conclude your answer with the final description in the format: ’Description: The cube’s x, y position is ___.’

If the object’s coordinates match the origin’s, tell me the distance between the robot’s end effector and the object. Again, conclude with the final description in the format: ’Description: the distance is ___ units.’

#### Output

In the scene, a robot arm’s end effector is positioned at coordinates [6.5, -0.5, -0.3], while an object is located at [0, 1.0, 0.2]. The origin of the coordinate system is at [0, 0, 0]. Since the object’s coordinates do not match the origin’s coordinates, we focus on the object’s position relative to the origin.

Description: The object’s x, y position is [0, 1.0].

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - In Fig. 4, the coverage for the manipulation task is worse than the locomotion task. More analysis or explanations would be necessary to understand the limitations of the proposed approach.
- In Fig. 7, showing the coverage for LGSD in the whole space would be better. Although the proposed approach may have a larger coverage than the baselines, it is less evenly distributed compared to CSD. Is this always the case for different training runs? Any analysis or explanations would be helpful.
soundness: 3
strengths: - A theoretical proof of language-distance as a valid pseudometric is provided.
- Both locomotion and manipulation tasks have been demonstrated, showing the effectiveness of the proposed approach.
- The paper is well presented.
summary: This paper proposed to incorporate semantic diversity in skill discovery.  The authors show that the proposed approach with language guidance from LLMs outperforms existing skill discovery baselines. The proposed approach is demonstrated in both locomotion and manipulation tasks.
weaknesses: - More complex scenarios should be designed to make the use of LLMs necessary rather than simple positions.
- The lack of real-world experiments makes the proposed approach less convincing.
- A video showing the robot in action should be included to give readers an intuitive understanding of learning performance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Is it correct that for each state $s_t$, the LLM will be called to generate a description in order to compute $d(s_{t-1}, s_t)$? If so, how fast is the training?
- How will LGSD generalize to complex scenarios? Is it necessary for experts to compose (skill)prompts for each possible skill?
soundness: 3
strengths: - The method tried to link skill states with their semantic meaning, resulting in meaningful and language-controllable skills.
- LGSD provides a theoretical foundation for the language-distance metric as a pseudometric, detailing how it can approximate semantic diversity.
summary: Rather than to maximize state coverage, this work introduces LGSD to constrain skill discovery inside a semantically meaningful subspace, created by LLMs with crafted user prompts.
weaknesses: - The experimental scenarios are simple, in which the exampled prompts and semantically controlled spaces are easy to follow yet fail to demonstrate the generalizablity and scalability --- after all, the method relies much on the description of states. LGSD’s dependence on LLMs for real-time distance evaluation might limit scalability to complex, real-time environments.
- As I understand, users have to provide specified "skill constraints"  (via prompts, such as "move north" etc.), then how can it still be called "skill discovery" since users are specifying skills?
- The comparison with some baselines is somehow unfair since they lack the prior knowledge of users or any language embedding computation. A better comparison should be considered.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: ## Questions:
1. What is the dimensionality of the skill-space here? Is there any benefit to having higher dimensions / are more skills learned?

2. The performance of other skill discovery methods seems quite poor on downstream Ant tasks, which is surprising especially for LSD/CSD/METRA. Are the tasks or rewards defined here differently from their original papers?

3. Is the LLM necessary given those prompts? Could these examples not be done just as well with simple logic gates? For example, prompt G.3 could be defined as “if object not at origin, return position of object, else return distance between end effector and object” without any LLM. 
I understand that this is more or less a proof of concept, but I think it would go a long way if you can show some examples where the “common knowledge” of LLMs can be properly utilized. 


4. I am curious if specifying just the object’s Cartesian position as the sole encoder observation under a Euclidean distance constraint (LSD) would yield similar results. I guess it might struggle more with exploration (no signal before the end-effector has collided with the object), but it would be interesting to see.


5. I like the visualizations (and analysis) in Appendix D. As I understand, the language distance d_lang stays quite small when the object is at the origin, but increases when the object is pushed. Is there a specific reason why changes in the LLM output for the former (“The object’s x, y position is A,B”) results in smaller language distance than the latter (“The distance is X units.”)? On this note, it would be interesting to see how the language distance changes as a function of the robot state - is it more or less linear? 
Since the LLM output is constrained to the “’Description: The cube’s x, y position is ___ .’” and “’Description: the distance is ___ units.’”, I can see how the language distance could serve as a metric between the two behaviors, but would it really correspond to something much different than Euclidean distance on description-level? For example, “The distance is 4 units” vs “The distance is 6 units”.

## Summary:
Overall I think it’s a very well-written paper, with a good mathematical basis and satisfactory results. The results are promising, and I think there are good direction for building upon the methods, for example with VLMs. In my opinion, the proposed method has a lot of potential which is not fully utilized. My main concern is that the examples shown in the paper can be easily represented in a simpler way without an LLM. I can see that there are situations where this might not be the case, and showing these will make a much stronger argument for the proposed approach.
soundness: 3
strengths: - Interesting and novel contribution - while using LLMs is not rare, this is an interesting way of using them for encouraging skill discovery.

- The proposed method is well-supported mathematically.

- Good zero-shot language-specified goal tracking with the learned skill predictor network.
summary: “Language Guided Skill Discovery” proposes a method for learning diverse skills with unsupervised reinforcement learning by aligning skills with latent space transitions under a distance constraint. The novelty in the proposed method stems from the use of language distance as the constraint. This is done by a Large Language Model (LLM) which describes the robot state, and a language embedding model which encodes this description in a fixed real-valued vector where the distance between descriptions can be measured. The results show successful skill discovery on locomotion and robot manipulation environments.
weaknesses: - Not fully convinced that LLMs are necessary for the proposed scenarios. Showing the performance on more complex situations where the semantic understanding of LLMs is necessary would make the case much stronger.

- More qualitative evaluation (i.e. videos) would be good for evaluating the learned skills.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Why enforcing $\phi$ to be 1-Lipschitz continuous under $d_{lang}$? Is there any other way to constrain the difference in the latent space?
- In Section 5.1. In FrankCube environment, the intended half plane is only for NSEW directions. Can we see arbitrary directions, e.g., north-east or south-west?
soundness: 3
strengths: - This paper is pretty clear wrt the high-level topic and what the high-level intuitions are. The idea of incorporating LLMs to guide the skill discovery is novel and intuitive. Figure 2 is a very clear figure except one minor issue (see weakness).
- By introducing the representation function \phi which transform the state space s to latent space z, the skill discovery problem in the Euclidean space is transformed to an equivalent problem in the latent space. Figures in Appendix D show that identifying skills in the latent space is much easier than in the original space. Furthermore, by formulating the reward as $r(s, z, s^\prime) = ( \phi(s) - \phi(s^\prime))^Tz$, LGSD successfully transform the problem of finding diverse skills to a problem of maximizing the culmulative designed reward.
- The section 2.1 is a great section that summarizes works related to unsupervised skill discovery.  Researchers might already read some related articles, but such a classification (definition)  as "mutual information based " and "distance maximization" approaches is still helpful to the whole community yet very accurate.
- Experiments results show LGSD is effective and competitive in discovering semantically distict skills compared to other skill discovery baselines.
summary: This paper presents a novel skill discovery framework called Language Guided SKill Discovery (LGSD). LGSD aims to maximize the semantic diversity of discovered skills by utilizing language guidance from LLMs. In LGSD, the interactions between human experts and LLMs are utilized for 1) constraining the skill search space into a semantically desired subspace, and 2) encouraging the agent to visit semantically diverse states within the subspace. Various experiments are conducted to demonstrate the capability of the LGSD framework to find more diverse and distinctive skills compared to previous methods.
weaknesses: - Figure 1 shows a set of discovered semantically distinctive skills but the caption is not very informative. It could be better if description of each skill can be presented.
- Figure 2 is very good. However, it is unlikely for reader to understand $d_{lang}(s, s^\prime)$ without any additionally context. People may mistakenly treat this as some Euclidean distance between s and s^prime in the original space until they encounter equation (2). I would like to see some quick explanation that the $d_{lang}$ here happens in latent space, and this procedure requires LLMs.
- Prompts are required in this work to interact with LLMs to acquire the semantic knowledge. The design of the prompt is critical to the direction or subspace of skill discovery. However, what if we only want to leverage the semantic knowledge from LLMs without setting up to much language prior? From my understanding, too much control from human means less diverse and improvising skills.
- This is minor. In Line 319, it is unclear what is $s$ and $g$ without defining them.

Meta-review: The paper is well-presented. It proposes a novel method Language Guided Skill Discovery (LGSD), using language distances as for skill learning with LLMs. It demonstrates zero-shot goal tracking and skill discovery in diverse tasks. 

Strengths include a mathematically sound approach, effective language-constrained skill prediction, and promising results in new tasks. It includes theoretical validation and simulated demonstrations in locomotion and manipulation tasks. 

Weaknesses include limited evidence for the necessity of LLMs, and a lack of real-world experiments. Comparison with baselines is not completely fair since those methods did not use prior language-based knowledge.

Decision: Accept (Poster)