Language Guided Skill Discovery
Authors: Seungeun Rho
Laura Smith
Tianyu Li
Sergey Levine
Xue Bin Peng
Sehoon Ha
Venue: ICLR 2025 Poster (2025)
Keywords: Unsupervised Skill Discovery
Guided Skill Discovery
Reinforcement Learning
Language Guided RL
Primary area: reinforcement learning
TL;DR: We present a skill discovery method that enable discovery of semantically diverse skills using LLM.

Abstract: Skill discovery methods enable agents to learn diverse emergent behaviors without explicit rewards. To make learned skills useful for downstream tasks, obtaining a semantically diverse repertoire of skills is crucial. While some approaches use discriminators to acquire distinguishable skills and others focus on increasing state coverage, the direct pursuit of ‘semantic diversity’ in skills remains underexplored. We hypothesize that leveraging the semantic knowledge of large language models (LLM) can lead us to improve semantic diversity of resulting behaviors. In this sense, we introduce Language Guided Skill Discovery (LGSD), a skill discovery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. The prompts serve as a means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace. We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in discovering more diverse skills compared to five existing skill discovery methods in robot-arm manipulation environments. Lastly, LGSD provides a simple way of utilizing learned skills via natural language.

Published as a conference paper at ICLR 2025
LANGUAGE GUIDED SKILL DISCOVERY
Seungeun Rho
Laura Smith
Georgia Institute of Technology @cho3l@gatech... edu
University of Californaa, Berkley
smithlaara@berreley edu
Tianyu Li
Sergey Levine
Georgia Institute of Technology t11471 14471gggtecc.... .du
University of California, Berkley
svlevine@eecs hetpeeeeees.bbeeeee..e.e.. berkeley.ee....k.......... edd:
Xue Bin Peng Simon Fraser University xbpeng@sfu.ca
Schoon Ha
Georgia Institute of Technology sehoonha@gatesn edu
ABSTRACT
Skill discovery methods enable agents to learn diverse emergent behaviors with- out explicit rewards. To make learned skills useful for downstream tasks, obtain- ing a semantically diverse repertoire of skills is crucial. While some approaches use discriminators acquire distinguishable skills and others focus increasing state coverage, the direct pursuit 'semantic diversity" in skills remains underex- plored. We hypothesize that leveraging the semantic knowledge of large language models (LLM) can lead us to improve semantic diversity of resulting behaviors. In this sense, introduce Language Guided Skill Discovery (LGSD), a skill dis- covery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs set of semantically dis- tinctive skills. The prompts serve as means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace. We demonstrate that L.SDD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in dis- covering more diverse skills compared to five existing skill discovery methods robot-arm manipulation environments. Lastly, LGSD provides simple way of utilizing learned skills via natural language.
INTRODUCTION
One of the key capabilities of intelligent agents is autonomously learn useful skills applicable downstream tasks without task-specific objectives. Consider how human infants acquire manipula- tion skills through simple play with toys, which can later be applied to a broader set of tasks, such as using a spoon or holding a bottle. Our research aims to emulate this process of skill learning. Prior works in unsupervised skill discovery suggest that maximizing diversity of behaviors can be a way to develop such skills (Gregor et al., 2016 Eysenbach et al., 2018 K Kwon 2020). These ap- proaches operate under the assumption that acquiring wide range of diverse behaviors may lead to the development of useful skills for downstream tasks. Building on this premise, studies (harmal et al. (2019): Hansen et al. (2019): Liu & Abbeel (2021a): [Lskin et al. (2022) have associated be- haviors with random vectors by maximizing Mutual Information between them acquire diverse skills. Additionally, there are other approaches aimed increasing coverage of the state space (Liu) & Abbeel 2021b: Yarats et al. 2021: Park al. 2021), exploration-basee strategies that leverage
emrors in predictive models to enhance learning (Burda et al. 2018 Pathak et al. 2019 Park et al.
2023a).
However, we contend that these measures are proxies for what we term "semantic diversity" and they do not necessarily reflect the semantic diversity of repertoire of skills. For instance, in high

---

Published as a conference paper at ICLR 2025
Innreres brick
hanares mug
meat can, brick
meat can, mug
Figure We proposed LGSD which can discover a semantically distinctive set of skills. We showcase four sample skills acquired from a single training run. Our approach successfully learned skills that manipulate only 'edible' objects (banana and meat.can) from a total of four objects.
degrees freedom (DOF) systems, such as robot-arm manipulation tasks, unstructured movements of each DOF might exhibit high diversity in terms of state coverage distinctiveness from neural discriminator, but nonetheless lack meaningful semantic diversity. more accurately measure semantic diversity, propose the use of Large Language Models (LLM) as they are well-suited for understanding and assessing the semantic significance of different behaviors in a way that better aligns with human judgment.
In this work, we present Language Guided Skill Discovery (LGSD), a skill discovery algorithm that utilizes the guidance of LLMs to learn semantically diverse skills. LGSD aims fulfill three desiderata for skill discovery. (i) Firstly, we want to discover skills that have diverse semantic mean- ings. We LLMs generate descriptions for each of agent states. Based on these descriptions, we measure the semantic difference between states using language-distance, and train skills to max- imize this distance. Because LLMs are equipped with semantic understanding of the world, this provides a way to maximizing semantic differences between skills. Fig. [ illlustrates skills learned from LGSD, showcasing the semantic diversity of resultant behaviors. (ii) Secondly, we seek to restrict the search space for skills into desired semantic subspace. Intuitively, restricting the skill space akin to enforeing an infant to grab toys only with their hands, and not with their mouth or feet. We implement this desideratum by utilizing language prompts. Humans provide prompts to LLMs, and the generated descriptions are then constrained by these prompts. This will allow our method to ignore differences outside the intended semantic space. (iii) Lastly, aim for the learned skills to be easy to use. When the skill space is continuous, merely selecting the best skills for target tasks becomes non-trivial. LGSD) supports zero-shot language instruction-following capability by training separate network t for skill inference. Humans can provide a natural language description of the desired state, and then LGSD can infer which skill should be used reach that goal state.
The primary contributions of this work are as follows. 1) that enables the discovery of semantically diverse skills by utilizing language guidance from LLMs. To the best of our knowledge, this is the first work to incorporate semantic diversity into skill dis- covery. 2) We present method of constraining the skill search space to user-defined semantic space using language prompts. This allows users to control the resulting skills by simply specifying different prompts. We emphasize that, prior to training, existing methods only offer limited control over the skills to be learned. 3) We provide theoretical proof of how language-distance a proxy for semantic distance, can be employed as a valid pseudometric. 4) We propose a method that enables an agent to reach goal state specified by natural language, facilitating the convenient use of learned skills. 5) We demonstrate that our method can train semantically diverse skills. Through extensive experiments, we show that LGSD outperforms five different skill discovery baseline methods both locomotion and manipulation tasks, in terms of both diversity and sample efficiency.
We propose a skill discovery framework
2 RELATED WORK
2.1
UNSUPERVISED SKILL DISCOVERY
Mutual information based approach Common approaches to associate skills with corresponding
behaviors are based on maximizing the mutual information (MI) between states S and skills Z, or
I(S; Z) (Gregor et al. 2016 Eysenbach et al. 2018; Sharma et al. 2019 Hansen et al.) 2019 Liu & Abbeel 2021a| Laskin et al. 2022). MI can be viewed as a dependency measure that quantifies the association between random variables. To maximize this quantity, popular early formulation
2

---

Published as a conference paper at ICLR 2025
utilizes the decomposition I(S; 2) = -(2)) where H|- refers to Shannon entropy. Since directly computing H( is intractable. Eysenbach E et al. (2018) introduce a variational posterior
(=(=))) to optimize the lower bound. Similarly, CIC (Laskin et al. 2022) maximizes I((s, s'); Z), promoting distinct state transitions across different skills. However, a significant limitation of using MI the objective arises from the use of Kullback-Leibler (KI.) divergence to define MI:
I(S 2) def DxL(P((/2)IIP(S))))
Maximization of MI using KI. divergence often results in skills with less distinctive behaviors. This is because KL divergence is fully maximized when two densities have no overlap. posing a problem as there is no further incentive to distinguish the densities beyond this point. This is problematic because having no overlap between two densities does not necessarily mean two skills are noticeably distinctive. In practice, this becomes a severe shortcoming of DIAYN (Eysenbach where its objective can be fully optimized as long as the discriminator Is can distinguish them perfectly. Neural networks can easily distinguish minor numerical differences, so the skills learned with this objective often have only these minute differences.
al.) 2018),
Distance maxinization approach To overcome these shortcomings, recent skill discovery works (He et al. [022 Park et al. 2021 2023a b) propose using the Wasserstein Dependency Measure
(WDM, Ozair (2019))
Iw 2) dff W(P(S, ((P(S)))
WDM is defined using the Earth-Mover (EM) distance and retains the advantage of always pro- viding an incentive to maximize the distance between distributions, even when they are already distinguishable.
An essential characteristic of Tw is that it must be defined within a metric space. Therefore, the choice of metric to measure the difference between skills comes to the gist of the algorithm since it ultimately governs the resulting behavior of each skills. LSD (Park et al. (021) employs the Eu- clidean distance state space as its metric. Maximizing Iw under this metric encourages agents to visit states that are are far apart in terms of Euclidean distances. Conversely, CSD (Park et al.) 2023a) points out the problem of using Fuclidean distance as the metric. They argue that maxi- mizing Euclidean distance tends to focus on "easy" changes in the state space, such moving the robot arm itself rather than the target object. Thus, CSD proposed to maximize "controllability- aware distance" instead. This metric measures the negative log likelihood of transition dynamics - llo p(('(s). Is). Using this metric means the algorithm favors visiting rare transitions. LGSD adopts a similar approach but differentiates itself by employing language-based distance metric within the WDM framework, to maximize the semantic difference.
2.2 USING LANGUAGE AS GUIDANCE FOR BEHAVIOR LEARNING
Recent work has studied using semantic knowledge in large pre-trained models for control tasks in several ways. A popular approach is use the common-sense reasoning capabilities of LLMs to produce high-level plans over known, existing skills through the interface of natural language
2022 Ha et 2023 or (Sharma al. 2021 Ahn et 2022 Singh et al. 2023 Huang al.
generated programs (Liang et al. 2022 Zeng et al. 2022] Vemprala et al. 2024 In addition. LLMs
have also been shown be effective in directly producing low-level actions that can be executed by an agent (Wang et al. 2023 Kwon et al., 2023 Driess et al.] 2023 Brohan et al., 2023). Our
work does not use LLMs to generate actions at any level abstraction but to provide a meaningful distance measure to guide learning towards discovering semantically meaningful skills.
Several works have used L.LMs to guide agents learn semantically meaningful behavior by using them to design reward functions for either given desired task (Yu et al. 2023 Ma et al. [2023). This showcases LLMs' capacity to understand specific physical embodiments assorting it with semantic knowledge reason about given tasks. We note that these works aim solve single specific task, whereas our method aims to learn repertoire of diverse behaviors without explicit supervision.
The work most related to ours is Du et al. (202)), which employs LLM to guide exploration. Their approach uses the LLM to suggest plausible set of goals for the agent, maximizing the cosine similarity between the language embeddings of the goals and the current state. While they focus
3

---

Published as a conference paper at ICLR 2025
Φ:8-2
State space
Latent space
LAME br
Robot is far from the origin"
N(0
ding(8,,
((a)
origin
"Robot near the origin"
Prompt
the robot from origin?
[ - <('')l & dhang(o,
Figure 2: Overview how LGSD works. Given a prompt. the LLM generates the description for each state. We then measure the difference between these descriptions and denote it as dlang- Based on duung we constrain the latent space by enforcing the 1-Lipschitz condition on Φ. Then the agent is encouraged visit states that make the vector o(s') ((s) aligns well with randomly sampled vector z from an isotropic Gaussian prior. This makes the agent explore the latent space in diverse directions depending on the sampled z.
on using an LLM to reason with a low-dimensional, discrete set of high-level actions in complex environments like Crafter and Housekeep, our work focus on discovering meaningful behaviors in complex, continuous control spaces where actions (e.g. joint targets) not readily interpretable. Furthermore, their reward function, based on language similarity, lacks theoretical support as it is not valid metric. In contrast, method effectively handles the language-distanee distance measure and provides a theoretical framework for its use. We elaborate on this Section
3 PRELIMINARIES
Unsupervised skill discovery can be formalized within the framework reward-free Markov De- cision Process (MDP). denoted as M AS P}. Here. S represents the state space, A denotes the action space, and P x A - [0, oc) defines the transition dynamics. During training, a latent vector z sampled from prior distribution fixed throughout the episode. This vector informs the policy [als, 2), guiding the generation of corresponding rollouts through π. The training procedure aims to foster a dependency of the latent vector z on its resultant behavior, allowing it to function effectively as a skill. It is referred to as the skill because being conditioned on different skill æ aims behave differently by encouraging agents to visit distinct distributions of states s, transitions (s, s'), or trajectories A common approach to training skills involves defining intrinsic rewards int x A - and optimizing the accumulation >'r?"" Various
p(=) the start of each episode and remains
of discounted intrinsic rewards with the following objective: J Ez.P T
off-the-shelf reinforcement learning (RL) algorithms can be employed to optimize π to maximize this objective. Notably, this process does not utilize any task-related explicit rewards.
LANGUAGE GUIDED SKILL DISCOVERY
In this section, we begin by providing an overview of algorithm, followed by in-depth explana- tions of the prompting, language distance measurement, and how we can utilize the acquired skills using language prompts.
4.1 ALGORITHM
Our goal is for agent to learn corpus of skills that visits semantically different states. Discov- ering semantically distinct states requires 1) a way of measuring the semantic difference between states, and 2) means of maximizing this difference measure. To evaluate the semantic differences between different behaviors, we leverage the power of LLMs. We will consider two states as sub- stantially distinct if the language descriptions produced by LLMs semantically different. As
4

---

Published as a conference paper at ICLR 2025
LLM is queried to produce natural language description for each state. This description links the state to its semantic meaning. We then measure the difference between these language descriptions. To measure the difference between two language descriptions, we lever- age pre-trained natural language embedding model, 2019). to map the language descriptions into fixed-dimension real-valued vectors. This allows us to measure the difference between these vectors using cosine similarity. We refer to this difference measure as language-distane and denote as diang-
illustrated in Fig. 2
(Reimers & Gurevych,
Given a measure of the difference between language descriptions diang. which acts as a proxy for the semantic differences between states, the next step distance in order to discover diverse skills. We utilize existing distance maximization skill discovery algorithms, discussed in Section 2.T Intuitively, we aim for agents to spread out from the origin in the latent space, which is constrained under dhang- This constraint ensures that spreading out in the latent space leads to maximizing sum of thang along the path in the state space. More specifically, we train a representation function Φ S + Z that maps state space S onto latent space Z. Here, enforce that Φ is I-Lipschitz continuous under dhang. meaning that V(s,s') E S, l - d(s)]l s drang (s, s'). Thus, the Euclidean distance between and s' in latent space is always less than or equal to diang (s, s'). Hence, simply maximizing the Euclidean distance in the latent space results in maximizing tlang in the state space.
incentivize an agent maximize this
Spreading out in the latent space can be achieved by aligning the vector d(s') o(s) with a latent skill vector sampled from Gaussian distribution ~ (O0 7). Because the vector 2 is sampled isotropically, aligning ((s') ((=")= o(s) with fosters a diversity of behaviors depending on the sampled (o(s') - (0s))² "² z and maximize the cumulative reward achieve this objective. Here, Park et al. (2023b) showed that maximizing the cumulative reward r(s, z, s') with a I-Lipschitz continuous @ is equivalent maximizing the WDM between the terminal state ST and Z:
2. We use the reward 2, 2, 5')
arg max Iw(ST; 2) arg max E xxp() 3
(6( = -((s)))
1=0
s.t. V(s E E, l1o(s') x(s)) S dlang(s,
(1)
B Therefore, the meaning of maximizing traveled language- distance is equivalent to maximizing the WDM between $T and 2 under
We provide the proof in Appendix
distance. Any off-the-shel model- -ree RL algorithms can then be used to maximize the sum of rewards. We used PPO (Schulman et al. 2017) for as our primary RI. algorithm because of stable convergence. To ensure @ is I-Lipschitz continuous under distance metric d, we use dual gradient descent with with a Lagrange multiplier (Boyd& & Vandenberghe/ 2004).
4.2 CONSTRAINING THE SEARCH SPACE VIA PROMPTS
The potential space for skills is vast, yet only narrow region possesses semantic meanings. We want to explore only within subspace which has inherent semantic meanings. To achieve this desiderata, we use language prompts to constrain the entire search space into the desired semantic subspace.
Proupt
cube noved, tell ne cube's noved distance.
If not, tell the distance betaeen end-effector and the cube.
dj
Fig. 3 shows a manipulator environment that aims to move an object. In this environment, we can consider skills enabling object interactions as semantically meaningful while consid- ering unstructured robot-arm movements not. Our language- based distance allows us to capture such delicate semantics of skills. For instance, we prompted an L.MM to describe the dis- tance dj between the robot arm and the object unless the object is moved already. Therefore, the scene description remains the same unless there is change in the robot-object distance di. Because daag becomes positive only when the description of state is changed, our distance maximization algorithm courages an agent to visit states that can change descriptions, which makes robot arm to approach the object.
d2
Figure By prompting the LLM to generate different descriptions depending on the state, LGSDD can adapt its focus during training.
5

---

Published as a conference paper at ICLR 2025
Moreover, highlight that induce the LLM to not only focus single semantic aspect of the scene but also to adapt its focus onto different semantic aspects throughout the training. For instance, when the robot arm finally reaches the cube, the LLM is then asked to describe the object's moved distance d2. This additional prompt shifts the focus of the semantics to the object's traveled distance and any other changes in the states are ignored by the LLM. This leads the robot arm pushing the object to its maximum extent. Note that existing skill discovery algorithms have limited capability to constrain its focus by feeding subset of observations (Eysenbach al. 2018 Park al. 2023a). In contrast, LGSD enables not only focusing on semantic dimensions but also adapting its focus across different semantics during training.
4.3 LANGUAGE DISTANCE AS A VALID METRIC
In Section 4.1
stressed that diang establishes criterion for determining the distance between any two points in the latent space. Therefore, maintaining a coherent criterion is crucial for building the latent space. In this sense, diang should be valid distance metric. In this section, we present how duag can be employed as valid (pseudo-) metric, despite not satisfying the triangle inequality.
We begin with defining diang formally. Let us denote a user prompt as (prompt- and rule-based state annotator as : S + latate- The annotator g is a pre-defined function that translates state vector into corresponding natural language sentence without any addition or loss of information. Then, we denote the output description of an LLM for state s as (s) = LLM p-Ppprppp Isate). We configure the LLM to be deterministic with zero temperature during generation, thus treating as a function. Next, we employ pre-trained language embedding model Jemba laese R.N. Using this setup, we can now define the language distance thang as follows:
V(s, S, chang (8, 8¹) fembbd (Iaeso(s)) (s))² Semba (ldesc(s') (s')) fenba (Idess (s) Tembel (ldese (s'))]I
However, to be considered as pseudometric, function d S x S + R must satisfy three con- ditions: Vz, y, 2 E S. (i) d(x,, æ) = (ii) d(x,, y) d(y, .)), and (ii) d(x, S d(z, y) + d(y, 2). In our case, drang satisfies conditions (i) and (ii), but does not meet the condition (iii), the triangle inequality.
To overcome this problem, we propose to impose the constraint in Eq. I only for adjacent pairs of states s') E Sadj. where two states and s' are considered adjacent if s' can be reached from within single state transition. Therefore, every encountered transition sample of (s, s') during training is adjacent. We claim that this modification to Eq. T will implicitly induce a valid pseudometric for all state pairs V(s,s')E E.
Claim 1. For any function drang S x S R4, there exists a valid pseudometric d SSXx S + R+ that imposing the Eq. [ on adjacent pairs of states lead to imposing the same equation every possible state pairs under d, i.e.
Va, y E Sadj, [}((x) d())1 S tlang (I, Vz, y E S, ((((s)
- أ(y)ه s d(z,y) v)-
We provide a proof of the claim in Appendix C objective function as follows:
Applying this modification, we came the final
Σ ((66s))))((s1))
JLOSD = E (x))pp(
s.t. ((s E Suij, (1 S dlan (8, s").
Consequently, this objective encourages agents to explore diverse and distant states in latent space, resulting in greater language-distand traveled along the trajectory within the state space.
4.4 UTILIZING LEARNED SKILLS USING NATURAL LANGUAGE
After training is complete, need way of selecting specific values to utilize the behaviors associated with each skill to solve concrete tasks. However, we lack knowledge about which 2 leads to the desired behavior. We can sweep over the skill space fixed intervals (Laskin et al.

---

Published as a conference paper at ICLR 2025
0.4
0.3
03
00
-0.
North
Clice
Weat
Figure 4: Trajectories of different skills trained with different prompts. For the Ant (top row), recorded the base's i æ, y coordinates. For the Franka robot-arm agent (bottom row), we recorded the 2, y coordinates of the object on the table.
2022 sample number of skills and evaluate each with hundreds of rollout trajectories (Park
et al. 2023a). To overcome these challenges, Park et al. (202T) 2023b) have suggested using 2 (l(g) <(s)) /l((g))) o(s)]2 2 for continuous skills, when have Lipschitz continuous Φb. Yet, in many cases, we cannot fully specify the desired goal state. For instance, consider an agent using a robotic arm move cube. The state includes the end-effector's 's rotation and position, as well as the cube's coordinates. While you can specify the cube's 's position, the required rotation and position of the end-effector to achieve that position may not be known.
To address the challenge of goal state specification, we propose training separate network v (eues + Z to infer the skill. Intuitively, & takes the embedding of the language description of the state as input and infers which 2 was used to reach that specific state. Since we are gathering pairs of state and the corresponding skill during the training of т, we can train & using the same data. After training. we can use v and Sgaal to produce Zgoal, which enables the agent to reach Sgoal- We then feed Egoal into π, allowing the agent to reach the goal state as described in natural language in zero-shot manner. The full algorithm, including the training of vb is presented in Appendix E
EXPERIMENTS
In this section, we evaluate our proposed LGSD by conducting a series of experiments on continuous control environments, encompassing both locomotion and manipulation setups. We aim to answer four questions: (1) Can prompting constrain the skill space into a desired semantic subspace? (2) Can language guidance lead obtaining more diverse skills compared to unsupervised skill discov- ery baselines? (3) Can we utilize learned skills for solving downstream tasks? (4) Can we employ learned skills using natural language?
Experimental setup We trained our algorithm and baselines using Isaac Gym (Makoviychuk et al., 2021), a high-throughput GPU-based physics simulator. For the language model, we em- ployed gpt- --tuuroo-oo-o- 202 4 04- 0 Achiam al. 2023). We the temperature parameter of the language model to 0 to get consistent, low-variance measure of diang. To reduce the number of unique queries, we discretized states and cached the input and output of these queries and reused them during training. We provide the exact prompts used for each experiments in Appendix G
5.1 CONSTRAINING THE SKILL SPACE INTO DESIRED SEMANTIC SUBSPACE
We first explored how LGSD constrains the skill search space into desired semantic subspace using prompts. To visually present this concept, we designed experiments where agents visit mutually exclusive states depending the prompts. We tested this idea in environments, Ant and
7

---

Published as a conference paper at ICLR 2025
0,6
1000
0.5
CS0
E23
METRA
1000
METRA
LSD
DIAYN
DIAYN
600
0.3
100
0.2
0.0
1000 3000
5000
10000
15000
20000
1000 3000
5000
10030
15000
20000
rgddates
updacces
(a) Object moved distance
(b) Object state coverage
Comparison of the object's moved distance and state coverage by LGSD against baselines. LGSD moved the object (a) further and toward (b) more diverse directions. Additionally, it is more sample-efficient with the aid of language guidance.
Figure 6:
FrankaCube. In the Ant environment, we required the agent to traverse only half of the plane. For instance, we allowed an agent to visit any location the northern part of the plane but restricted it from entering the southern area. We repeated these experiments for each cardinal direction: North, South, East, and West (NSEW). Similarly, in the Fz FrankaCube environment, we intended for the robot arm to move the object only half of the plane for each of the NSEW directions starting from the origin. We trained four agents, each with its own unique prompt, encouraging the agent to explore only within the intended semantic subspace.
Fig. (and Fig. 10 in Appendix) shows the discovered skills trained with each prompt. For each plot, sampled
(o I) 150 times independently and then generated trajectories using skill-conditioned policy. As shown in Fig. 4 LGSD effectively constrained the skill space into the desired subspace, especially
the Ant environment. For the Franka agent, the robot arm was able to move the object into the intended region, but it could not fully cover all skills within the constrained subspace. We believe this limitation is due the inherent nature of the exploration problem. In the Ant environment, the dynamics of the agent is invariant to horizontal translation, which allows effective control at any location. On the other hand, moving the cube along a long trajectory often requires changes in pushing strategies with different contacts and force profiles.
5.2 LANGUAGE GUIDANCE AIDS DISCOVERING DIVERSE SKILLS
To observe the effect of language guidance, we evaluate the diversity measure of the skills learned using LGSD against unsupervised skill discovery base- lines. For the baselines,
take two classical mutual information-based skill discovery algorithms: DIAYN (Eysenbach et al. 2018) and DADS (Sharma al. 2019), well three state-of-the-art distance maximization based ap- proaches: LSD (Park et al. 2021), CSD Park et al. 2023a), and METRA (Park
al. 2023b).
Although METRA was originally designed for pixel-based input tasks, it can also handle vector-based inputs. Therefore, we used vector-based state inputs for all methods to facilitate direct comparison. For all methods, we utilized induce
Figure 5: Initial state of robot arm.
no human prior knowledge in selecting parts of the state dimension desired behaviors; instead, passed full observations directly to the skill dis- covery algorithms. Detailed information about the observations is provided Appendix F2.1
F221
We set up challenging manipulation task illustrated in Fig. 5 Here, expect to discover skills that can manipulate the object toward diverse directions. The challenge arises from the initial distance between the robot arm and the object, which is set to the maximum possible value. We believe this task can confirm the validity of the two-stage prompt suggested section 4.2 After training, randomly sampled 200 skills and measured both the average moved distance of the object and the state coverage that counts the total number of visited horizontal grids, each sized lcm lem.
8

---

Published as a conference paper at ICLR 2025
(GSD(Ours)
CSD
DIAYN
-62
0.6
Figure 7: Paths of objects moved horizontal table by LGSD and five baseline algorithms, illustrated using the best- pest-performing checkpoint from four runs for each algorithm.
ArrSingleSoal
AntMurtGaal
150
0.3
METRA
DADS
03
0.
0.4
0.6
cm Steps
steps
Figure 9: The training curve of the high-level controller in locomotion tasks, using the low-level controller trained with LGSD) and five state-of-the-an unsupervised skill discovery methods. LGSI) is able to achieve better returns and exhibits better sample efficiency.
We present the results in Fig. 6 As shown in Fig. 6 LGSD discovers more diverse states compared to baseline algorithms. Skills trained using LGSD moved the object an average of 0.5m, which is five times greater than the value achieved by the second-best algorithm. This result confirms that language guidance from an LLM aids in discovering both diverse and meaningful skills. Notably, all maximmiatiiiiiiiiinnn methods (CSD, METRA, LSD) could manipulate the object, whereas MI- based methods could not. Fig. 7 shows qualitative results of the learned behaviors by each method, where LGSD demonstrated the ability manipulate objects to more grid areas. Additionally, visualizations and an analysis the agent's movements in latent space are provided in Appendix
5.3 HARNESSING LEARNED SKILLS
Zero-shot task solving via natural language We first demonstrate how our agent can solve down- stream task in zero-shot manner via natural language instruction. As discussed in Section [..4] we utilize Sembea to infer a skill to reach the goal state. For instance, our instruction can be a description of the desired state, "The ob jbject is located at [0.3, 0.2]". which does not contain information regarding the robot-arm's position or orientation. The learned v allows an agent to translate the given language instruction into an effective skill for reaching the goal state.
Goal 1
Goal 2
Goal
Goal 6
Goal 5
Goal
Figure 8: Users can specify goals in natural language. Blue dots represent the desired position of the object, while red dots indicate the object's trajectory.
Fig. 8 illustrates scenarios that robot-arm moves the object the six different goal positions described in natural language. This experiment showcased that the agent can understand the desired goal state with the help of v. execute the skill accurately, and successfully move the cube to the target positions in zero-shot manner. We also provide the results of the zero-shot goll-following agent locomotion domain Appendix, Fig.

---

Published as a conference paper at ICLR 2025
Training high-level controller with learned skills Although our method enables training de- sired set of skills to some extent using tailored prompts, it can be challenging for users to specify method can act
exact prompts that produce behaviors for more complex tasks. In such scenarios, as a pre-training module, which trains a low-level controller Toow(a 2), which can then be lever- aged by high-level controller Thigh (2) select the appropriate skills for new downstream task, represented by g.
We evaluate the performance of high-level controllers that leverage a pre-trained LGSD low-level controller on two downstream tasks: Ant Sintiingla eeGoal and AntMult Goaal. Details of the downstream tasks are available in Appendix F.1 Fig. shows that by using the set of skills trained with our method, the high-level controller can achieve higher returns and better sample complexity compared to prior skill discovery methods. We hypothesize that these improvements comes from the fact that the skills discovered through LGSD can better focus on more semantically meaningful changes in the x. y positions by leveraging guidance from the LLM,, while other methods simply seek changes in all 39 dimensions of the states equally.
CONCLUSION
Our work introduced Language Guided Skill Discovery (LGSD), novel framework for skill dis- covery that leverages the semantic understanding capabilities of large language models (LLMs) to guide the learning of semantically diverse skills. By incorporating language as a tool for both con- straining the skill space and measuring semantic diversity, LGSD offers a novel approach to learning diverse skills, We have demonstrated through various experiments that LGSD not only constrains skills within semantically meaningful subspaces but also enhances the diversity and applicability of the skills learned.
Despite these advancements, there areas for further exploration and improvement. The approach could benefit from a more nuanced understanding of the scene, potentially by incorporating vision- language models (VLMs). We conjecture that some semanties can be more easily captured through vision. VLMs would effectively help to utilize these dimensions. Furthermore, we suggest that incorporating trajectory-level semantic differences, instead of state-level, could be an interesting future direction. We could provide entire trajectories to LLMs/VLMs and query them to associate them with semantic meanings.
Overall, we believe our work marks the beginning of a series of efforts that capture semantic diver- sity between skills. We hope this work facilitates further research endeavors in discovering seman- tically meaningful skills with the aid of external knowledge sources, including LLMs.
Reproducibility Statement We have made significant efforts to ensure the reproducibility of our work across various aspects.
Detailed information about the observations used in our experiments is provided in Ap- pendix F2.1
A comprehensive pseudo-code of our algorithm is available in Appendix
A rigorous proof of Claim included Appendix C
Visualizations of the learned representation space can be found in Appendix D
Details of the high-level tasks training are provided in Appendix F.I
The exact version of the LLM used and all prompts are documented in Appendix G
This research has been funded by the Industrial Technology Innovation Pro- gram (P0028404, development of a product-level humanoid mobile robot for medical assistance equipped with bidirectional customizable human-robot interaction, autonomous semantic naviga- tion, and dual-arm complex manipulation capabilities using large-scale artificial intelligence mod- els) of the Ministry of Industry, Trade and Energy of Korea.
Acknowledgements
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wrxXiv:333308877: 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv e-prints, pp. arXiv-2204, 2022.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-aue models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.158/1 2023.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus). arXiv preprint arXiv:1511.0728) 2015.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.0333 2023.
Yuging Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning. 2023. URL https://api. semanticschon ortp:rrooo/ooosss2155555555
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- fions, 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra, Variational intrinsic control. arXiv preprint 161
I. 1.07507, 2016.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-gune guided robot PMLR, 2023.
skill acquisition. In Conference on Robot Learning, pp.
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- tional Conference on Learning Representations, 2019.
Shuncheng He, Yuhang Jiang, Hongchang Zhang, Jianzhun Shao, and Xiangyang Ji. Wasserstein unsupervised reinforcement learning. Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 36, pp. 6884-6892, 2022.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng. Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint urXiv:2207.05603 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Taehwan Kwon. Variational intrinsic control revisited. In International Conference on Learning Representations, 2020.
Teyun Kwon, Norman Di Palo, and Edward Johns. Language models as zero-shot trajectory gener- ators. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023. URL
https://spenrevii net wor.morrmmmeooom

---

Published as a conference paper at ICLR 2025
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. In Advances in Neural Information Processing Systems, 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprini arXiv:2209.07753 2022.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-6747. PMLR, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 14:18459-1847,, 20211.
Yecheng Jason Ma, William Liang Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representa- tions, 2023.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning, Advances in Neural Infor- mation Processing Systems, 32, 2019.
Seohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- tations, 2021.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability unsupervised skill discovery. In International Conference on Machine Learning, pp. 27225-27245. PMLR, 2023a.
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised with metric-awane abstraction. In The Twelfth International Conference on Learning Representations, 2023b.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, pp. 5062-5071. PMLR, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). Association for Computational Linguistics, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1006::11506.02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awan unsupervised discovery of skills. In International Conference on Learning Representations, 2019.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.015/1 2021.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay. Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-23-11530, IEEE, 2023.
12

---

Published as a conference paper at ICLR 2025
Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. IEEE Access, 2024.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Yen-Jen Wang, Bike Zhang, Jianyu Chen, and Koushil Sreenath. Prompt a robot to walk with large language models. arXiv preprint arXiv:2309.09990 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference Machine Learning, pp. 11930-11931. PMLR, 2021.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint 22N4::224.00555 2022.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL RESULT FIGURES
(a) Moving North
(b) Moving East
(c) Moving South
(d) Moving West
Figure 10: LGSD constrains different prompts into different semantic subspace of skills. Each yellow line indicates trajectory generated using randomly sampled skill learned with each prompt. Using different prompts, each set of skills is constrained to visit the (a)Northern, (b)Eastemn, (c)Southern, and (d)Western areas the plane.
Goal
Goal 2
Goal
Goal
Goal 6
Goal
Figure 11: Ant agents move toward the goal. Red dots indicate the actual path of the agent's 's base, while the blue dot represents ths goal. Each goal is specified using natural language.
B PROOF MAXIMIZING WASSERSTEIN DEPENDENCY MEASURE
We begin with the alternative form of Wasserstein distance using Kantorovich-Rubensteinn duality following Villani et al. (2009), to make Iw tractable.
(S:
sup JECsx2
where Lsxz is set of all I-Lipschitz function defined in S x Z + R under the distance metric d. For the remainder, we follow the proof outlined in Park et al. (2023b), but we provide a simplified version.
14

---

Published as a conference paper at ICLR 2025
We first parameterize f(S. 2) (()) using S RD and vb 2 RD, which are 1-Lipschitz constrained functions. This safe decomposition because the expressive power of "(S)* (Z) is universal. Please refer to Park al. (2023b) for more detail.
Iw(S;
sup Stiills
(((s ( Exss) (
Then we use instead of (Z) to eam simplicity at the cost of expressiveness. Also we choose the prior distribution pl Z) to have zero-mean. This makes the second term ( [Z] to Finally, similar VIC, we measure WDM between the final state ST and the skill, insteaf of all states.
Tw(ST; 2) s sup (p(s)2)(((S+)" loll
[2| = because the distribution of the initial state So and p( ()) are mutually independent, and p(Z) has zero-mean. So subtracting from the right-hand side doesn't change the equation. Now we have the final objective:
Next, we utilize (p)()))(((5o)) = (
Iw(ST:2) s sup Bp(s.2)(o(ST)=2 (x(ss2)(6(s0)'
1
sup Ep(r.)
Σ (6(842))(($))))
1
Therefore, we can use RL with the reward function r(8,2,8') = (Ø(s') - o(s)) to maximize (w; Z).
C PROOF OF CLAIM 1
Our goal is to prove the following two claims.
(ar, y) Suff (1 <(v))l thang (2, v) Var, y S, [1((x) Φ(v)11 s a(z, v)
(ii) d(. y) a valid pseudometric.
We follow the proof of CSD Park et al. 2023a) similarly, but it has significant differences which will be covered in later of this section.
Proof (i) We begin with introducing P(x, y) as the set of all finite path from state or to state y, for Vz, y E S. Here, we only consider existing path (8o v. 81, 8T-1,87 8T y) P(r., y)- The term existing indicates that for every consecutive state pairs (s,, 5:+1) in p. 8:41 should be reachable from 8, More concretely.
V(s,, 81+1) E p, 3a A.t., P(s a, 81+1)
where, P a transition probability. Then, for a path p. we denote the sum the language distance along the path Duun (p), i.e. Duany (p) def Σ t= drang (St, 9141). Now can define d(xz, y) as follows:
inf p( Drane (p)
if # y if r =
d(x.v) v)
Now we are ready to move on to the main proof. We assume that V(x, y) E Sady. [1((x) 0(y) < drang (I., y). Also, note that Var, y S, both duan (Ir, x) and d (2r, y) drang (v. x)) holds
15

---

Published as a conference paper at ICLR 2025
trivially. We denote the optimal path p" (2. 87-119))) P(x,v) which satisfy d(x, y) = Dang (p"). Then,
Vx, y E S, |1((x) - x(v)) = 1(((x) - o(s)) + + o(s²) + (( - -
- (l((z) - o(si)) + ([o(s)) - Q(5;)) + ... + 11 - d(y)l
I daagg (I,, xi) (8,, 82) + dang -1 = d(x, v).
This concludes the proof of (i).
Proof of (ii) To show that d is a valid pseudometric, we need to prove following three conditions: 2, E S, d(x, x) = 0, d(x, y) = d(y, r), and d(æz, y) < d(x, 2) + d(z, v).
At first, given that Vx E S, dlang (z, z) 0, d(xr, x) = 0 holds trivially.
Secondly, given that Vz, y E S, dhang( = dlang (v. x) _ā(,,)) v) d(y, x)) also holds trivially.
For the triangle inequality of d.
Yx, y, E S, d(z,g)= = inf
D.(p)
pEP(a.)) inf MCM(aE))
D2(11) Ds(p2)
inf pIEl(a,;))
Dz(Pi) + inf
Dz(p2)
peEl(z.))
= d(z, x) + d(z,y).
This completes the proof of (ii).
Difference from CSD The primary distinction is that we focus exclusively the existing path E P(x, y) only. Here, we refer path "exists" between state x and only if state is reachable from within finite number of transitions. During the actual optimization process, encounter existing path only, so we believe this constraints align our theory more closely with the actual training scenario. Due to this specific focus, unlike from CSD, can no longer serves as lower bound for diang-
agents
Another notable difference is that we update only with the adjacent pairs of states. Thanks to our claim 1., we still can induce a valid pseudo-metric across all pairs of states within the state space. This approach aims to bridge the gap between our theoretical model and the training data used for model updates.
16

---

Published as a conference paper at ICLR 2025
D LATENT SPACE VISUALIZATION
Real world
Labent space
Resi world
Latant Space
12.0
4.0
110 clvv
3.
24
Sellelale m!
1.0
090
.2
AA
0.
(a)
(b)
Resl world
Latont space
Resl world
Latent space
2.0
oo
220 pm
12
310
13R
(c)
(d)
Latent space
Beal
space
1.8
1.
4.0
-04 63.07 p3g8
-0 oz
0.9
(c)
(I)
Figure 12: Each sub-figure corresponds to trajectory drawn from policy conditioned on different skills sampled randomly. In each subfigure. the left figure displays the actual paths of both the end-effector (blue) and the object (red) in the real world while the right figure illustrates the corresponding trajectory in latent space, generated using the learned & function. A black arrow indicates the skill vector used to produce the trajectory. The numbers indicate transition step in each of the episode.
As discussed in Section LGSD encourages the agent to spread out evenly in the latent space. The distance between two consecutive points the latent space is regulated by the language distance, drang- Fig. 12 demonstrates that the orientation of the agent's trajectory in latent space aligns well with the sampled skill vector. Since the skill vector is sampled isotropically. agent effectively spread in the latent space. Additionally, it is noteworthy that the distance between two consecutive points in the latent space increases especially when the cube is moved. On the other hand, when the cube stationary, there is no further extension in the latent space can be seen in the Fig, 12 (a). This indicates that our language distance diang serves as meaningful proxy for semantic distance.
17

---

Published as a conference paper at ICLR 2025
E FULL ALGORITHM OF LGSD
Algorithm Language Guided Skill Discovery
1: Initialize sxill-condiioond policy (aas, 2), representation function 6(s), prompt 4prmpp LLM function LLM,, language embedding model fembed, skill inference network vb, Lagrange multi- plier A, and data buffer D of epochs do 3: for j < 1 to # of episodes per epoch do
2: for
4:
Sample skill 2 ~ NO
5:
while episode not terminates do
6:
Sample action a ~ [ [8 2)
7:
Execute and receive s
8:
Query (. (prompu) to produce (s) and ldesc (s')
9:
Compute reward r = (0(s') o(s))² 2
10:
Compute daag (s, s') using eq. 2
11:
Compute embedding vector
Sembed (ldeec
Add (8, a, T, s',, diang s'), cs. 2 to buffer D
13:
end while
end for
15:
for {s, a, r, 8', diang(6 (8, 8),, Cs. =} in D do
16:
Update ø to maximize B (( [(((s)) [ A -iimii, (ing(s (8, [0(s) - (
17:
Update À to minimize (e,, [X min(e, ding( (8, 8²) - [1á(s) <('')
Update ı using PPO with reward
19:
Update to minimize Mean-Squared Error Error between (e.) and
20:
end for
21:
end for
18

---

Published as a conference paper at ICLR 2025
F IMPLEMENTATION DETAILS
F.1 DETAILS OF DOWNSTREAM TASKS
We used similar settings for the downstream tasks in the locomotion domain as those used in Metra single goal is sampled for every episode within range of - 50, 50], centered at the origin. The task considered complete when the agent reaches within radius of 3 units around the goal, at which point the agent receives reward of 1 and the episode terminates. For AntMultiooa
(Park et al. 2023b). For iettSingeeGoal
there are four sequential goals total. When the first goal is reached, the next goal is sampled randomly within range of [-7.5, 7.5], centered at the agent's 's current location. Reaching each goal gives a reward of
On the other hand, we have two differences from the task used in Metra. The first difference is in the termination condition. When the agent's torso height drops below the threshold height of 0.31 m, the episode is instantly terminated. This threshold is the default value used in the Ant environment from the official IsaacGymEnvs library. We believe this condition encourages the agent to walk rather than "roll", which often occurs when attempting to maximize all possible state dimensions without constraints. Another difference is that we use a higher-dimensional observation space, as detailed F.2.1 Our observation space has 39 dimensions, whereas the original Metra work has 26 dimensions, as it is based on the MuJoCo Ant environment.
We trained the low-level controller five times with different seeds for reporting final performance. We used 20,000 updated checkpoints from each training runs of low-level controller to train the high-level controller.
F.2
OBSERVATIONS AND HYPERPARAMETERS
F.2.1
OBSERVATIONS
Table 1: Ant Environment Observations
Name
Description
Dimension
Base position
x.y.z position of the robot's base velocity of robot's 's base in x.y.z direction
3
Base velocity
3
Base angvel
angular velocity of robot's base Yaw, Pitch, Roll of robot's 's base
3
Base rotation
3
Gravity projection
Vector indicates direction of the gravity
3
DOF position
Current angle of each DOF
8
DOF velocity
Angular velocity of each DOF
8
Previous action
Action executed in previous step
8
Sum
39
Table 2: Franka Manipulator Environment Observations
Name
Description
Dimension
Cube pos
x.y.z coordinate of the target cube Rotation of the target cube
Cube quat
EEF pos
x.y.z coordinate of robot arm's end-effector
EEF quat
Rotation of the Robot arm's 's end-effector
Gripper q
Gripper width
Sum
F.2.2 HYPERPARAMETERS
19

---

Published as a conference paper at ICLR 2025
Table 3: Hyperparameters of L.GSD
Name
Value
Learning rate
0.0001
Optimizer
Adam (iiggma & Ba
2014
Minibatch size
32768(Ant), (6384(Franka)
Horizon length
32
PPO clip threshold
0.2
PPO number of epochs
5
GAE (Schulman et 2015
0.95
Discount factor
0.99
Entropy coefficient
0.0001
Initial Lagrange coefficient À
300
Dim. of skill
((At), 3(Frnkaa)
Policy network
MLP with [256. 256, 128]
Activaion of π Representation function ø
E et al. 2015
MLP with [256, 256, 128]
Activaion of ø
ReLU
Skill inference network 1b Activaion of vi
MLP with [256, 256, 128]
ReLU
G PROMPTS USED FOR EACH OF THE EXPERIMENTS
In this section, share the prompts used for each of the experiments, and the examplary outputs from the LLM. One key detail is that we prompted LLM to finalize its sentence with a keyword of "Description: ", that can parse the final description out of whole generated sentences.
G.1 PROMPTS FOR ENFORCING ANT AGENT TO MOVE NORTHI
Input You will describe given robot 's state where the robot is moving on plane. The state of the robot as follows:
Robot's y, coordinates: [6.5, -0.5, -0.3] Robot's rotation: [0.9770, 0.1942, 0.0884, 0.0012]
If the y-coordinate is less than 0, consider the area south- ern. Conversely, if the y-coordinate of the robot is greater than equal to 0, consider the area as northern.
If the robot is in the northern area relative to the origin, pro- vide the robot's s [x, position. Please conclude your answer with the final description in the format: 'Description: The robot's x, position is _..
If the robot is in the southern area relative to the origin, just state that 'Description: The robot is located in the south- ern area.'
Output Since the y-coordinate of the robot is -0.5, which is less than 0, the robot is located the southern area.
Description: The robot is located in the southern area.
G.2 PROMPTS FOR ENFORCING FRANKA AGENT TO PUSH OBJECTS TOWARD NORTH
Input You will describe a scene where a robot arm is approaching object. The state of the robot arm is follows:
20

---

Published as a conference paper at ICLR 2025
Robot's end effector's y, coordinates: [6.5, -0.5, -0.3], Object's s x, y, 2 coordinates: [0, 1.0, 0.2], Origin's s x, y, 2 coordinates: [0, 0, 0J
If the object's y coordinate is less than 0, tell me 'Description: The object is lecated in southern area.'
If the object's y coordinate greater than or equal to 0, tell me the object's x, y coordinates. You words, but please conclude your answer with the final description in the format: 'Description: The object's x, y position is
any introductory
If the object's coordinates match the origin's, tell me the dis- tance between the robot's end effector and the object. Again, conclude with the final description in the format: 'Description: the distance
units.'
Outputl when object is moved In this scene, the object's s y coordinate is 1.0, which is greater than 0. Thus, the x and y coordinates of the object are and 1.0 respectively.
Description: The object's a x, y position is 0, 1.0.
Output2 When object not moved In this scene, we observe a robot's s end effector positioned at coordinates [6.5, -0.5, -0.3] aiming to approach an object. The object itself is found at the coordinates [0, 0, 0], which are identical to the origin's coordinates.
analyze this setup, we calculate the distance between the robot's end effector and the object. Using the Euclidean distance formula:
Distance = - (m (22 )1
= -0 6.5)2 + (0 (-0.5))
(0.3))
= -6.52 + 0.5 + 0.32
442.25 + 0.25 + 0.09
V42.59
6.53 units
Description: The distance is 6.53 units.
G.3 PROMPTS FOR FRANKA AGENT MANIPULATEES ORJECT IN GENERAL
Input You will describe scene where robot arm is approaching an object. The state of the robot arm is as follows:
Robot's end effector's
y, coordinates: [6.5, -0.5, -0.3], Object's a x, y, z coordinates: [0, 1.0, 0.2], Origin's s x, y, z
coordinates: [0, 0, 0]
the object's s coordinates differ from the origin's, tell me the object's x, coordinates. You can any introductory words, but please conclude your answer with the final description in the format: 'Description: The cube's y position is
If the object's s coordinates match the origin's, tell me the dis- tance between the robot's s end effector and the object. Again, conclude with the final description in the format: 'Description: the distance is units.'
21

---

Published as a conference paper at ICLR 2025
Output In the scene, robot arm's end effector is positioned coordinates [6.5, -0.5, -0.3], while an object is located at [0, 1.0, 0.2]. The origin of the coordinate system is at [0, 0, 0]. Since the object's coordinates do not match the origin's coordi- nates, we focus the object's s position relative to the origin.
Description: The object's a x, y position is [0, 1.0).
22


Reviews:
[{"review_id": "KZXcwTDEZT", "reviewer": "Reviewer_UwKy", "summary": "This paper proposed to incorporate semantic diversity in skill discovery.  The authors show that the proposed approach with language guidance from LLMs outperforms existing skill discovery baselines. The proposed approach is demonstrated in both locomotion and manipulation tasks.", "questions": "- In Fig. 4, the coverage for the manipulation task is worse than the locomotion task. More analysis or explanations would be necessary to understand the limitations of the proposed approach.\n- In Fig. 7, showing the coverage for LGSD in the whole space would be better. Although the proposed approach may have a larger coverage than the baselines, it is less evenly distributed compared to CSD. Is this always the case for different training runs? Any analysis or explanations would be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A theoretical proof of language-distance as a valid pseudometric is provided.\n- Both locomotion and manipulation tasks have been demonstrated, showing the effectiveness of the proposed approach.\n- The paper is well presented.", "weaknesses": "- More complex scenarios should be designed to make the use of LLMs necessary rather than simple positions.\n- The lack of real-world experiments makes the proposed approach less convincing.\n- A video showing the robot in action should be included to give readers an intuitive understanding of learning performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "An47mCOPKq", "reviewer": "Reviewer_XxQY", "summary": "Rather than to maximize state coverage, this work introduces LGSD to constrain skill discovery inside a semantically meaningful subspace, created by LLMs with crafted user prompts.", "questions": "- Is it correct that for each state $s_t$, the LLM will be called to generate a description in order to compute $d(s_{t-1}, s_t)$? If so, how fast is the training?\n- How will LGSD generalize to complex scenarios? Is it necessary for experts to compose (skill)prompts for each possible skill?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method tried to link skill states with their semantic meaning, resulting in meaningful and language-controllable skills.\n- LGSD provides a theoretical foundation for the language-distance metric as a pseudometric, detailing how it can approximate semantic diversity.", "weaknesses": "- The experimental scenarios are simple, in which the exampled prompts and semantically controlled spaces are easy to follow yet fail to demonstrate the generalizablity and scalability --- after all, the method relies much on the description of states. LGSD’s dependence on LLMs for real-time distance evaluation might limit scalability to complex, real-time environments.\n- As I understand, users have to provide specified \"skill constraints\"  (via prompts, such as \"move north\" etc.), then how can it still be called \"skill discovery\" since users are specifying skills?\n- The comparison with some baselines is somehow unfair since they lack the prior knowledge of users or any language embedding computation. A better comparison should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ddS6xWu4d4", "reviewer": "Reviewer_Dsd7", "summary": "“Language Guided Skill Discovery” proposes a method for learning diverse skills with unsupervised reinforcement learning by aligning skills with latent space transitions under a distance constraint. The novelty in the proposed method stems from the use of language distance as the constraint. This is done by a Large Language Model (LLM) which describes the robot state, and a language embedding model which encodes this description in a fixed real-valued vector where the distance between descriptions can be measured. The results show successful skill discovery on locomotion and robot manipulation environments.", "questions": "## Questions:\n1. What is the dimensionality of the skill-space here? Is there any benefit to having higher dimensions / are more skills learned?\n\n2. The performance of other skill discovery methods seems quite poor on downstream Ant tasks, which is surprising especially for LSD/CSD/METRA. Are the tasks or rewards defined here differently from their original papers?\n\n3. Is the LLM necessary given those prompts? Could these examples not be done just as well with simple logic gates? For example, prompt G.3 could be defined as “if object not at origin, return position of object, else return distance between end effector and object” without any LLM. \nI understand that this is more or less a proof of concept, but I think it would go a long way if you can show some examples where the “common knowledge” of LLMs can be properly utilized. \n\n\n4. I am curious if specifying just the object’s Cartesian position as the sole encoder observation under a Euclidean distance constraint (LSD) would yield similar results. I guess it might struggle more with exploration (no signal before the end-effector has collided with the object), but it would be interesting to see.\n\n\n5. I like the visualizations (and analysis) in Appendix D. As I understand, the language distance d_lang stays quite small when the object is at the origin, but increases when the object is pushed. Is there a specific reason why changes in the LLM output for the former (“The object’s x, y position is A,B”) results in smaller language distance than the latter (“The distance is X units.”)? On this note, it would be interesting to see how the language distance changes as a function of the robot state - is it more or less linear? \nSince the LLM output is constrained to the “’Description: The cube’s x, y position is ___ .’” and “’Description: the distance is ___ units.’”, I can see how the language distance could serve as a metric between the two behaviors, but would it really correspond to something much different than Euclidean distance on description-level? For example, “The distance is 4 units” vs “The distance is 6 units”.\n\n## Summary:\nOverall I think it’s a very well-written paper, with a good mathematical basis and satisfactory results. The results are promising, and I think there are good direction for building upon the methods, for example with VLMs. In my opinion, the proposed method has a lot of potential which is not fully utilized. My main concern is that the examples shown in the paper can be easily represented in a simpler way without an LLM. I can see that there are situations where this might not be the case, and showing these will make a much stronger argument for the proposed approach.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Interesting and novel contribution - while using LLMs is not rare, this is an interesting way of using them for encouraging skill discovery.\n\n- The proposed method is well-supported mathematically.\n\n- Good zero-shot language-specified goal tracking with the learned skill predictor network.", "weaknesses": "- Not fully convinced that LLMs are necessary for the proposed scenarios. Showing the performance on more complex situations where the semantic understanding of LLMs is necessary would make the case much stronger.\n\n- More qualitative evaluation (i.e. videos) would be good for evaluating the learned skills.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjb8Hgk6bs", "reviewer": "Reviewer_BQeq", "summary": "This paper presents a novel skill discovery framework called Language Guided SKill Discovery (LGSD). LGSD aims to maximize the semantic diversity of discovered skills by utilizing language guidance from LLMs. In LGSD, the interactions between human experts and LLMs are utilized for 1) constraining the skill search space into a semantically desired subspace, and 2) encouraging the agent to visit semantically diverse states within the subspace. Various experiments are conducted to demonstrate the capability of the LGSD framework to find more diverse and distinctive skills compared to previous methods.", "questions": "- Why enforcing $\\phi$ to be 1-Lipschitz continuous under $d_{lang}$? Is there any other way to constrain the difference in the latent space?\n- In Section 5.1. In FrankCube environment, the intended half plane is only for NSEW directions. Can we see arbitrary directions, e.g., north-east or south-west?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is pretty clear wrt the high-level topic and what the high-level intuitions are. The idea of incorporating LLMs to guide the skill discovery is novel and intuitive. Figure 2 is a very clear figure except one minor issue (see weakness).\n- By introducing the representation function \\phi which transform the state space s to latent space z, the skill discovery problem in the Euclidean space is transformed to an equivalent problem in the latent space. Figures in Appendix D show that identifying skills in the latent space is much easier than in the original space. Furthermore, by formulating the reward as $r(s, z, s^\\prime) = ( \\phi(s) - \\phi(s^\\prime))^Tz$, LGSD successfully transform the problem of finding diverse skills to a problem of maximizing the culmulative designed reward.\n- The section 2.1 is a great section that summarizes works related to unsupervised skill discovery.  Researchers might already read some related articles, but such a classification (definition)  as \"mutual information based \" and \"distance maximization\" approaches is still helpful to the whole community yet very accurate.\n- Experiments results show LGSD is effective and competitive in discovering semantically distict skills compared to other skill discovery baselines.", "weaknesses": "- Figure 1 shows a set of discovered semantically distinctive skills but the caption is not very informative. It could be better if description of each skill can be presented.\n- Figure 2 is very good. However, it is unlikely for reader to understand $d_{lang}(s, s^\\prime)$ without any additionally context. People may mistakenly treat this as some Euclidean distance between s and s^prime in the original space until they encounter equation (2). I would like to see some quick explanation that the $d_{lang}$ here happens in latent space, and this procedure requires LLMs.\n- Prompts are required in this work to interact with LLMs to acquire the semantic knowledge. The design of the prompt is critical to the direction or subspace of skill discovery. However, what if we only want to leverage the semantic knowledge from LLMs without setting up to much language prior? From my understanding, too much control from human means less diverse and improvising skills.\n- This is minor. In Line 319, it is unclear what is $s$ and $g$ without defining them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)