Do's and Don'ts: Learning Desirable Skills with Instruction Videos
Authors: Hyunseung Kim
Byungkun Lee
Hojoon Lee
Dongyoon Hwang
Donghu Kim
Jaegul Choo
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Skill Discovery
Primary area: reinforcement_learning

Abstract: Unsupervised skill discovery is a learning paradigm that aims to acquire diverse behaviors without explicit rewards. However, it faces challenges in learning complex behaviors and often leads to learning unsafe or undesirable behaviors. For instance, in various continuous control tasks, current unsupervised skill discovery methods succeed in learning basic locomotions like standing but struggle with learning more complex movements such as walking and running. Moreover, they may acquire unsafe behaviors like tripping and rolling or navigate to undesirable locations such as pitfalls or hazardous areas. In response, we present **DoDont** (Do’s and Dont’s), an instruction-based skill discovery algorithm composed of two stages. First, in instruction learning stage, DoDont leverages action-free instruction videos to train an instruction network to distinguish desirable transitions from undesirable ones. Then, in the skill learning stage, the instruction network adjusts the reward function of the skill discovery algorithm to weight the desired behaviors. 
Specifically, we integrate the instruction network into a distance-maximizing skill discovery algorithm, where the instruction network serves as the distance function. Empirically, with less than 8 instruction videos, DoDont effectively learns desirable behaviors and avoids undesirable ones across complex continuous control tasks. Code and videos are available at https://mynsng.github.io/dodont/

Do's and Don'ts: Learning Desirable Skills with Instruction Videos
Hyunseung Kim 1.2
Byungkun Leel
Hojoon Lee
Dongyoon Hwang!
Donghu Kim- KAIST, KRAFTON
Jacgul Choo
(mynang, byungkun. lee, joonleesky bawnnleeekyyodnpeteeeee godnpeter, quagmire, jchoo Jakaist. ac kr
Abstract
Unsupervised skill discovery is a learning paradigm that aims acquire diverse be- haviors without explicit rewards. However, faces challenges learning complex behaviors and often leads to learning unsafe or undesirable behaviors. For instance, in various continuous control tasks, current unsupervised skill discovery methods succeed in learning basic locomotions like standing but struggle with learning more complex movements such as walking and running. Moreover, they may acquire unsafe behaviors like tripping and rolling navigate undesirable locations such as pitfalls or hazardous areas. In response, we present DoDont (Do's s and Dont's s), instruction based skill discovery algorithm composed of two stages. First, in instruction learning stage, DoDont leverages action-free instruction videos to train instruction network distinguish desirable transitions from undesirable ones. Then, in the skill learning stage, the instruction network adjusts the reward function of the skill discovery algorithm to weight the desired behaviors. Specifically, we integrate the instruction network into distance-maximizinn skill discovery algo- rithm, where the instruction network serves the distance function. Empirically, with less than 8 instruction videos, DoDont effectively learns desirable behaviors and avoids undesirable ones across complex continuous control tasks. Code and videos are available at httph:://// /ttpss//myysng. gi thub. 1o/dooontt
Introduction
Recent advancements in unsupervised pre-training methodologies have led to the creation of large- scale foundational models across diverse domains, including computer vision 3 and natural language processing (2). These methodologies exploit self-supervised learning objectives to extract meaningful representations without using explicit, supervised learning signals. In attempts expand this paradigm to reinforcement learning, researchers have explored crafting self-supervised objectives develop foundational policies capable of learning diverse behaviors without predefined reward signals, termed unsupervised skill discovery (USD) [1499 43] 31, 27 35 38 (3).
Despite notable advancements in USD methodologies, acquiring foundational policies in environ- ments with large state and action spaces (e.g., multi-jointed quadrupeds) remains a significant challenge. Two major issues arise when training agents with USD in these complex environments. First, the vast state and action spaces allow the agent to acquire a wide variety of behaviors. While learning simple behaviors like standing may feasible, acquiring complex behaviors that require intricate joint movements, such as walking or running, can take an exceedingly long time, agents develop undesirable and risky behaviors during training, such as tripping, rolling, or navi- gating hazardous areas like pitfalls. These challenges raise a key question: Is the purely unsupervised assumption of USD ideal for learning foundational policies in the real-world?
Second,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(o) Offine dotoset with instructinn lobel
Case L:Avoiding fallures
Avoldiing hazordous areas
Case Avoiding prohibited behovlors
!!!
n!
Unssperrissd skill discovery
Dobonn
Trokk lndtrcction network usigg Des Donts dooser
K(AjK.))
iitilss cewerd
E
cepmee Seals
Lenozed
Figure 1: (a) The offiin instruction video dataset includes videos of desirable behaviors (Do's) and undesirable behaviors (Don'ts). (b) Unsupervised skill discovery algorithms tend to learn undesirable behaviors. (c) In DoDont, an instruction network is first trained with the Do's and Don't 'ts videos to distinguish desirable and undesirable behaviors, Then, this instruction network adjusts the intrinsic reward of the skill discovery algorithm, promoting desirable skills while avoiding undesirable ones.
According social learning theory, humans learn behaviors through both internal and external motivations [ Internally, they seek to perform diverse behaviors they haven't 't tried before. Externally, they adopt socially desirable behaviors and avoid those that are not. In the skill discovery algorithm, this can be mimicked by combining internal motivation (USD objective) with external motivation based on human's intention. A straightforward way incorporate external motivation in USD is to use a band-designed reward function 226 10]. However, hand-designing reward functions are not scalable for learning diverse and desirable behaviors for several reasons. One of the reasons the complexity of designing an intricate reward signal for guiding a desired behavior, and another is the difficulty of balancing multiple behavioral intentions into single reward function.
To address these challenges, we propose DoDont, skill discovery algorithm that integrates USD objectives with intended behavioral goals. Instead of relying on hand-designed reward, DoDont learns a reward function from a small set of instruction videos that demonstrate desirable and undesirable behaviors. Videos are chosen because they are inexpensive collect and do not require action or reward labels [13 [6).
As illustrated in Figure 1 DoDont starts by collecting instruction videos of desirable (Do's) and undesirable behaviors (Don'ts) We then train an instruction network which assigns higher values to desirable behaviors and lower values undesirable ones. This network re- re-weights the internal USD objective for the skill discovery phase. We utilize a maximmiinn skill discovery algorithm as
our main USD objective [35] 38 39 where the instruction network serves as the distance metric,
To validate DoDont, we conduct experiments on various continuous control tasks that require complex locomotion (c.g., Cheetah and Quadruped (45)) precise manipulation (e.g., Kitchen [17). Our results show that with fewer than eight instruction videos, DoDont effectively learns complex locomotion skills (c.g., running quadruped), which are challenging acquire with standard USD algorithms 33 Moreover, our instruction network effectively captures human intentions better than hand-designed reward functions in balancing multiple behavioral objectives. Additionally, DoDont learns diverse skills while avoiding undesirable and unsafe behaviors (e.g.. backflipping), which are difficult prevent even with previous skill discovery algorithms which utilize prior knowledge (26)..
Related work
2.1 Learning diverse behaviors without pre-defined task
Numerous unsupervised skill discovery (USD) methods have been developed to create founda- tional policies capable of learning a diverse behaviors without pre-defined task. The most com-

---

mon approach involves maximizing mutual information (Ml) between states and skills (T Z) = D (p(s, 2) (p(s)p(z))) 14 31 9 27. This is typically done by using an auxiliary neural net- work (=())) ()) to estimate a lower bound of [(; 2) (T(S; Z) 2 Ezz, slog go (2(s))). This network acts as a skill discriminator, predicting the skill 2 from the state 8, which encourages the policy to create distinct trajectories for different skills and promotes learning wide range of skills. However, these methods often struggle to learn complex, dynamic skills, as the MI objective can be met with simple,
static skills, leaving agents unmotivated to explore more intricate behaviors [44 23 30 31 24].
To overcome this problem, distance maximizing skill discovery (DSD) methods have been intro- duced [35 38 39]. Instead of maximizing MI between states and skills, DSD explicitly maximizes a
predefined distance function of states d' SXS + B For instance, LSD 331 uses Euclidean distance between states to encourage agents to move further distances (dlsn = I|s - s|l. CSD [88] employs density model based distance, promoting agents to visit less frequented states (desp = log go [s), where qo is the density model). METRA 39 defines the distance temporally, pushing agents to learn
skills that are temporally far apart (dMETRA = minimum number of environment steps reach s' from
8). DoDont is built upon the DSD framework, which will be elaborated in Sections 3 and
2.2 Learning diverse behaviors with pre-defined task
In contrast to USD, there is a line of research focused on learning policies that exhibit diverse behaviors by integrating intrinsic motivation (c.g., MI-based rewards) with human intention, as
the form of task-specific reward functions 10 26
55 54 or using demonstration datasets [40].
While traditional RL methods typically aim to find single optimal policy, these approaches aim to learn multiple policies To (a|s, 2==1:N). ). This allows for multiple ways to solve given tasks, resulting in increased robustness to environmental changes and the development of policies with distinct characteristics.
SMERL (26 and DGPO 10] combine intrinsic and task rewards, maximizing their sum when task rewards exceed a given threshold. RSPO (55] iteratively finds novel policies by switching between task rewards and intrinsic diversity rewards, ensuring each new policy is distinct from previous ones. DOMiNO (54) addresses constrained optimization problem by maximizing intrinsic diversity rewards while ensuring that all policies achieve sufficiently high performance (task rewards as constraints). ASE 40 utilizes a action-free demonstration dataset instead of a task reward function, minimizing a behavioral cloning loss while simultaneously maximizing diversity intrinsic reward. DoDont also employs action-free demonstration datasets extract human prior knowledge through an instruction network, pioneering the incorporation of both positive and negative behavior examples to guide learning towards desired behaviors and away from undesirable ones.
3 Preliminaries and problem setting
Markov decision process and skill-conditioned policy. Unsupervised skill discovery (USD) focuses on identifying a range skills without pre-defined reward function. In this approach, we use reward- free Markov Decision Process defined as M (S. μ., p). Here, S represents the state space, .A is the action space, i denotes the initial state distribution, and p is transition dynamics function. We utilize latent vector E Z and their associated policy π <[ass, 2). which we refer to as skills. To generate a skill trajectory (i.e., behavior), we first sample a skill a from the prior distribution, denoted as p(=), and then execute trajectory using the skill policy x (als, 2). For the skill prior, we utilize a standard normal distribution to represent continuous skills by following
previous works 35 38 39
Distance-maximizznng skill discovery. As discussed in Section 2.1 numerous distance- maxiumiiii
skill discovery methods have been proposed [35 38 39]. Specifically, DSD introduces the Wasserstein dependency measure (WDM) as a learning objective for unsupervised skill discovery (USD) (4
(w(S; Z) (V((ss)))
Here, W the I-Wasserstein distance on the metric space (S x Z,d), where is a distance metric. Unlike MI-based skill discovery methods (maximizing T(S; Z) = -rr.. (p(s, =) [[(s))(=))) which does not explicitly motivate the agent to learn more complex behaviors (as discussed Section 2. DSD maximizes the distance-aware (w(S; Z). By doing so, DSD not only discovers set of
3

---

distinguishable skills but also maximizes the distance d between different states. By defining the distance function to encourage desired behaviors, DSD effectively induces the agent learn these behaviors.
In practice, by leveraging the Santorovich-RRuennsee duality assumptions, the following concise objective can be derived:
48
under simplifying
T- Σ (0(s)1) ((s)))
(w Z)= sup (p(r,)) 1
where RD is function that maps states into D-dimensional space, | denotes the Lipschitz constant for the function & under the given distance metric d. Now, we can rewrite Equation 2 with arbitrary distance function d: S XxS [R as:
sup Bp(r,)) n
Σ (0(5(41) -
s.t. 1lo(x) 2(v)l2 S dd(,v), v(r,y) E. (3)
For detailed derivation, advise readers refer Section 4 of METRA 39 It's important to note that generic distance function, d, does not necessarily need to meet the criteria valid distance metric, such symmetry or the triangle inequality. As shown in previous work (38). the original constraint in Equation 3 can be implicitly converted into one with a valid pseudometric, allowing us use any non-negative function as distance metric.
4 Do's and don'ts (DoDont)
Our goal is to integrate human intention into USD, where diverse policies are learned without predefined reward signals. The core concept of this work involves training an instruction network using video data to distinguish desired and undesired behvaiors (Section Then, we set the trained skill discovery framework
instruction network the distance metric in the (Section[4.2 Consequently, this will result in an agent that not only learns diverse behaviors but also incorporate human intention since the instruction network is set as the distance metric.
4.1 Instruction network
We aim to train an instruction network that assigns high values for desirable behaviors (Do's) D and low values for undesirable behaviors (Don't 'ts). The distance metric in the WDM objective (in Equation 3) takes two states as imput and should predict a non-nest non-negative value.
To meet this requirement, we first prepare videos depicting desirable behaviors (i.e., Do's) and videos illustrating behaviors avoid (i.e., Don'ts) 's) according to our intentions. We then label our video dataset by assigning y = 1 to adjacent state pairs (st. 8t+1) in Do's videos and 0 to those in Don'ts videos. This results our Do's and Don't video dataset Dv which is comprised of triplets (st, $t41, y). After preparing our video dataset Dv, we use it to train instruction network p which is designed to classify whether given pair of adjacent states is from a Do's video or a Don'ts video. The training objective for the instruction network is simple binary classification loss:
clnstruction
E
(0) [y) log (8e, 81+1) + (1 - y) log(1 - (86, 8t+1))]
4.2 DoDont
Here, we integrate the learned instruction network into the online distance-maxe maxtimie- skill discovery algorithm. As P a non-negative function, we can directly use this instruction network as the distance metric of the WDM objective in the Equation as:
T-1
sup (T.2) F.6
Σ (0(5)+1)
2 ((s))
s.t. (('))l (o)), V(s,s') E Sodj (5)
where Sady represents the set of adjacent state pairs. By assigning higher values to behaviors deemed desirable by humans (i.e., large distances) and lower values to behaviors that should be avoided, guides the agent to learn skills that correspond to human intentions.

---

Then, following previous work [38 39 49), we can optimize Equation S with dual gradient descent, incorporating Lagrange multiplier and a small slack variable 0:
DoDont = (ó(s') - o(s))
J JDa = [[(((s') ((s) T 2] + X min((,,pp(s,, |0(()) o(s II)),
(6)
JDaDomtA := Emmin( (s, s') (1o(s) ((s))
where DoDeat is the intrinsic reward for the latent conditioned policy, and JDoDom, and J the objectives for & and A. We note that since the instruction network Pe is already trained on Dv, we keep Ps frozen for training stability.
However, one drawback of Equation 5 is that DoDont can reflect instruction signals from Pa. only after (s)) has been sufficiently trained through DoDDomm,,d This indicates that in the early stages training where ((s) has not yet been sufficiently optimized, the agent may not properly receive instruction signals. Such delay in receiving instruction signals could potentially slow down the training process. Therefore, to ensure the agent receives instruction signals regardless of the training state of ((8), we note that the following equation can be easily derived from Equation
T-1 Σ 1==
sup (p,,,)) 8
Dols, (0(#111))
s.t. [1b(s) - *6))l2 1 V(s,d') E Sudjs
For detailed derivation, please refer the Appendix Through this rephrasing, we now separate Py from the training of ((s). This allows the agent to receive direct instruction signal, as P explicitly multiplies with the original intrinsic reward, independent of the training of (0s). In our experiment, we found that this direct signal significantly enhances the agent's ability to learn diverse and desirable behaviors (Section 5.3). addition, another advantage of Equation []s that it can be interpreted simply multiplying instruction network Py. to the original learning objective function of METRA. This allows us to implement our algorithm by simply adding a single line code on top of the METRA framework. We provide the pseudocode of DoDont Algorithm [T Furthermore, our instruction network can be applied to zero-shot offline RL to learn diverse behaviors while prioritizing desirable behaviors within the offline unlabeled dataset. Detailed explanations and experiments are presented in Appendix
5 Experiments
5.1
Experimental setup
Our experiments aim evaluate the effectiveness of the DoDont method in learning desirable behaviors while avoiding undesirable ones across various environments and instruction scenarios. We test DoDont in two contexts: complex locomotion tasks (Cheetah and Quadruped envi- ronments from the DeepMind Control (DMC) Suite [45]
and precise manipulation tasks (Kitchen environment [17
Chectah
Quadruped
Kitchen
33 ) (Figure 2 For pixel-based inputs in the DMC envi- ronments, we colored the floors to help the agent be aware of its location from the pixel data, consistent with previous studies 36
Figure 2: Benchmark environments.
20
39
For quantitative comparisons in locomotion tasks, we use two main metries: state coverage and (ii) zero-shot task reward. State coverage is measured by counting the æ-axis coverage for Cheetah and x, y-axxii coverage for Quadruped achieved by the learned skills at each evaluation epoch, following prior literature (39. Zero-shot task reward evaluates the learned skills without task-specific training rewards to assess whether the agent efficiently learns or avoids specific behaviors. Our quantitative analysis uses four random seeds and provides 95% confidence intervals, represented by error bars or shaded areas. For each instruction scenario, we use eight videos (four "do" videos and four "don't" videos), unless otherwise specified. Additional details claborated in Appendix D
5

---

To comprchensively evaluate DoDont's performance, we compare it against three types of baselines. First, determine if instruction network facilitates acquiring intended behavior, we compare DoDont with METRA 39 state-of-the-ar online unsupervised skill discovery algorithm. Second, to assess the effectiveness our video-based intention network against hand-designed reward function, we introduce METRAÍ, modified version of METRA that uses hand-designed reward functions as the distance metric. Finally, to verify if using an instruction network as distance function effectively integrates our intention with skill discovery algorithms, compare it to SMERL [26] and DGPO [T)) which integrate task rewards in specialized ways. this comparison, we use our intention network the task reward function for both SMERL and DGPO, Details are in Appendix
5.2
Main experiment
In this section, our experiments are designed to answer the following research questions:
How efficiently does DoDont learn complex behaviors locomotion tasks? (Section 5.2.
Does DoDont learn diverse behaviors while avoiding hazardous areas? (Section 5.2.2
Can DoDont learn diverse behaviors without learning unsafe behaviors? (Section 5.2.3)
Can DoDont be applied to manipulation environment? (Section 5.2.4
5.2.1 How efficiently does DoDont learn complex behaviors in locomotion tasks?
METRAA
Cheetah
Quadruped
Quadruped
the Retum
State Conerase
Satte Coverge
52
10 (K10")
Coverage
zsot Reward
Coverage
Zero-shot
Learned skills
Figure 3: Left: State coverage and zero-shot task reward for Cheetah and Quadruped. Right: Visual- ization of Do videos in our instruction video dataset and learned skills by DoDont. We are able to observe that DoDont does simply mimic instruction videos but extracts desirable behaviors (e.g.. run) from the videos and learn diverse skills.
In this experiment, we aim to evaluate whether DoDont can effectively learn diverse complex behaviors by simply providing videos desirable but complex behaviors. accomplish this, we set videos of agents successfully running Do's ss and random action videos Don't 'ts for the Cheetah and Quadruped environment. As shown in Figure 3 DoDont achieves higher running rewards than METRA in both environments. Specifically, in the Quadruped environment, METRA primarily learns rolling movements for locomotion, while DoDont learns to run upright. Please refer to the project page videos for further details (link). Additionally, DoDont exhibits higher state coverage than METRA, since DoDont effectively learn running behaviors which allows the agent to cover longer distances whereas METRA only learn mediocre rolling behaviors.
We also observe that both SMERL and DGPO fail to learn diverse behaviors and only learns simple behaviors such standing upright at the starting point with minimal movement. This limitation is likely due to their use of the MI objective, which appears insufficient for promoting diverse behavior learning in complex continuous control environments. METRAÍ, variant of METRA where we set the run task reward as the distance metric, is effective for simple environments with low- dimensional action spaces (Cheetah), but struggle in learning effective behaviors for environments with high-diig gigh-dimnsiionll action spaces (Quadruped). We speculate that as the environment complexity increases, designing single reward function which captures variety of desirable behaviors becomes increasingly challenging.
An important point which we would like to emphasize is that DoDont only requires total of eight videos to learn running behaviors. Despite the direction each running video is limited (instruction videos in Figure 3), DoDont learns skill which run in all directions. This indicates that DoDont is not

---

merely imitating the behaviors in the Do's videos but discover a variety of behaviors which resemble those seen the Do's videos. In addition, to further evaluate the meaningfulness of the learned behaviors, we conduct downstream task evaluation for each method in the Appendix A
5.2.2 Does DoDont learn diverse behaviors while avoiding hazardous areas?
METRA
METRA
DGPO
SMERL
DoDont
Cheetah
Quadruped
METRA
METRAL
DoDont
Safe state coverage
ME TRA
METRA
DoDont
Safe state coverage
Figure 4: Visualization and comparison of learned skills. In both environments, the left side is hazardous and the right side safe. Safe state coverage assesses the agent's 's ability to cover safe areas and avoid hazards.
In practical scenarios, it is erucial for agents avoid hazardous areas. For instance, in navigation tasks, robots must avoid dangerous areas such as pitfalls or private property. However, traditional skill discovery methods, which aim cover the entire state space, face difficulties in controlling the regions an agent explores. We now aim to evaluate whether the Don't tt videos in DoDont can reliably prevent agents from learning unwanted area navigation. To replicate real-world conditions, we perform experiments where we designate the left side of the environment as hazardous areas and the right side as safe areas. Thus, we use videos that move to the right as Do's and videos that move to the left as Don'ts ts, aiming to direct the agent away from hazardous zones and towards safer regions.
To evaluate the agent's 's ability to cover safe areas while avoiding hazardous ones, introduced the concept of safe state coverage. This metric assigns a score of +1 for states in the safe area and -1 for states in the hazardous area. As illustrated in Figure METRA without prior knowledge attempts to cover the entire state space, including hazardous areas, whereas DoDont effectively incorporates human intentions, avoiding dangerous areas while adequately covering the safe regions. We also observe that SMERL and DGPO fail to learn diverse skills, resulting in static behavior within the safe region, possibly due the difficulty of learning diverse behaviors in pixel-based environments using the mutual information objective. Furthermore, METRA demonstrates superior performance in this experiment, likely because designing reward function that incorporates human intentions straightforward (assigning +1 for the safe region and 0 for the hazardous region).
5.2.3
Can DoDont learn diverse behaviors without learning unsafe behaviors?
In real-world scenarios, agents should also avoid learning risky behaviors. For in- stance, actions such as rolling on the
METRA
METRA
DGPO
SMERL
DoDont
Cheetah
Quadruped
ground or flipping upside down can po-
Zero-shot Run
Zero-shot Run
Zerorshot Roll
tentially damage the robot. Therefore, this experiment, we aim to investigate whether DoDont can learn variety of de-
Zero
sirable behaviors while avoiding certain unwanted behaviors. We trained our agent using running videos as desirable behav- iors (Do's) s) and rolling or flipping videos undesirable behaviors (Don' ts) for both Quadruped and Cheetah environments.
Figure 5: Learning safe and diverse behaviors. Zero- shot rewards assess how effectively each method learns desired behaviors while avoiding hazardous ones.
As shown in Figure 5 compared to METRA without any prior knowledge, DoDont effectively learns running behaviors while avoiding hazardous actions such as rolling or flipping using our instruction videos. Again, SMERL and DGPO fail to learn diverse behaviors, merely learning the simple behavior of standing upright at the starting point. For METRA!, we designed the reward function as Trmm Tilipp or mill to encourage running while avoiding flipping rolling. We observe that
METRA cannot fully encapsulate human intention in Cheetah environment. We believe this issue
7

---

arises from the entanglement of the task signals in the reward function which makes the agent difficult from discerning which type of behaviors are beneficial and which are not. To receive the run reward, the agent must move forward, but as the body moves, it sometimes triggers the flip reward, which penalizes the agent, hindering the agent from properly learning the run behavior. This illustrates that balancing multiple behavioral intentions is challenging, making the creation of hand-crafted reward functions difficult and unscalable.
5.2.4 Manipulation task
In this experiment, we aim to demonstrate that our inten- tion network can be applied to manipulation environment to learn multiple tasks. We use six target tasks following previous works [39 33 To effectively learn these six tasks, we use the D4RI. dataset as Do's and random ac- tion videos as Don't ts. As shown in Figure [ through our instruction network, DoDont successfully performs more tasks compared to METRA. SMERL and DGPO only learn two to three tasks, which we believe is due to the failure of the MI-based objective to capture diverse behaviors in pixel-basee environments. For METRA we use hand-
METRA
KETRA
DoDont
Policy Task Coverage
2 2.
0.00
0.25
0.50
0.75
1.00
1.25 1.50
designed reward function by summing the rewards for all six tasks. Nevertheless, METRA does not effectively learn the tasks, likely due to the entanglement of multiple behavioral signals within a single reward function.
Environment Steps (x10')
Figure Policy task coverage. The number of tasks successfully completed by sampled skills.
5.3
Ablation Studies
5.3.1 Model components
We conduct an ablation analysis to demonstrate the importance each component of our algorithm in achieving high performance. Specifically, experimented with three variants: (i) using only one Do's video, (ii) optimizing Equation 5 referred to as delayed reward, and (iii) focusing solely on encouraging Do's behaviors. We tested these variants in Quadruped environment, using run videos as Do's and random action videos as Don'ts.
Impact of data quantity. While we are al- ready training with relateively small amount of videos as our default setting (four Do's s,
DeDont
Delayed
video
Do's
Zero shot Run Returnn
four Don'ts), ts), we also conduct an ablation study where we use only two videos (one Do, one
State Coverage
300
12
203
Don't) to see how well our method performs un- der extremely limited data conditions. Accord- ing to Figure 7 even with only one Do's video, the agent can mimic the running behavior, as evidenced by the zero-shot run reward. Remark- ably, despite being exposed to just one video showing movement in a single direction, the agent is still able learn movements in multiple directions; please refer to the project page videos (link).
100
00
03
1.
03
0.5
3.0
1.5
Environment Steps (x10')
Environment Steps (x10")
Figure 7: Ablation study on model components.
Direct reward signal. To evaluate the effectiveness of Equation 7 we trained an agent using Equation [5] As shown in Figure [7] the direct instruction signal effectively captures the desired behavior from the early stages of training. In contrast, the delayed instruction signal initially fails capture the desired behavior, evidenced by drop in the zero-shot reward at the beginning. Consequently, the agent takes a long time to overcome hefore it can stand up again.
Only encouraging Do's behaviors. To further analyze the importance of penalizing Don't 's be- haviors, we trained an agent only to encourage Do's behaviors. As illustrated in Figure 7 solely encouraging Do's 's behaviors fails to capture desirable behaviors. We speculate that without penalizing unwanted behaviors, the agent cannot avoid these undesirable actions, leading to a drop in the zero-
8

---

shot Run return at the beginning. Consequently, it takes a significant amount of time for the agent to learn the desired behaviors. Further details of this experiment Appendix C
5.3.2 The importance of utilizing instruction network as the distance metric
Cheetah
Prefored State Coverage e
Addiiim
METRA
Additive
Additive
Additive
Multiplicative
Visualization of acquired skills, with the hazardous zone on the left and the safe zone on the right. Right: Quantitative comparison of each method.
Figure 8: Left:
In this study, we use the instruction network as the distance metrie for the distance-maximizing skill discovery algorithm [39] This setup is convenient because it only requires multiplying the instruction network with the intrinsic USD reward to train the policy. However, an alternative approach is to add the output of the instruction network with the intrinsic USD reward, similar to SMERL [26] and DGPO [IO]. To compare these two approaches (multiplication and addition), we conducted experiments in the Cheetah environment following the setup from Section 5.2.2
Our qualitative results, shown in Figure indicate that none of the additive variants (Fioral "MMTRA ap (s, s')) match the performance of the multiplicative approach (Frotal = "MEETA (s. We attribute this the fact that multiplying (s, with the intrinsic reward function controls the scale of the total reward effectively since it directly impacts the influence of "METRA while adding (s, s') indirectly impacts the influence of TMETRA- Although additive variants might achieve similar performance, they would likely require careful tuning of coefficients to balance the multiple reward components.
Conclusions, limitations, and future work
In this paper, we introduce DoDont, an instruction-based skill discovery algorithm designed to combine human intention with unsupervised skill discovery. We empirically demonstrate that our instruction network enables the agent to effectively learn desirable behaviors and avoid undesirable ones across a range realistic instruction scenarios.
However, our study faces certain limitations. Specifically, in this research, we trained the instruction network using in-domain video data, which is not a readily available resource real-world applica- tions. We propose that training the instruction model with general, in-the-wild video data represents scalable and compelling direction for future investigation.
Despite these constraints, DoDont exhibits promising results even with a limited video dataset. Moreover, believe that videos are a cost-effective form of data for representing behaviors as they do not require action and reward labels. Thus, preparing data is both practical and feasible. On the other hand, as video generation models advance, several recent works have utilized these models as simulators [50, 51 [2.00 [). There is potential to generate Do's 's and Don't 'ts videos with these models, which could allow for the use of generated data without the need for additional data collection.
As final remarks, in unsupervised skill discovery, there are many excellent prior works that have succeeded in learning diverse behaviors without any supervision, data, or prior knowledge. However, these methods often struggle to acquire complex behaviors and may learn hazardous behaviors, making them unsuitable for direct real-world application. Therefore, we believe that the next step in unsupervised reinforcement learning focuses on leveraging minimal supervision or data (i.e., resources that are easily obtainable the real world) become scalable and applicable in complex control tasks and practical for the real world.

---

Acknowledgments
We are grateful Seohong Park for providing insightful discussions and valuable feedback. This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea governmen((MSTT) School Program(k, (AASST), and the National Research Foundation of Korea (NRF) grant funded by the Korea govermment (MSIT) (No. A
R-R--00--II199000 Artificial Intelligence Graduate
References
[1] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10299, 2018.
[2) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
[3] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 15:23716-237555 2022.
[4] Albert Bandura. Influence of models' reinforcement contingencies on the acquisition of imitative responses. Journal of Personality and Social Psychology, 1965.
[5] Albert Bandura, Ross Ross, Dorothea, and Sheila A. Transmission of aggression through imitation of aggressive models. The Journal of Abnormal and Social Psychology, 1961.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[8] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 181..1289..... 2018.
[9] Víctor Campos. Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference Machine Learning, pages 1317-1327. PMLR, 2020.
[10] Wentse Chen, Shiyu Huang, Yuan Chiang, Tim Pearce, Wei-Wei Tu, Ting Chen, and Jun Zhu. Dgpo: discovering multiple strategies with diversity-guided policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1990-1999,, 2024.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1 113, 2023.
[12] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[13) Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[14] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. Proc. the International Conference on Learning Representations (ICLR), 2018
10

---

[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[16]
Anthony G Greenwald and Stuart M Albert. Observational learning: A technique for elucidating s-r mediation processes. Journal of Experimental Psychology, 1968.
[17] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving Long-horizon tasks via imitation and reinforcement learning. arXiv preprint 1910. 2019.
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMIR, 2018.
[19] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini arXiv: 18/20099 2018.
[20] Danijar Hafner, Kuang-Huce Mueng Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. Advances in Neural Information Processing Systems, 553:6691-261000 2022.
[21] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. Proc. the International Conference Learning Representations (ICLR), 2019.
[22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
[23] PeereeAlexandree Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Di- rect then diffuus: Incremental unsupervised skill discovery for state covering and goal reaching. Proc. the International Conference on Learning Representations (ICLR), 2021.
[24] Hyunseung Kim, Byung Kun Lee, Hojoon Lee, Dongyoon Hwang, Sejik Park, Kyushik Min, and Jaegul Choo. Learning to discover skills through guidance. Advances in Neural Information Processing Systems, 36, 2024.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[26] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198- -2200, 2020.
[27] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
(28) Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541- 551, 1989.
[29] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficien interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2/06.05091, 2021.
[30] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. arXiv preprint 1rXiv1196.005244 2019.
[31] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proc. the International Conference on Machine Learning (ICML), 2021.
[32] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems,
2021.

---

[33] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379- 24391, 2021.
[34] Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019.
[35] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. Proc. the International Conference on Learning Representations (ICLR), 2022.
[36] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
[37] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representa- tions. arXiv preprint arXiv:2402.155577 2024.
[38] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabilly--awaae unsuper- vised skill discovery. arXiv preprint arXiv:2302.05103 2023.
[39] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric- aware abstraction. arXiv preprint arXiv:2310.08887, 2023.
[40] Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics, 41 (4):1117. July 2022.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[43] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. Proc. the International Conference on Learning Representations (ICLR), 2019.
[44] DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. Proc. the International Conference on Learning Representations (ICLR), 2022.
[45] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 006900 2018
[46] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint 1807 2018.
[47] Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2022.
[48] Cédric Villani. Optimal transport: Old and new. 2008
[49] Tongzhou Wang, Antonio Torralba, Phillip Isola. and Amy Zhang, Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 1-34430. PMLR, 2023.
[50] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-worke simulators. arXiv preprint arXiv:2310.06/1 2023.
12

---

[51] Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprini arXiv:2402.17139, 2024.
[52] Denis Yarats, David Brandffonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't 't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201. 13425, 2022.
[53] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
[54] Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. Proc. the International Conference on Learning Representations (ICLR), 2022.
[55] Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. Proc, the International Conference on Learning Representations (ICLR), 2022.
13

---

Appendix
A Additional Experiments
15
Improving zero-shot offline RI. with instruction videos
15
A.2 Downstream task performance
15
Derivation of equation]
16
Implementation Details of DoDont
16
Data curation
16
Instruction network
16
Skill learning
17
4 Pseudocode of DoDont
17
Experimental details
17
Environments
17
D.2
Implementation details of baseline Agorithms
19
D.3 License
19
F ull results
20
Zero-Shof RL
20
Broader impact
20
14

---

A Additional Experiments
A.1 Improving zero-shot offline RL with instruction videos
Recently, HILP 337 has introduced novel framework for pre-training generalist policies that learn diverse behaviors from task-agnostic unlabeled datasets [52]. HILP can be considered as an offline variant of METRA, as both methodologies focus on learning temporal distance- preseeving representations to facilitate the acquisition of diverse behaviors. Similar to our previous application in METRA, we can easily integrate our instruction network into HILP's intrinsic reward function. The pseudocode for the offlin version of DoDont provided in Algorithm 2 our instruction network, the policy is trained to learn diverse behaviors while prioritizing desirable behaviors and avoiding hazardous ones within the offline dataset.
Intuitively, by utilizing
In this experiment, we further investigate the effectiveness of our method when applied
offline zero-shot rein- forcement learning (RL) approach that learns diverse be- haviors from task-agnostic, unlabeled datasets. Zero-shot RL is divided into two phases: reward-free fweedd-fee pretraining with an offline dataset and maximizing an arbitrary reward function provided at test time without additional train- ing. We primarily compared our results with HILP (37], FB 47 and FDM [37 477, the leading methods in offline zero-shot RI. We conducted experiments four environ- ments (Walker, Cheetah, Quadruped, and Jaco) using two EXORL datasets [52] collected by APS [31] and RND [8]. We followed the experimental protocol outlined in the HILP paper (37]. where detailed information available
EXORL Overall Performance
FDM
FB
HILP
DoDont
Figure 9: Aggregated performance. The overall results are aggregated over 4 environments, 4 tasks, datasets, and seeds (i.e., 128 values in total).
in Appendix D
During the pre-training phase, we employed four downstream task videos Do's and four ran- dom action videos as Don'ts 'ts for each environment. This strategy aims to prioritize the learning of task-relevant behaviors. We use the interquartile mean (IQM) 2 for overall aggregation to assess performance. As illustrated in Figure 9 DoDont is superior over other methods by large margin in effectively capturing downstream task behaviors from large, aas-aagostti,, unlabeled dataset. We would like to note that although DoDont benefits from downstream task relevant data, it significantly outperforms baselines while utilizing only a small set instruction videos. Detailed results are in Appendix B
A.2 Downstream task performance
METRA
METRAT
DGPO
SMEAL
OuDont
In this section, we employed hierarchical con- troller that selects (frozen) learned skills to max- imize the downstream task reward to evaluate the meaningfulness of the learned behaviors. As depicted in Figure I0, behaviors learned through DoDont showed higher performance than ME- TRA, indicating that DoDont effectively learned useful behaviors that be applied to down- stream tasks.
Cheetah
Quadruped
Downstream Return
Downstream Return
Envirommart Stops (H10%)
Stops (K30))
Figure 10: Downstream task performance.
15

---

B Derivation of equation 7
We first start with Eq.
sup B((a))
(0(s++)) ((s))) 2 s.t. (|o()) $p,ss)s), V(s,s') Sodj. (8)
Let scaled state function o(s)
(e(a, Then, we can transform the constraint term in Eq. 8 as follows (fu(s, 2 since it is the output value of binary classification network),
o(a)
sup E p(r,2) 9
Σ (6(s++)) - ((s)))
s.t. llo(s) o(s')l2 11, V(s,s') )') E Sudf.
(9)
By replacing ((s) with Ø(s) P (s, s') in Eq. 9 we derive
Σ (s se+1) -((s))))((ss))
sup E ((r.2) a
s.t. 1, ((s, E Sudj.
(10)
C Implementation Details of DoDont
Here, we provide further details regarding the implementation of DoDont. Our experiments are based on two different codebases: METRA A [39] for online learning and HILF (37 for offline learning.
Our experiments run on NVIDIA RTX 3090 GPUs, with each run taking no more than 28 hours.
C.1 Data curation
We now explain how the action-free Do's and Don't video dataset we used as our human intention dataset were collected. For DMC tasks, we first trained expert Soft Actor-Critic (SAC) [8]] policies based on proprioceptive states with the ground truth task reward for over 2 million environment steps and generate Do vidcos by deploying the expert policy the environment and saving the visual observations rendered by the simulator. In addition, we set the ExORL dataset [52] as our Don't ' videos for DMC tasks since the ExORL dataset is mainly composed of transitions generated via undesirable random behavior. On the other hand for the Kitchen environment, we set the the D4RL dataset [15] as our Do's video dataset and additionally gather videos of random action rollouts and set them as our Don't 't videos. The number of videos used for each environment can be found in Table []
Table 1: The number of videos used for each environment.
Environment
# of Do's videos
# of Don'ts videos
Online DMC
4 of videos in D4RL dataset 12 videos per task
4 10
Kitchen
Offline Zero-shot DMC
# of videos in ExORL dataset
C.2 Instruction network
For the instruction network, connected layers for the prediction head. To improve stability, we bound the output using the Tanh function, following previous work [29]. Additionally, employ simple random shift augmentation, a widely used data augmentation technique. The Adam optimizer (5] is employed with a learning rate of x 10-4 and batch size of 1024.
four convolutional layers for the backbone and three fully
http:///htthh.cooocccooooag//// Bhapss//gittubbcco//ccooonggggi
16

---

C.3 Skill learning
We implemented DoDont based on two algorithms: METRA (39 and HILPP [37] We used the hyperparameters from these algorithms without modification. Additionally, DoDont introduces only one extra hyperparameter, which is the coefficient for the instruction network. The complete list of hyperparameters can be found in Table 2
C.4 Pseudocode of DoDont
We present the pseudocode for the online version of DoDont in Algorithm | and the offline version of DoDont in Algorithm
Algorithm 1: Do's and Don'ts ts (Online)
Initialize instruction network Pis (8, s'), video dataset Dv
2 for number of epochs do Sample batch (a "o0,a".,3 ,,,y) from video dataset Dv
Update (8, 8') to minimize Clastruction in (4)
Initialize policy π <alss 2), representation function @(s), Lagrange multiplier X, replay buffer D
for number of policy epochs do
for number of episodes per epochs do
Sample skill 2 p(z)
Sample trajectory with (a]s, 2) and add to replay buffer D
10
A min(e, 1 (1(()) - o(s)) Update to minimize (( [X min(c, 1 (o(s) o(s')
Update d(s) to maximize (> (d)(') <(s))
= T
11
Update (als, z) using SAC with reward r(s, 2,8) (s. s'). (o(s') - o(s))
Algorithm 2: Do's and Don'ts (Offline)
Initialize representation (ss) Initialize policy π (a]s,, z)
3 while not converged do
Sample (s, .'.') D
Train ((s)
Train instruction network (s,
7 while not converged do
Sample (8, a, 8') x')== ~ D and : ~ gD-1 Compute intrinsic reward 25))
10
Train π (a]s,,)) 2) with weighted reward P (8, 8 2)(8,,88)
D Experimental details
D.1 Environments
Benchmark environments. In this work, we perform experiments with the Cheetah and Quadruped
environment from DMC (45) and pixel-based version of the Kitchen environment [17 33 Follow-
ing previous studies 339 36 20], for the pixel-based DMC environments
gradient-colored floors which allows the agent to infer its location from pixels. For the Kitchen environment, we utilize
the same camera settings LEXA
33].
we use state information as input for both the policy and the critic. For Sections 5.2.2 and 5.2.4 we use pixel data as input for the policy and the critic. The reason for this experimental setup that in pixel-based environments with gradient-colored floors, the discriminator might exploit the background color as shortcut to distinguish between different states rather than observing the agent's embodiment. This makes it challenging to incorporate behavioral intentions. However, without colored floors, METRA cannot learn diverse behaviors. Therefore,
For Sections 5.2.1
5.2.3 and 5..3
17

---

Table 2: Hyperparameters used in Online DoDont. We adopt default hyperparameters from METRA 39. introducing only one additional hyperparameter.
Hyperparameter
Value
Learning rate
0.0001
Optimizer
Adam [25
episodes per epoch
8
Encoder
CNN 28
hidden layers
# hidden units per layer Minibatch size
1024
256
Target network smoothing coefficient
0.995
50 (state-based DMC), 200 (pixel-based DMC).
gradient steps per epoch
100 (Kitchen)
10 (state-based DMC), 10 (pixel-based DMC),
Replay buffer size
10 (Kitchen)
Entropy coefficient
auto-adjust [9 (DMC), 0.01 (Kitchen)
0.001
initial
30
Latent dimension
4 (continuous) (DMC), 24 (discrete) (Kitchen)
Instruction network coefficient
Table 3: Hyperparameters used in Offline DoDont. We adopt default hyperparameters from
HILP (37), introducing only one additional hyperparameter.
Hyperparameter
Value
# gradient steps
10
Learning rate
0.0005 (ø). (others)
Optimizer
Adam
25
Minibatch size
1024
MLP dimensions
(512, 512) (0), (1024, 1024, 1024) (others) 0.99
TD3 target smoothing coefficient
Latent dimension
50
# state samples for latent vector inference
10000
Successor feature loss
Q loss on (Quadruped, Jaco), vector loss (others)
Hilbert discount factor
0.96 (Walker), 0.98 (others)
Hilbert expectile
0.5
Hilbert target smoothing coefficient
0.995
Instruction network coefficient
(Walker), 2 (others)
5.3 where behavioral intentions are included, we use state-based environments and non-colored default pixel human intentions. Notably, all experiments use videos (i.c., pixel data) as a human intentions.
for Sections 5.2.
5.2.3
For offline zero-shot RL, we use four environments (Walker, Cheetah, Quadruped, and Jaco) and two different ExORL datasets 52 (APS BII, RND K). Although experiments in the HILP Bi7
paper is based on four EXORL datasets (APS BTI RND 8,, Proto [53), APT (32)), due to limited computational resources, we performed experiments only with the APS and RND ExoRL dataset since these two dataset are where HILP achieved the best performance.
Metrics. For the state coverage in DMC, we count the number of 1x y bins occupied by any of the target trajectories for Quadruped and 1-sized bins for the Cheetah. In the Kitchen environment, we count the number of predefined tasks achieved by any of the target trajectories. We use the same six predefined tasks LEXA 33 and METRA [39]: Kettle (K), Microwave (M),
18

---

Light Switch (LS), Hinge Cabinet (HC), Slide Cabinet (SC), and Bottom Burner (BB). Policy state coverage is computed using 48 deterministic trajectories with 48 randomly sampled skills at the current epoch.
For offiin zero-shot RL performance, follow the method outlined in HILP (37 We sample small number of (s, a, s') tuples from the dataset, compute the optimal latent x with respect to the test-time reward function, execute the corresponding policy a(a|s, x*) and compute the return value. We use all the hyperparameters from HILP without modification.
Downstream tasks. We utilize two downstream tasks: QuadrupedGoal and CheetahGoal, primarily following METRA [39]. For each task, the objective is to reach a target spot randomly sampled from [-7.5, 7.5 for QuadrupedGoal and [-10.0, 10.0]² for CheetahGoal. The agent receives a reward of 10 upon reaching within a radius of 3 units from the target spot. We train a hierarchical high-level controller on top of the frozen skill policy. The high-level controller selects a skill for every = 50 environment steps, and the pre-trained skill policy x [alss, 2) executes the same skill for those K steps. We use PPO 42 for discrete skills and SAC [8] for continuous skills, maintaining all hyperparameters as in METRA 39
upon
D.2 Implementation details of baseline algorithms
In this work, we use four baselines: METRA [39], METRA', DGPO [10), and SMERL 266 For METRA, we used the open-source METRA codebase and retained the original hyperparameters. METRA is a variant of METRA that incorporates hand-designed reward functions as a distance metric. As described in the main paper, we manually designed the reward function and rescaled it to be greater than zero since it is used as a distance metric. Like the DoDont method, we multiplied METRA's intrinsic reward by this hand-designed reward function.
d DGPO introduces intrinsic reward function مinر min log +g( With this intrinsic reward, we use our instruction network as human intention. Total reward function for DGPO is g(s]e')
r = Pr (8, 5') + a ming' = log {(27))))))))
where ar is the balancing coefficient.
Like DGPO, we employed our instruction network as a human intention. To train SMERL, we first trained a baseline SAC agent with our instruction network and considered the maximum return achieved by the trained SAC agent as the optimal return RM (TT)) ). We used the same F = (2(s)) as described in the original paper. The total reward function for SMERL is (8, 8') + a (an(R))2RR((().) Io (2[s), where a the balancing coefficient.
SMERL uses intrinsic reward defined as ri 1 (x)
In Section 5.3
we introduce "Only Do's," a variant our algorithm that exclusively promotes desired behaviors. We achieve this by eliminating the penalties for undesirable behaviors. Specifically, we define Paus (s, s') as follows:
s')
if ps s') 2 0.5 if (f 8') < 0.5
Pdus (8,5') 5')
(11)
0.5
This approach ensures that the probability Poos (s. s¹) never falls below 0.5, thereby encouraging only the desired behaviors.
only
D.3
License
Environment
Deepmind Control suite: We used the publicly available environment code in
39
45
We used the publicly available avviromment code in
Kitchen:
39
15
Dataset
EXORL:
a We used the publicly available dataset code in 2
-
httpa...
itt
19

---

E
Full results
E.1
Zero-shot RL
Table Full results on the zero-shot RL performance. The table shows the zero-shot RL perfor- mance averaged over four seeds in each setting. We adopted the results from HILP 37
Dataset
Environment
Task
FDM
FB
HILPP
DoDont
Flip
4261 68
334 178
573 1 37
506 70
Run
248 1: 23
388 27
348 1
355 46
Walker
Stand
865 + 77
824 + 54
883 + 42
953 22
Walk
634 + 91
842 + 105
862 + 31
930 + 34
Run Backward
116 = 116
250 + 135
373 + 72
383 - 19
Run
360
316 21
290 44
Cheetah
Walk
396 + 287
683 267
939 + 55
984 +
APS
Walk Backward
982 +
980 + 3
985 2
985
Jump
707 30
757 +
623 - 149
721 30
Run
481 1
474 33
411 1 62
489 1 6
Quadruped
Stand
961 1 20
949 1. 30
797 1 117
965 14
Walk
578 145
583 1 12
605 1 75
568 1 55
Reach Bottom Left
12 18
14 1 12
88 + 41
61 + 17
Reach Bottom Right
28 - 20
24 77
48 + 24
Jaco
Reach Top Left
21 16
23 17
49 +
88 31
Reach Top Right
34 40
17
51 32
64
Flip
415+ 20
548 + 94
563+ 136
644
Run
295 = 70
409 15
401 + 30
436 - 22
Walker
Stand
821 1: 84
866 1 120
800 1 61
894 31
Walk
476 259
811 1 52
855 34
872 1: 32
Run Backward
107 + 26
183 + 83
262 + 53
257 + 6
Run
177 = 62
153 153
187 5
226 10
Cheetah
Walk
494 + 122
636 291
823 - 141
944 + 15
RND
Walk Backward
652
677
843
955 1
Jump
758
642 +
556 + 101
684 16
Run
491 + 7
436 26
393 - 42
494 - 35
Quadruped
Stand
971 = 11
797 72
810 97
859 79
Walk
601 = 82
642 + 202
542 32
753 58
Reach Bottom Left
53 20
18 + 11
19 20
33 1 14
Reach Bottom Right
38 + 14
29 + 12
18 + 17
29 - 31
Jaco
Reach Top Left
36 19
45 1 16
8
33 15
Reach Top Right
44 24
22
5
9
F
Broader impact
Our research has a broader impact on the safety and reliability of unsupervised skill discovery (USD) algorithms. By incorporating instruction-bare learning to differentiate between desirable and undesirable behaviors, such as tripping or navigating hazardous areas. This improvement enhances the practical usability of USD in real-world scenarios where safety is critical, especially in applications like autonomous vehicles and robotic manipulation. However, a major issue with USD remains sample inefficiency due to the need for extensive simulations.
method significantly lowers the risk of agents developing unsafe behaviors,
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately reflect the paper's contribution and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitation and future work in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
Yes, we provide the full set of assumptions and complete proof in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes)
Justification:
We provide pseudocode and hyperparameter list in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We provide our code link in Appendix C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiipe.cc/pulli//uuiii http:p/bll.//giiiidddddeeeeetiii CodeeSubeeiieeeeeeeeee ssionn ceiiionPoicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training and test details Appendix C
D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars in all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide our GPU resources, and time of execution in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaappp.opoo//////ccaaaaa
Answer: [Yss]
Justification: We conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our paper in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models) used the paper are properly credited, and the license and terms of are explicitly mentioned and properly respected in Appendix D.3
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcorsxithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do involve crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "ZznxVY8O8x", "reviewer": "Reviewer_AtB3", "summary": "This paper proposes a method, DoDont, to avoid hand-crafting reward functions in unsupervised skill discovery. DoDont first learns a reward function from labelled instruction videos that discriminates desired and undesired behaviors, and then use the reward function in unsupervised skill discovery. The authors evaluate the method in several experimental settings and find DoDont can learn more diverse and safer skills than the baselines.", "questions": "1. According to Appendix C.1, the Do videos for DMC tasks is collected from a policy that's trained on ground truth reward functions. Does this mean DoDont still needs hand-crafting \"ground-truth\" rewards for a new task? If so then DoDont is not solving the motivating problem of USD that it needs hand-crafted rewards.\n2. From the same section, it seems the Don't videos are collected from random action rollouts. Will these be enough for learning a reward that can teach the model to avoid hazardous behaviors? For example, walking into a hole is a bad behavior, but random action rollouts probably won't touch this kind of trajectory because random actions won't make an agent walk in the first place. Therefore, the reward module hasn't seen such bad behaviors during training and probably can't learn to assign a low reward value to this behavior.", "limitations": "See above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clear motivation that hand-crafting rewards in unsupervised skill discovery is tedious. The proposed method makes sense and is presented fairly clearly.", "weaknesses": "I think the major weakness here is using a implicit reward function instead of explicit hand-crafted rewards has been explored before while those baselines are missing in the paper. DoDont learn the reward on some labelled instruction videos. There are several other approaches of automatic reward design in the past, for example, using a LLM [1, 2] or a vision-language model [3], which the authors should compare DoDont to and explain the advantages of DoDont.  \n\n[1] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\" arXiv preprint arXiv:2310.12931 (2023).\n\n[2] Kwon, Minae, et al. \"Reward design with language models.\" arXiv preprint arXiv:2303.00001 (2023).\n\n[3] Fan, Linxi, et al. \"Minedojo: Building open-ended embodied agents with internet-scale knowledge.\" Advances in Neural Information Processing Systems 35 (2022): 18343-18362.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer AtB3,\n\nThank you for your constructive comments. We have provided a detailed response to your comments below. Please let us know if you have further questions or feedbacks.\n\n>**Question 1.1**\nI think the major weakness here is using a implicit reward function instead of explicit hand-crafted rewards [...], using a LLM [1, 2] or a vision-language model [3], which the authors should compare DoDont to and explain the advantages of DoDont.\n\nWe are grateful for your constructive feedback and the chance to elucidate aspects of our research.\n\nWe apologize for any ambiguity in defining the scope and objectives of our study. \nWe would like to emphasize the distinct scope and objectives of our research in comparison to the studies [1,2,3].\nIn our study, DoDont addresses the challenges inherent in unsupervised skill discovery (USD), specifically the inefficiencies and potential hazards of learning undesired behaviors. Our approach leverages instruction videos to guide the discovery of **diverse, desirable, and safe behaviors**, by applying *distance metric* learned from instruction videos to distance-maximizing skill discovery. \n\nIt's important to note that our approach is distinct from the objectives of the studies cited in the review [1, 2, 3], which predominantly focus on crafting a reward function for a **single** task using LLMs or VLMs. As such, a direct empirical comparison might not be readily feasible. \n\nWhile our current approach doesn't directly utilize LLMs or VLMs, we recognize their potential as a viable \"distance metric\".\nThis could result in more human-aligned behaviors, for instance, by capturing the significant semantic distance between states such as \"jumping to the right\" and \"remaining seated\".\n Nevertheless, adopting these models would introduce notable computational challenges and resource demands, which we outline as follows:\n\n- **Eureka [1]** employs complex prompt tuning and multiple iterations of RL policy training, which can be resource-intensive.\n- **Reward Design with Language Models [2]** necessitates frequent queries to LLMs during environment step, leading to high inference costs.\n- **Minedojo [3]** depends on extensive image-text datasets for training VLMs, which are not always feasible to compile.\n\nGiven these considerations, and the constraints of the rebuttal period, we conducted preliminary tests using LLMs to establish distance metrics, specifically using [2] for the 5.2.2 experimental setting to avoid hazardous areas.\n\nWe would like to note that [2] employs the 'text-davinci-002' GPT-3 model as its LLM. However, querying the LLM at every environment step incurs high inference costs, so we opted for the open-source LLM, Llama-3-8B. To generate the distance metric, we used prompts such as:\n\n\"You will describe the consecutive x, y, z positions of a given robot moving on a plane. The robot's consecutive positions are given as: [x1, y1, z1], [x2, y2, z2]. If the x-coordinate value increases, the robot moves to the right. In this case, output the scalar value 1.0. If the x-coordinate value decreases, the robot moves to the left. In this case, output the scalar value 0.0.\"\n\nAs shown in the attached PDF file under \"Global Response,\" the agent effectively incorporates human intention, avoiding hazardous areas while adequately covering the safe regions. However, even with an open-source LLM, inferring the 8B model at each environment step incurs high computational time and cost. Moreover, while creating prompts for basic tasks like \"move to the right\" is relatively simple, developing prompts for more complex desired behaviors becomes increasingly challenging. In comparison, DoDont operates efficiently with minimal data, typically requiring only one to four pairs of instructional videos. We believe that our methodology remains a simple and practical solution. \n\nWe appreciate your feedback and will incorporate this information into the appendix.\n\n>**Question 1.2**\nAccording to Appendix C.1, the Do videos for DMC tasks is collected from a policy that's trained on ground truth reward functions. [...] If so then DoDont is not solving the motivating problem of USD that it needs hand-crafted rewards.\n\nWe wish to clarify that instruction videos does not necessitate the availability of task-specific reward functions. These instruction videos can be acquired through several means.\nFor instance, as outlined in Section 5.2.4 (kitchen environment), they can be obtained from human-collected datasets. \nAdditionally, demonstration videos on platforms such as YouTube (e.g., Minecraft videos) can serve as instruction videos.\nFor humanoid robots, teleoperation techniques [1,2] offer another viable source for these videos. \nThese diverse examples illustrate that the DoDont algorithm is not constrained to the pre-defined task-specific rewards but can operate effectively when provided with suitable behavioral videos.\n\n[1] HumanPlus: Humanoid Shadowing and Imitation from Humans., Fu et al., arXiv 2024\\\n[2] Open-TeleVision: Teleoperation with Immersive Active Visual Feedback., Cheng et al., arXiv 2024\n\n>**Question 1.3**\nFrom the same section, it seems the Don't videos are collected from random action rollouts. [...] probably can't learn to assign a low reward value to this behavior.\n\nAddressing the reviewer's concern, it is important to clarify that our methodology for learning diverse skills while avoiding hazardous zones and unsafe behaviors does not merely use random actions as Don’t videos. Specifically, in our experimental setup outlined in Section 5.2.2, where we identify the left areas as hazardous, our Don't videos depict leftward movements rather than random actions (refer to lines 243-244). In Section 5.2.3, which deals with avoiding unsafe behaviors, we use videos depicting rolling or flipping as Don’t videos (refer to lines 267-268). \n\nWe acknowledge the omissions in Appendix C.1 and will provide a thorough explanation in the finalized version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wbzgFCgHpM", "reviewer": "Reviewer_oJKL", "summary": "Unsupervised skill discovery is an RL task to learn interesting behaviors without rewards from environments. However, since there is no specification of desired behavior either, a lot of learning is wasted on acquiring skills that people may not be interested in eventually. The paper studies a setting where a few demonstration video are provided as a minimal specification of desirable skills. Then the proposed method can train a GAN like loss as a distance measure for unsupervised skill discovery, so the skill learned is more desirable. Experiments on dmlab are shown with interesting analysis.", "questions": "1. What's the camera angle for \"Quadruped\"? Clearly the acquired skill is more diverse than instruction video as shown in figure 3. I am wondering how, because intuitively those trajectories are equally out of distribution.", "limitations": "The paper discusses the limitation of scaling to in-the-wild videos. I think this is reasonable address. No ethics concerns.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written\n2. The setup is well-motivated. It's indeed interesting to see something between complete unsupervised RL & imitation learning. \n3. A lot of the ablation gives the readers good insights about the proposed method, which I find enjoyable to learn.", "weaknesses": "1. The idea of combining unsupervised RL with some form of task specification is not new. In the very early days of unsupervised RL, people already add intrinsic reward and extrinsic reward together. I hope the authors could justify their novelty\n\n2. While the authors discusses USD, I feel a lot of works in RL exploration are not mentioned. e.g. Curiosity [1]. There are other works about using very high level information as guidance, such as DeepMimic [2]. I am curious to see whether the proposed method can be combined with general USD methods?\n\n\n\n[1] Curiosity-driven Exploration by Self-supervised Prediction, Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell\n[2] DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oJKL,\n\nThank you for your valuable feedback on our paper. We have carefully considered your concerns and would like to address them as follows. Please let us know if you have further questions or feedbacks.\n\n>**Question 2.1**\nThe idea of combining unsupervised RL with some form of task specification is not new. In the very early days of unsupervised RL, people already add intrinsic reward and extrinsic reward together. I hope the authors could justify their novelty\n\nWe appreciate the reviewer's comment about the existing work in combining intrinsic rewards with extrinsic rewards. While it's correct that there exist early approaches combined intrinsic and extrinsic rewards, DoDont's main contributions are in:\n   1. Modeling \"human-aligned behaviors\" (i.e., how to design human-aligned extrinsic rewards)\n   2. Integrating this reward into multiple-behavior learning (i.e., how to combine extrinsic and intrinsic rewards effectively)\n\nPrevious approach such as ICM [1] or SMERL [2] added intrinsic rewards with hand-crafted extrinsic rewards. However, as shown in Figure 6, designing hand-crafted extrinsic rewards for multiple human-aligned behaviors is very difficult. DoDont overcomes this challenge by:\n   1. Training a \"human-aligned instruction network\" using easily obtainable instruction videos\n   2. Employing this network as a \"distance-metric in distance-maximizing skill discovery\", thus facilitating the learning of diverse human-aligned skills\n\nThis approach effectively induces multiple human-aligned behaviors, allowing for diverse behavior learning without unsafe actions. We will emphasize these aspects in our revised manuscript to clarify our contribution.\n\n>**Question 2.2**\nWhile the authors discusses USD, I feel a lot of works in RL exploration are not mentioned. e.g. Curiosity [1]. There are other works about using very high level information as guidance, such as DeepMimic [2]. I am curious to see whether the proposed method can be combined with general USD methods?\n\nFor unsupervised RL methods (e.g., ICM[1], RND[3]), it's possible to combine these with DoDont (e.g., reward = r_ICM + r_DoDont). This would likely result in improved exploration while maintaining **single** human-aligned policy. However, our paper focuses on learning **multiple** human-aligned behaviors with a limited instruction dataset, we chose the USD algorithm (DSD) as our baseline.\n\nFor unsupervised skill discovery methods, as outlined in Section 4, DoDont involves two key components: (1) modeling human-aligned behaviors (extrinsic rewards) and (2) combining these extrinsic rewards with intrinsic rewards. The instruction network learned in step (1) can also be applied to other methods like DIAYN [4] and CIC [5] by combining our instruction-network reward with their intrinsic rewards (e.g., reward = r_DIAYN + r_DoDont). However, given that MI-based algorithms such as DIAYN and CIC face inherent pessimistic exploration challenges [6], we believe our approach would be most effective when applied to distance-maximizing skill discovery algorithms.\n\n>**Question 2.3**\nWhat's the camera angle for \"Quadruped\"?\n\nFor DMC Quadruped, we employ the standard camera angle as depicted in Figure 2.\n\n>**Question 2.4**\nClearly the acquired skill is more diverse than instruction video as shown in figure 3. I am wondering how, because intuitively those trajectories are equally out of distribution.\n\nThis likely stems from the generalization capabilities of the instruction network. Instruction network is trained to classify behaviors, categorizing moving upright as '1' and rolling on the floor (i.e., random actions) as '0'. Consequently, even when faced with previously unseen directions of movement, the network tends to assign a high value to upright movements. \n \n[1] Curiosity-driven Exploration by Self-supervised Prediction., Pathak et al., ICML 2017\\\n[2] One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL., Kumar et al., NeurIPS 2020\\\n[3] Exploration by Random Network Distillation., Burda et al., ArXiv 2018\\\n[4] Diversity is All You Need: Learning Skills without a Reward Function., Eysenbach et al., ICLR 2019\\\n[5] CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery., Laskin et al., ArXiv 2022\\\n[6] Learning More Skills through Optimistic Exploration., Strouse et al., ICLR 2022\\", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4u2ALt0Tqr", "reviewer": "Reviewer_DB1p", "summary": "The paper introduces \"DoDont\", an instruction-based skill discovery algorithm designed to learn desirable behaviors and avoid undesirable ones through unsupervised skill discovery (USD). The method uses instruction videos to train an instruction network that distinguishes between desirable (Do’s) and undesirable (Don’ts) behaviors. This network adjusts the reward function of the skill discovery algorithm to encourage desired behaviors. The authors validate their approach through experiments in complex continuous control tasks, demonstrating that DoDont can effectively learn desirable behaviors with minimal instruction videos.", "questions": "- What are the specific characteristics of the instruction videos required for effective training? For example, do they need to be of a certain length, resolution, or context?\n- How does the quality and clarity of the instructional videos impact the performance of the DoDont algorithm?\n- In Section 2.2, alongside DOMiNO, I think you should cite [1] and [2] that also balance a trade-off between intrinsic reward and task reward using constrained optimization.\n- I am not clear on the state space for each tasks. Do you use exclusively visual pixel inputs or do you combine the visual input stream with proprioceptive data? For example, the labeled consecutive states $(s_t, s_{t+1})$ are only pixel values or combined with proprioceptive data?\n\n[1] Skill-Conditioned Policy Optimization with Successor Features Representations\\\n[2] Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The integration of instructional videos into the USD framework is innovative and addresses the challenge of learning desirable behaviors without predefined reward signals.\n- This paper stands out for its practical value, offering a USD algorithm that effectively learns meaningful and complex behaviors rather than merely generating skills that are variations of simple actions/jittering.\n- The paper provides thorough experimental validation on three tasks, showing that DoDont outperforms state-of-the-art methods in learning complex and desirable behaviors.\n- The presentation, writing and clarity of the paper are great.", "weaknesses": "- The instruction network is trained using in-domain video data, which might not always be readily available in real-world scenarios, but can be fairly easy to obtain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer DB1p,\n\nThank you for your insightful feedbacks and the positive support. We have provided a detailed response to your concerns below. If you have any further comments, please let us know.\n\n>**Question 3.1**\nThe instruction network is trained using in-domain video data, which might not always be readily available in real-world scenarios, but can be fairly easy to obtain.\n\nWe agree. Using in-the-wild video datasets is an important direction for future work, which would expand our method's real-world applicability. This limitation has been discussed in detail in Section 6 of our paper; please refer to that section for further details.\n\n>**Question 3.2**\nWhat are the specific characteristics of the instruction videos required for effective training? For example, do they need to be of a certain length, resolution, or context\\\nHow does the quality and clarity of the instructional videos impact the performance of the DoDont algorithm?\n\nThe most critical characteristic of instruction videos for effective training is visual consistency with the training environment. Specifically, the resolution and quality of the videos should closely match those of the training environment. If there's a significant discrepancy between the pixel characteristics in the instructional videos and the training environment, the network's predictive accuracy diminishes, resulting in less accurate human-aligned behaviors.\n\nRegarding length, it's worth noting that our method trains the instruction network by sampling two consecutive frames from the videos. Therefore, the total duration of each video is not a crucial factor in the training process.\n\n>**Question 3.3**\nIn Section 2.2, alongside DOMiNO, I think you should cite [1] and [2] that also balance a trade-off between intrinsic reward and task reward using constrained optimization.\n\nThank you very much. We will ensure that they are included in the final version.\n\n>**Question 3.4**\nI am not clear on the state space for each tasks. Do you use exclusively visual pixel inputs or do you combine the visual input stream with proprioceptive data? For example, the labeled consecutive states (st, st+1)  are only pixel values or combined with proprioceptive data?\n\nWe apologize for the ambiguity regarding the state space for each task. We employ only pixel inputs for the instruction network in both state-based and pixel-based environments. For the policy and critic networks, we utilize state inputs in state-based environments and pixel inputs in pixel-based environments.\n\nWe will include a comprehensive explanation in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to All Reviewers (General Response)**

We deeply appreciate the thoughtful feedback and valuable suggestions from all three reviewers. R1, R2, and R3 correspond to reviewer AtB3, reviewer oJKL, and reviewer DB1p, respectively.

The reviewers highlighted the following strengths in our submission:

- The main idea is intuitive and well-motivated (R1, R2, R3).
- The proposed method was evaluated against several methods in various domains, yielding promising empirical results (R2, R3).

However, the reviewers also suggested conducting several key experiments to enhance our paper:

- Comparison with previous automatic reward design approaches (R1).
- Justification of our novelty (How our approach differs from existing studies that combine extrinsic rewards with unsupervised RL) (R2).

We hope our responses address all the reviewers’ concerns, and we welcome any additional comments and clarifications.

Additionally, we have attached a PDF file containing the qualitative and quantitative evaluation results for R1.

Meta-review: This paper presents an unsupervised skill discovery algorithm that addresses safe and effective exploration through the concept of desired and undesired behaviors learned from a few example videos. The empirical evaluations are comprehensive, demonstrating complex continuous control skills.

The reviewers initially expressed concerns, but after the rebuttal phase, they agreed to accept the paper, contingent on additional experimental results and clarification regarding comparisons to prior work.

The authors are encouraged to address the following points discussed during rebuttal:
* provide comprehensive comparisons to some LLM- and VLM-based reward designing approaches
* discuss the challenges associated with acquiring undesirable behavior videos, especially in real-world settings
* address the limited application to real-world environments (e.g. pixel or high-dimensional observations)
* moderate the claims regarding safety guarantees during training
Decision: Accept (poster)