One After Another: Learning Incremental Skills for a Changing World
Authors: Nur Muhammad Mahi Shafiullah
Lerrel Pinto
Venue: ICLR 2022 Poster (2022)
Keywords: Skill discovery
Incremental reinforcement learning

Abstract: Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive. However, current skill pre-training methods, like many RL techniques, make a fundamental assumption -- stationary environments during training. Traditional methods learn all their skills simultaneously, which makes it difficult for them to both quickly adapt to changes in the environment, and to not forget earlier skills after such adaptation. On the other hand, in an evolving or expanding environment, skill learning must be able to adapt fast to new environment situations while not forgetting previously learned skills. These two conditions make it difficult for classic skill discovery to do well in an evolving environment. In this work, we propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion. This framework allows newly learned skills to adapt to new environment or agent dynamics, while the fixed old skills ensure the agent doesn't forget a learned skill. We demonstrate experimentally that in both evolving and static environments, incremental skills significantly outperform current state-of-the-art skill discovery methods on both skill quality and the ability to solve downstream tasks. Videos for learned skills and code are made public on https://notmahi.github.io/disk


Published as a conference paper at ICLR 2022
ONE AFTER ANOTHER: LEARNING INCREMENTAL SKILLS FOR A CHANGING WORLD
Nur Muhammad (Mahi) Shafiullah mahilcs .nyu.euu
Lerrel Pinto lerrelêcs, werreeees.ny.eeee...
New York University
New York University
ABSTRACT
Reward-free, unsupervised discovery of skills is attractive alternative the bottleneck of hand-designing rewards in environments where task supervision is scarce expensive. However, current skill pre-training methods, like many RL techniques, make a fundamental assumption stationary environments during training. Traditional methods learn all their skills simultaneously, which makes it difficult for them to both quickly adapt to changes in the environment, and to not forget earlier skills after such adaptation. On the other hand, in an evolving expanding environment, skill learning must be able to adapt fast to new environment situations while not forgetting previously learned skills. These two conditions make it difficult for classic skill discovery to do well in an evolving environment. In this work, we propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion. This framework allows newly learned skills to adapt to new environment or agent dynamics, while the fixed old skills ensure the agent doesn" 't forget learned skill. We demonstrate experimentally that in both evolving and static environments, incremental skills significantly outperform current tate-of-theear skill discovery methods on both skill quality and the ability to solve downstream tasks. Videos for learned skills and code are made public on: https: httpmaiintiiiiio//////i/.......a gi hig........ / disk.
INTRODUCTION
Modern successes of Reinforcement Learning (RL) primarily rely on task-specific rewards to learn motor behavior (Levine et al., 2016; Schulman et al., 2017; Haarnoja al., 2018: Andrychowicz et al., 2020). This learning requires well behaved reward signals in solving control problems. The challenge of reward design is further coupled with the inflexibility in the learned policies, i.e. policies trained on one task do not generalize to a different task. Not only that, but the agents also generalize poorly any changes in the environment (Raileanu al., 2020; Zhou et al., 2019). This is in stark contrast to how biological agents including humans learn (Smith & Gasser, 2005). We continuously adapt, explore, learn without explicit rewards, and are importantly incremental in our learning (Brandon, 2014; Corbetta & Thelen, 1996).
Creating RL algorithms that can similarly adapt and generalize to downstream tasks has hence become an active area of research in the RL community (Bellemare et al., 2016; Eysenbach et al., 2018; Srinivas et al., 2020). One viable solution is unsupervised skill discovery (Eysenbach et al., 2018; Sharma et al., 2019b; Campos et al., 2020; Sharma et al., 2020). Here, an agent gets access to a static environment without any explicit information on the set of downstream tasks or reward functions. During this unsupervised phase, the agent, through various information theoretic objectives (Gregor et al., 2016), is asked to learn set of policies that are repeatable while being different from each other. These policies are often referred as behavioral primitives or skills (Kober & Peters, 2009; Peters et al., 2013; Schaal et al., 2005). Once learned, these skills can then be used to solve downstream tasks in the same static environment by learning a high-level controller that chooses from the set of these skills in a hierarchical control fashion (Stolle & Precup, 2002; Sutton et al., 1999; Kulkarni al., 2016) by using Behavior Transfer aid in agent exploration (Campos al., 2021). However, the quality of the skills learned and the subsequent ability to solve downstream tasks are dependent on the unsupervised objective.

---

Published as a conference paper at ICLR 2022
Skill
(a)
Consistency
(b)
(c)
SLIE
Diversity
-20
-40
SKGH
-20
Movemaent planc
Figure I: We present Discovery of Incremental Skills (DISk). As illustrated in (a), DISK learns skills incremen- tally by having each subsequent skills be diverse from the previous skills while being consistent with itself. DISK lets learn skills in environment where dynamics change during training, example of which is shown in (b); Ant environment where different leg breaks every few episodes. In this environment, DISK's learned skills' trajectories are shown (c) under each broken leg each color shows different skill.
Unfortunately, current objectives for unsupervised skill discovery make major assumption about the environment stationarity. Since all skills are simultaneously trained from start to finish, if the environment changes, every skill is equally exposed to highly non-stationary experience during its training. Moreover, all skills trying adapt to the changed environment means that the agent may catastrophically forget previously learned skills, which is undesirable for an agent with expanding scope. This severely limits the application of current skill discovery methods in lifelong RL set- tings (Nareyek, 2003; Koulouriotis & Xanthopoulos, 2008; Da Silva et al., 2006; Cheung et al., 2020), a simple example of which is illustrated in Figure 1(b), or real-world dynamic settings (Gupta et al., 2018: Julian et al., 2020).
In this paper, we propose new unsupervised model-free framework: Discovery of Incremental Skills (DISk). Instead of simultaneously learning all desired skills, DISk learns skills after another to avoid the joint skill optimization problem. In contrast to prior work, we treat each individual skill as an independent neural network policy without parameter sharing. which further decouples the learned skills. Given a set of previously learned skills, new skills are optimized to have high state entropy with respect to previously learned skills, which promotes skill diversity. Simultaneously, each new skill also required to have low state entropy within itself, which encourages the skill to be consistent. Together, these objectives ensure that every new skill increases the diversity of the skill pool while being controllable (Fig. ((a)). Empirically, this property enables DISK to generate higher quality skills and subsequently improve downstream task learning in both dynamic and static environments compared to prior state-of-the-an skill discovery methods. Especially evolving environments, DISK can quickly adapt new skills to the changes in the environment while keeping old skills decoupled and fixed prevent catastrophic forgetting. We demonstrate this property empirically in MuJoCo navigation environments undergoing varying amounts of change. While DISK was designed for dynamic, evolving environments, we observe that it performs noticeably better than previous skill learning methods in static environments like standard MuJoCo navigation environments by improving skill quality and downstream learning metrics.
To summarize, this paper provides the following contributions: (a) we propose a new unsupervised skill discovery algorithm (DISk) that can effectively learn skills in a decoupled manner. (b) We demonstrate that the incremental nature of DISk lets agents discover diverse skills in evolving or expanding environments. (c) Even in static environments,
demonstrate that DISk can learn diverse set of controllable skills on continuous control tasks that outperform current state-of-the- art methods on skill quality, as well as in sample and computational efficiency. (d) Finally, we demonstrate that our learned skills, both from dynamic and static environments, can be used without any modifications in hierarchical setting to efficiently solve downstream long- morghorizon tasks.
BACKGROUND AND PRELIMINARIES
To setup our incremental RL framework for unsupervised skill discovery, we begin by concisely introducing relevant background and preliminaries. A detailed treatment RL can be found in Sutton & Barto (2018), incremental learning in Chen Liu (2018), and skill discovery in Eysenbach al. (2018).
2

---

Published as a conference paper at ICLR 2022
Given MDP with actions A, states 8 S, standard model-free RL aims to maximize a task specific reward rr(s, a) by optimizing the parameters of a policy Te. In contrast to task-specific RL, where the rewards correspond to making progress on that specific task, methods in unsupervised RI. (Bellemare et al., 2016; Eysenbach et al., 2018; Srinivas et al., 2020) instead optimize an auxiliary reward objective ry that does not necessarily correspond to the task-specific reward rT.
Reinforcement Learning:
Incremental, lifelong, or continual learning covers a wide variety of problems, both within and outside of the field of Reinforcement Learning. Some works consider the problem to fast initialization in a new task T given previous tasks T; 1 S i S n (Tanaka & Yamamura, 1997; Fernando et al., 2017), while others focus on learning from sequence of different domains D Dr such that at time t +++ 1, the model does not forget what it has seen so far (Fei et al., 2016; Li & Hoiem, 2017). Finally, other works have considered bidirectional flow of information, where performance in both old and new tasks improves simultancously with each new task (Ruvolo & Eaton, 2013; Ammar et al., 2015). In all of these cases, there is distribution shift in the data over time, which differentiates this problem setting from multi-tast learning. Some of these methods (Tanaka & Yamamura, 1997; Ruvolo & Eaton, 2013; Fei et al., 2016; Ruvolo & Eaton, 2013) learn set of independently parametrized models address this distribution shift over time.
Incremental Learning:
Skill Discovery: Often, skill discovery is posited as learning z-dependent policy (a|s; f) (Eysen- bach al., 2018; Sharma et al., 2020), where: 2 E Z is a latent variable that represents an individual skill. The space Z hence represents the pool of skills available to an agent. To learn parameters of the skill policy, several prior works (Hausman et al., 2018; Eysenbach al., 2018; Campos et al., 2020; Sharma et al.. 2020) propose objectives that diversify the outcomes of different skills while making the same skill produce consistent behavior.
One such algorithm, DIAYN (Eysenbach et al., 2018), does this by maximizing the following information theoretic objective:
+ +
=0)=1((5
(1)
Here, T(S: Z) represents the mutual information between states and skills, which intuitively encour- ages the skill to control the states being visited. H(A|S, Z) represents the Shannon entropy of actions conditioned on state and skill, which encourages maximum entropy (Haarnoja et al., 2017) skills. Operationally, this objective is maximized by model-free RL optimization of an intrinsic reward function that corresponds to maximizing Equation 1, i.e. rr(s, al0) F()), where F(0) is an estimate of the objective. Note that the right part of the objective can be directly optimized using maximum-entropp RI. (Haarnoja et al., 201 8; 2017).
Other prominent methods such DADS and off-DADS (Sharma et al., 2019b; 2020) propose to instead maximize I(S': Z S). the mutual information between the next state s' and skill conditioned on the current state. Similar to DIAYN, this objective is maximized by RL optimization of the corresponding intrinsic reward. However, unlike DIAYN, the computation of the reward is model- based, as it requires learning a skill-conditioned dynamics model. In both cases, the z-conditioned skill policy +=)+)==) requires end-to-end joint optimization of all the skills and shares parameters across skills through single neural network policy.
3 METHOD
In DIAYN and DADS, all skills are simultancously trained from start to end, and they are not designed to be able to adapt to changes in the environment (non-stationary dynamics). This is particularly problematic in lifelong RL settings Nareyek (2003); Koulouriotis & Xanthopoulos (2008); Da Silva al. (2006); Cheung et al. (2020) or in real-world settings Gupta et al. (2018); Julian et al. (2020) where the environment can change during training.
To address these challenges of reward-free learning and adaptability, we propose incremental policies. Here, instead of simultaneous optimization of skills, they are learned sequentially. This way newer skills can learn adapt the evolved environment without forgetting what older skills have learned. In Section 3. I we discuss our incremental objective, and follow it up with practical considerations and the algorithm in Section 3.2.
3

---

Published as a conference paper at ICLR 2022
3.1 DISCOVERY OF INCREMENTAL SKILLS (DISK)
To formalize our incremental discovery objective, we begin with Equation 1, under which categorical or discrete skills can be expanded as:
F(0)
E Fu (p))
[IG(S 2 =mm)==---+)-))m-=) H Z 2m)
(2)
Here, LG refers to the information gain quantity ]G(S; = H(S) Hm()zz)) Intuitively, this corresponds to the reduction in entropy of the state visitation distribution when a specific skill Zm is run. With a uniform skill prior p(z) and M total discrete skills, this expands to:
F(0)
1 M
Σ
(IG(S; = + H(ASS, Z = =m))
(3)
In incremental skill discovery, since our goal is to learn a new skill ZM given M - 1 previously trained skills (z1, (2,,222-. -M1 fix the learned skills 21:M-1, 1. and reduce the objective to:
F(0m) == IG(S; Z 2M) + H(AAS, Z 2M)
(4)
H(S) - - + (A|S,, Z = 2M)
(5)
The first term H(S) corresponds to maximizing the total entropy of states across all skills, which encourages the new skill 2M to produce diverse behavior different from the previous ones. The second term - (S) zMx) corresponds to reducing the entropy of states given the skill 2M. which encourages the skill to produce consistent state visitations. Finally, the third term H(AAS, 2 = 2M) encourages maximum entropy policies for the skill ZM- Note that this objective can be incrementally applied for m 1,... .M, where M can be arbitrarily large. A step-by-step expansion of this objective is presented in Appendix B.
3.2 A PRACTICAL ALGORITHM
Since treat each skill independent policy, framework involves learning the M skill TM given a set of prior previously learned skills TI,".. M- Note that each F. is a stochastic policy representing the corresponding skill 2m and has its own set of learned parameters 6... Let Sm ~ p(s) Tm). 5 EM= p(s) (m) denote random variables associated with the states visited by rolling out the policies Tim < m < M. Then, our objective from Equation 5 becomes:
(6) The rightmost term can be optimized used max-entropy based RL optimization (Haarnoja et al., 2018) as done in prior skill discovery work DIAYN or Off-DADS. However, computing the first two terms is not tractable in large continuous state-spaces. To address this, we employ three practical estimation methods. First, we use Monte-Carlo estimates of the random variables S, S, by rolling out the policy Tm- Next, we use a point-wise entropy estimator function to estimate the entropy H from the sampled rollouts. Finally, since distances in raw states are often not meaningful, we measure similarity in projected space for entropy computation. Given this estimate F(Oxs) of the objective, we can set the intrinsic reward of the skill as rr(s, al al0m)) := f00x). The full algorithm described in Algorithm 1, pseudocode is given in Appendix H, and the key features are described below.
F(OM) H(S)) H((M)) + H(ASS)r)
Monte-Carlo estimation with replay: : For each skill, we collect a reward buffer containing N rollouts from each previous skills that make up our set T. For TM. we keep a small number of states visited by our policy most recently in circular buffer such that our Ta cannot drift too much from the distribution Pr,y (s). Since posnt-wise entropy computation of set of size n has an O(n²) complexity, we subsample states from our reward buffer to estimate the entropy.
Point-wise entropy computation: A direct, Monte-Carde based entropy estimation method would require us to estimate entropy by computing - [8] Eees log p(s), which is intractable. Instead, use nor-mmratt, Nearest Neighbor (NN) based entropy estimator (Singh et al., 2003; Liu & Abbeel, 2021; Yarats et al., 2021):
X
(p) x
In x NN:
(7)
X
1=1
4

---

Published as a conference paper at ICLR 2022
Algorithm Discovery of Incremental Skills: Learning the Mih Skill
Projection function a, hyperparameter k, a, 39, off- off-policy learning algorithm A. Initialize: Learnable parameters as for MMM empty circular buffer BUF with size n, and replay buffer R.
Input:
Sample trajectories T T= UM Tmh from previous M 1 policies on current environment.
while Not converged do
Collect transition (8, a, s') by running Tas on the environment. Store transition data (s, a, s") into the replay buffer, R + (8, a, Add the new projected state to the circular buffer, BUF <- a (
for / = (22..Tupdate do
Sample (s, a, s') from R. Sample batch Ts T.
Find c. the 6th -Narees Neighbor of (s') within BuF,
Find dd the -Nearest Neighbor of (s") within (To).
Set consistency penalty T. == (s') 0ell2.
Set diversity reward ra -loo -aal2-
Set intrinsic reward ry := -are $ra-
Update 8. using A using (s, a, s rr) as our transition.
Return: PM
where each point 2; of the dataset X contributes an amount proportional to In ||,, NN (z;)]le to the overall entropy. We further approximate In R x similar to Yarats et al. (2021) and set k since any small value suffices. Additional details are discussed in Appendix B.
Measuring similarity under projection: To measure entropy using Equation 7 on our problem, we have to consider nearest neighbors in Euclidean space. A good choice of such a space will result in more meaningful diversity in our skills. Hence, in general we estimate the entropy not directly on raw state s, but under projection (s). where a captures the meaningful variations in agent. Learning such low-dimensional projections is in itself an active research area (Whitney et al., 2019; Du et al., 2019). Hence, following prior work like Off-DADS and DIAYN that use velocity- -ased states, we use a pre-defined a that gives us the velocity of the agent. We use instantancous agent velocity since in locomotion tasks it is a global metric that is generally independent of how long the agent is run for. Like Off- DADS and DIAYN, we found that a good projection is crucial for success of an agent maximizing information theoretic diversity.
Putting it all together: On any given environment, we can put together the above three ideas to learn arbitrary number of incremental skills using Algorithm 1. To learn a new skill JM in a potentially evolved environment, we start with collecting an experience buffer 'I with states from previous skills. Then, we collect transitions (s, a, x') by running TM on the environment. We store this transition in the replay buffer, and also store the projected state (s') in circular buffer BUF. Then, to update our policy parameter Ass, on each update step, we sample (s, a, 8') from our replay buffer and calculate the intrinsic reward of that sample. To calculate this, first find the hth Nearest Neighbor of (s') within BUP, called d. henceforth. Then, we sample a batch To C T, and find the beth Nearest Neighbor of (s") within (Tb) {a(s) s E Tb}. called da henceforth. Given these nearest neighbors, we define our consistency penalty re := ||a(s') - Tella and our diversity reward ra == ||a((') 0allæ which yields an intrinsic reward ry == are Br a and are estimated such that the expected value of are and Bra are close in magnitude, which is done by using the mean values of f. and f. from the previous skill TM-1-
Fm collected
Training the first skill with DISk uses a modified reward function, since there is no prior skills T to compute diversity rewards against. In that case, we simply set ry := are throughout the first skills' training, thus encouraging the agent to stay alive while maximizing the consistency. Exact hyperparameter settings are provided in Appendix D.
EXPERIMENTS
We have presented an unsupervised skill discovery method DISk that can learn decoupled skills incrementally without task-specific rewards. In this section we answer the following questions: (a)
5

---

Published as a conference paper at ICLR 2022
Does DISK learn diverse and consistent skills in evolving environments with non-stationary dynamics? (b) How does DISK compare against traditional skill learning methods stationary environments? (c) Can skills learned by DISK accelerate the learning of downstream tasks? (d) How does our specific design choices effect the performance of DISK? To answer these questions we first present our experimental framework and baselines, and then follow it with our experimental results.
4.1 EXPERIMENTAL. SETUP
To study DISK, we train skills on an agent that only has access to the environment without information about the downstream tasks (Eysenbach et al., 2018; Sharma et al., 2019b; 2020). We use two types of environments: for stationary tasks use standard MuJoCo environments from OpenAI Gym (Todorov et al., 2012; Brockman et al., 2016): HalfCheetah, Hopper, Ant, and Swimmer, visualized in Fig. 4 (left), while for stationtt environments we two modified variants of the Ant environment: one with disappearing blocks, and another with broken legs. Environment details provided in Appendix C.
Baselines: our experiments, we compare our algorithm against two previous state-of-the-art unsupervised skill discovery algorithms mentioned in Section 2, DIAYN (Fysenbach et al., 2018) 88 and Off-DADS (Sharma al., 2020). To contextualize the performance of these algorithms, whenever appropriate, we compare them with a collection of random, untrained policies. To make our compari- son fair, we give both DIAYN and Off-DADS agents access the same transformed state, (s)), that we use in DISk. Since Off-DADS is conditioned by continuous latents, for concrete evaluations we follow the same method as the original paper by uniformly sampling N latents from the latent space, and using them as the N skills from Off-DADS in all our experiments. Exact hyperparameter settings described in Appendix D. and our implementation of DISK is attached supplimentary material.
4.2 CAN DISK ADAPTIVELY DISCOVER SKILLS IN EVOLVING ENVIRONMENTS?
One of the key hypothesis of this work is that DISk is able to discover useful skills even when the environment evolves over time. To gauge the extent of DISK's 's adaptability, we design two experiments shown in Fig. (left) and Fig. (b) for judging adaptability of DISK in environments that change in continuous and discrete manner respectively.
We create a MuloCo environment similar to Ant with 40 blocks encireling the agent. The blocks initially prevent long trajectories in any direction. Then, as we train the agent, we slowly remove the blocks at three different speeds. If T' the total training time of the agent, in "fast", we remove one block per 1/40 steps, in "even" we remove two blocks per 7/00 training steps, and in "slow" we remove ten blocks per T/4 training steps. In each of the cases, the environment starts with a completely encircled agent, and ends with an agent completely free to move in any direction; across the three environments just the frequency and the magnitude of the evolution changes. This environment is meant to emulate an expanding environment where new possibilities open up during training: like a domestic robot being given access to new rooms in the house. The three different rates of change lets us observe how robust a constant skill addition schedule is with different rates of change in the world.
Continuous Change:
In Fig. 2 (middle), we can see how the skill learning algorithms react to the changing environment. The comparison between Off-DADS and DISK is especially noticeable. The Off-DADS agent latches onto the first opening of the block-circle, optimizes the skill dynamics model for motions on that single direction, and ignores further evolution of the environment quite completely. On the other hand, since the DISK agent incrementally initializes an independent policy per skill, can easily adapt to the new, changed environment during the training time of each skill. To quantitatively measure the diversity of skills learned, we the Hausdorff distance (Belogay al., 1997), which is a topological way of measuring distance between two sets of points (see appendix B.3.) We plot the mean Hausdorff distance between all endpoints of skill and all other endpoints in Fig. 2 (right). In this metric as well, DISk does better than other methods across environments.
We modify the standard Ant environment to changing dynamic environment by disabling one of the four Ant legs during test and train time. To make the environment dynamic, we cycle between broken legs every IM steps during the 10M steps of training. We show the trajectories of the learned skills with each of the legs broken in Fig. 3.
Discrete Change:

---

Published as a conference paper at ICLR 2022
DIAYN
OFf-DADS
DISk
307
Blnck (slow)
30
Mean Hausdorff distance between skills
-30
Movement in x-y plane
Block (medium)
50
1
-50
120
Blnck (Fast)
-100
-100 -50
0
1.00 1100
-50
300
200
550
300
Figure 2: Left: Block environments that evolve during training; red highlights portion of blocks vanishing at once. Middle: Trajectories of skills discovered by different algorithms; each unique color is different skill. Right: Mean Hausdorff distance of the skills discovered in these environments.
Off-DADS
DISk
In this case, the Off-DADS and the DISk agent both adapt to the broken leg and learn some skills that travel certain distance over time. However, big difference becomes apparent if we note under which broken leg the Off-DADS agent is performing the best. The Off-DADS agent, as seen Fig. 3 (left), performs well with the broken leg that was trained most recently. To optimize for this bcrontly-booken leg, this agent forgets previously learned skills (App. Fig. 7, 8). Conversely, each DISK skill learns how to navigate with particular leg broken, and once skill has learned to navi- gate with some leg broken, that skill always performs well with that particular dynamic.
Movement in x-y plane
28
-20
Sarder Ingt
-40
20
20
Figure 3: Evaluation of skills learned on the broken Ant environment with different legs of the Ant disabled; leg disabled during evaluation legend.
4.3 WHAT QUALITY OF SKILLS ARE LEARNED BY DISK IN A STATIC ENVIRONMENT?
While DISK created to address skill learning in an evolving environment, intuitively it could work just as well in a static environment. To qualitatively study DiSk-discovered skills in static environments, we plot the trajectories generated by them in Fig. (middle). On the three hardest environments (Hopper, Ant, and Swimmer), DISk produces skills that cover larger distances than DIAYN and Off- DADS. Moreover, Ant and Swimmer, consistent with tight packing of trajectories. On the casier HalfCheetah environment, we notice that DIAYN produces higher quality skills. One reason for this is that both DIAYN and Off-DADS share parameters across skills, which accelerates skill learning for less complex environments. Also, note that the performance of DIAYN is worse on the complex environments like Swimmer and Ant. The DIAYN agent stops learning when the DIAYN discriminator achieves perfect discriminability; since we are providing the agent and the discriminator with extra information (ss), in complex environments like Swimmer or Ant DIAYN achieves this perfect discriminability and then stops learning. While the performance of DIAYN on Ant may be surprising, it is in line with the observations by Sharma al. (2019b; 2020).
that skills from DISk are quite
As shown in the quantitative results in Fig. 4 (right), for the harder Hopper, Ant, and Swimmer tasks, DISk demonstrates substantially better performance. However, the easier HalfCheetah task,
7

---

Published as a conference paper at ICLR 2022
DIAYN
Off-DADS
DISK
HalfCheetah
Mean Hausdorff distance between skills
Distributiinn of states across skills
Hopper
Ant Swimmer
-23
21
25 19
-50
-30
100
-220
150
Figure 4: Left: Environments we evaluate on. Middle: Visualization of trajectories generated by the skills learned by DISk and baselines, with each unique color denoting a specific skill. For environments that primarily move in the x-axis (HalfChectah and Hopper), we plot the final state reached by the skill across five runs, For environments that primarily move in the x-y plane (Ant and Swimmer), Right: Hausdorff distance (higher is better) between the terminal states visited by each skill and terminal states of all other skills, averaged the of discovered skills give quantitative estimate of the skill diversity.
plot five trajectories for each skill.
DISK underperforms DIAYN and Off-DADS. This supports our hypothesis that incremental learning discovers higher quality skills in hard tasks, while prior works are better on easier environments.
4.4 CAN DISK SKILLS ACCELERATE DOWNSTREAM LEARNING?
A promise of unsupervised skill discovery methods is that in complicated environments the discovered skills can accelerate learning downstream tasks. In our experiments, find that skills learned by DISK can learn downstream tasks faster by leveraging the learned skills hierarchically. To examine the potential advantage of using discovered skills, set up experiment similar the goal-conditioned hierarchical learning experiment seen in Sharma et al. (2019b). In this experiment, we initialize a hierarchical Ant agent with the learned skills and task it with reaching an (x, y) coordinate chosen uniformly from 1-,,, 15]2. Note that this environment the standard Ant environment with no changing dynamics present during evaluations. To highlight the versatility of the skills learned under evolving environments, we also evaluate the agents that were trained under our evolving Block environments (see Section 4.2) under the same framework.
We generally find that DISK agents outperform other agents on this hierarchical learning benchmark. by generally converging within 100-200k steps compared to DIAYN agent never converging and Off-DADS taking around 500k steps (Fig. 5). This statement holds true regardless of the evolution dynamics of the environment where the agents were learned, thus showing that DISk provides an adaptive skill discovery algorithm under variety of learning cireumstances. This property makes DISK particularly suitable for lifelong learning scenarios where the environment continuously evolving.
8

---

Published as a conference paper at ICLR 2022
Ant
Bedks (4 owe)
Blodie (ever)
Blocks 7wet
Aox thm
2.4
3.3
SAC
143
243
312
403
342
3.25
213 345
402
345
342
243
313
413
342
2.0
243 325
413
343
Stoss
$0ags
Stops
Stops
Figure 5: Downstream Hicrarchical Learning. We plot the average normalized distance (lower better), given by T Ezzo(d ( g)) from our Ant agent to the target over 500K training steps. From left right, we plot the performance of agents that were trained in environment with obstacles, and three evolving block environments. Shading shows variance over three runs, and add goal-based Soft Actor Critic model trained non-hierarchically as a baseline. Across all settings, DISK outperforms prior work.
4.5 ABLATION ANALYSIS ON DESIGN CHOICES
We make several design choices that improve DISK. In this section we briefly discuss the two most important ones. Firstly, our implementation of DISK shares the data collected by the previous skills by initializing the replay buffer R of new skill with data from the previous ones. Once initialized with this replay, the new skill can relabel that experience with its own intrinsic rewards. This method has shown promise in multi-task RL and hindsight relabelling (Andrychowicz et al., 2017; Li et al., 2020; Eysenbach et al., 2020). Across our experiments, performing such relabelling improves performance significantly for the later skills (App. Fig. 10 (left)) it reuse vast amounts of data seen by prior skills, Training details for this experiment presented in Appendix G.
Compared to prior work, DISK not only learns skills incrementally, but also learns independent neural network policies for each skill. To check the performance gains in static environment are primarily due to the incremental addition of skills or the independent policy part, we run a version of our algorithm where all the independent skills are trained in parallel (App. Fig. 10 (left)) on Ant. Experimentally, we see that simply training independent skills in parallel does not discover useful skills; with the parallel variant achieving far lower mean Hausdorff distance. Note that we cannot have incremental addition of skills without independent policies, at least in the current instantiation of DISK, without significant modification to the algorithm or the architecture. This is because it highly nontrivial to extend or modify a network's 's input or parameter space, and the simplest extension is just new network which is DISK itself.
5 RELATED WORK
Our work on developing incremental skill discovery is related to several subfields of AI. In this section we briefly describe the most relevant ones.
Incremental Learning: Machine learning algorithms have traditionally focused on stationary, non-
incremental learning, where fixed dataset is presented the algorithm "Giraud-Carries 2000; Sugiyama & Kawanabe, 2012; Ditzler et al., 2015; Lee et al., 2015 8; 2017; Smith & Gasser, 2005). Inspired by studies in child development (Mendelson et al., 1976; Smith & Gasser, 2005; Smith et al., 1998; Spelke, 1979; Wertheimer, 1961). sub-areas such as curriculum learning (Bengio et al., 2009;
Kumar et al., 2010; Murali et al., 2018) and incremental SVMs (Cauwenberghs & Poggio, 2001; Syed et al., 1999; Ruping. 2001) have emerged, where learning occurs incrementally. We note that the general focus in these prior work is problems where labelled dataset is available. In this work, we instead focus on RL problems, where labelled datasets or demonstrations are not provided during the incremental learning. In the context of RL, some prior works consider learning for a sequence of tasks (Tanaka & Yamamura, 1997; Wilson et al., 2007: Ammar al., 2015) while other works (Peng & Williams, 1994; Ammar et al., 2015; Wang et al., 2019a;b) create algorithms that can adapt to changing environments through incremental policy or Q-learning. However, these methods operate in the task-specifee RL setting, where the agent is trained to solve one specific reward function.
Unsupervised Learning in RL: Prominent works in this area focus on obtaining additional in- formation about the environment in the expectation that it would help downstream task-based RI: Representation learning methods (Yarats et al., 2019; Lee al., 2019; Srinivas et al., 2020; Schwarzer

---

Published as a conference paper at ICLR 2022
et al., 2020; Stooke et al., 2020; Yarats et al., 2021) use unsupervised perceptual losses to enable better visual RL; Model-based methods (Hafner et al., 2018; 2019; Yan et al., 2020; Agrawal et al., 2016) learn approximate dynamics models that allow for acceerraeed planning with downstream rewards; Exploration-base
methods (Bellemare et al., 2016; Ostrovski et al., 2017; Pathak et al., 2017; Burda et al., 2018; Andrychowicz et al., 2017; Hazan et al., 2019; Mutti et al., 2020; Liu & Abbeel, 2021) focus on obtaining sufficient coverage in environments. Since our proposed work based on skill discovery, it is orthogonal to these prior work and can potentially be combined with them for additional performance gain. Most related to our work the sub-field of skill discovery (Ey- senbach et al., 2018; Sharma et al., 2019b; Campos et al., 2020; Gregor et al., 2016; Achiam et al., 2018; Sharma et al., 2020). A more formal treatment of this is presented in Section 2. Since our method makes the skill discovery process incremental, it allows for improved performance compared to DIAYN (Eysenbach et al., 2018) and Off- (Sharma et al., 2020), which is discussed in Section 4. Finally, we note that skills can also be learned in supervised setting with demonstrations or rewards (Kober & Peters, 2009; Peters et al., 2013; Schaal et al., 2005; Dragan al., 2015; Konidaris et al., 2012; Konidaris Barto, 2009; Zahavy et al., 2021; Shankar et al., 2020; Shankar & Gupta, 2020). Our work can potentially be combined with such prior work to improve skill discovery when demonstrations are present.
Creating RL algorithms that can adapt and learn in environments with changing dynamics is a longstanding problem (Nareyek, 2003; Koulouriotis & Xanthopoulos, 2008; Da Silva et al., 2006; Cheung et al., 2020). Recently, a variant of this problem has been studied in the sim-to-real transfer community (Peng et al., 2018; Chebotar al., 2019; Tan et al., 2018), where minor domain gaps between simulation and reality can be bridged through domain randomization. Another promising direction online adaptation (Rakelly al., 2019; Hansen et al., 2020; Yu et al., 2017) and Meta-RL (Finn et al., 2017; Nagabandi et al., 201 8a;b), where explicit mechanisms to infer the environment are embedded in policy execution. We note that these methods often focus on minor variations in dynamics during training or assume that there is an underlying distribution of environment variability, while our framework does not.
RL with evolving dynamics:
6 DISCUSSION, LIMITATIONS, AND SCOPE FOR FUTURE WORK
We have presented DISK, an unsupervised skill discovery method that takes the first steps towards learning skills incrementally. Although already powerful, we still have significant scope for future work before such methods can be applied in real-world settings. For instance, we notice that easy tasks such as HalfCheetah, DISK underforms prior works, which partly due to its inability to share parameters from previously learned skills. Sharing parameters efficiently can allow improved learning and reduction of the total parameters. Next, to measure similarity states, we use a fixed projection function. Being able to simultancously learn this projection function is likely to further improve performance. Also, we use fixed schedule based on skill convergence to add new skills to our repertoire. A more intelligent schedule for skill addition would help DISK be even more optimized. Finally, apply such skill discovery methods in the real-world, we may have bootstrap from supervised data or demonstrations. Such bootstrapping can provide an enormous boost in settings where humans can interact with the agent.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 18x7 2018. 10, 25, 26
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems, pp. 5074-5082, 2016. 10
Haitham Bou Ammar, Eric Faton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer policy gradient reinforcement learning via manifold alignment. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015. 3,
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, NIPS, 2017. 9, 10
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020. 1
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. arXiv preprintarxxlv:7666001888 2016. 3,
E Belogay, Cabrelli, U Molter, and R Shonkwiler. Calculating the hausdorff distance between curves. Information Processing Letters, 64(1), 1997.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009. 9
Robert N Brandon. Adaptation and environment, volume 1040. Princeton University Press, 2014. 1
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 1606:.0000005500 2016. 6, 20
Yuri Burda, Harrison Edwards, Amos Storkey. and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv, 2018. 10
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Tor- res. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference Machine Learning, pp. 1317-1327. PMLR, 2020. 3, 10
Víctor Campos, Pablo Sprechmann, Steven Stenberg Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell. Beyond fine-tuning: Transfer- ring behavior in reinforcement learning. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in neural information processing systems, pp. 409 415, 2001. 9
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience, 2019 International Conference on Robotics and Automation (ICRA), pp. 8973-8979. IEEE, 2019. 10
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 1207):1-207, 2018. 2
Wang Chi Cheung, David Sinchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, pp. 1843-1854. PMLR, 2020. 2, 3, 10

---

Published as a conference paper at ICLR 2022
Daniela Corbetta and Esther Thelen. The developmental origins of bimanual coordination: dynamic perspective. Journal of Experimental Psychology: Human Perception and Performance, 22(2):502, 1996.
Bruno C Da Silva, Eduardo W Basso, Ana LC Bazzan, and Paulo M Engel. Dealing with non- stationary environments using context detection. In Proceedings of the 23rd international confer- ence on Machine learning, pp. 217-224, 2006. 2, 3, 10
Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar. Learning in nonstationary environ- ments: A survey. IEEE Computational Intelligence Magazine, 10(4):12-2) 2015.
Anca Dragan, Katharina Muelling, Andrew Bagnell, and Siddhartha S Srinivasa. Movement primitives via optimization. In ICRA, 2015. 10
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning. pp. 1665-1674. PMLR, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without reward function. CoRR, abs/1802.06070, 2018. URL http: //arxiv. org/ awaa 66700. 1, 2, 6, 10, 18, 22, 24
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint arXiv:2002.11088 2020. 9
Geli Fei, Shuai Wang, and Bing Liu. Learning cumulatively become more knowledgeable. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1565-1574, 2016. 3
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 701.088744 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1226-113.. PMLR, 2017. 10
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
17
Christophe Giraud-Cariier. A note on the utility of incremental learning. Ai Communications, 13(4): 215-223, 2000. 9
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 161 1.07507, 2016. 1, 10
Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot Improving generalization and reducing dataset bias. Advances in Neural Information Processing Systems, 1119004-91144 2018. 2,
learning in homes:
Tuomas Haarnoja, Haoran Tang, Picter Abbeel, and Sergey Levine. Reinforcement learning with deep mnergy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.0500, 2018. 1. 4, 17
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 18/ 1. 1,0555, 2018. 10
12

---

Published as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01003 2019. 10
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. arXiv preprint arXiv:2007.04300 2020. 10
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018. URL https: www.wwwwwwweeeeeee.e.ee..e..eeee forum www-rkozzzzzeeee 3
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration, 2019. 10
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv e-prints, pp. arXiv-2004, 2020. 2, 3
Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems, pp. 1531-1538, 2002. 17
Jens Kober and Jan Peters. Learning motor primitives for robotics. In ICRA, 2009. 1, 10
George Konidaris and Andrew Barto. Skill chaining: Skill discovery in continuous domains. In the
Multidixciplinury Symposium on Reinforcement Learning, Montreal, Canada, 2009. 10
George Konidaris, Scott Kuindersma, Roderic Grupen, and Andrew Barto. Robot learning from demonstration by constructing skill trees. IJRR, 2012. 10
Dimitris E Koulouriotis and A Xanthopoulos. Reinforcement learning and evolutionary algorithms for non-stationary multi-armed bandit problems. Applied Mathematics and Computation, 196(2): 913-922, 2008. 2, 3, 10
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Sacedi, and Joshua B Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv preprint arXiv: 1604006577 2016.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pp. 1189-1197, 2010.
A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv e-prints, 2019.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. arXiv preprint 17 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. arXiv preprint arXiv: /807.0088 2018. 9
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. arXiv preprint arXiv:2002.11700 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947 2017. 3
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint / 2015. 17
Hao Liu and Pieter Abbeel. Unsupervised active pre-training for reinforcement learning. openreview, 6SB. 4, 10
2021. URL wtw.s:ttp////c pttttss:/oppnreeee eww..wwwwww...... net wwtp:/eetoooff/ef/mmmmm
13

---

Published as a conference paper at ICLR 2022
Morton J Mendelson, Marshall M Haith, and James J Gibson. The relation between audition and vision the human newborn. Monographs of the Society for Research Child Development, pp- 1-72, 1976. 9
Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, and Abhinav Gupta. Cassl: Curriculum accelerated self-supervised learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 6453-6460 IEEE, 2018. 9
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task-agnostic exploration, 2020. 10
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforreement learning. arXiv preprini arXiv: 3. 1347, 2018a. 10
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning:
Continual adaptation for model-based arXiv preprint arXiv: 2018b. 10
Alexander Nareyek. Choosing search heuristics by non-stationary reinforcement learning, In
Metaheuristics: Computer decision-making, pp. 523-544. Springer, 2003. 2, 3, 10
Georg Ostrovski, Marc G. Bellemare, Aaron van den Oord, and Remi Munos. Count-based exploration with neural density models. arXiv preprint arXiv, 2017. 10
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv, 2017. 10
Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceed- ings 1994, pp. 226-232. Elsevier, 1994. 9
Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 JEEE international conference roboties and automation (ICRA), pp. 3803-3810. IEEE, 2018. 10
Jan Peters, Jens Kober, Katharina Mülling, Oliver Krämer, and Gerhard Neumann. Towards robot skill learning: From simple skills to table tennis. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 627-631. Springer, 2013. 1, 10
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. https: bttps::/giiie http://gttttb.httt com/ wwt.///RRRR///taae-bbooe 2019. 21
Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation via policy-dynamics value functions. arXiv preprint arXiv:2007.02879 2020.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference machine learning, pp. 5331-5340. PMLR, 2019. 10
Stefan Ruping. Incremental learning with support vector machines. In Proceedings 2001 IEEE International Conference on Data Mining, pp. 641-642. IEEE, 2001.
Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In International Conference on Machine Learning, pp. 507-515. PMLR, 2013. 3
Stefan Schaal, Jan Peters, Jun Nakanishi, and Auke Ijspeert. Learning movement primitives. In RR, 2005. 10
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pp. 1889-1897, 2015. 17
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1700:00007777777 2017. 1
14

---

Published as a conference paper at ICLR 2022
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprini arXiv:2007.05599, 2020. 9
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pp. 8624-8633. PMLR, 2020. 10
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2020.
URL https:://// opppreeeeeeeeeeeeeeeeee net./ forum? id TKgHYONIWI. ONYwr. 10
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware
unsupervised discovery of skills. CoRR, abs/1907.01657, 2019a. URL http: / /arxiv.orgg/
abs/1901 01657. 24
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1977:0007..0657, 2019b. 1,3, 6, 7, 8, 10, 22
Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off off-policy reinforcement learning. arXiv preprint arXiv:2004. 12974, 2020. 3, 6, 7, 10, 23
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23 (3-4):301-321, 2003. 4, 19
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):1322 12,29, 2005. 1,9
Linda B Smith, Alexandra I. Quittner, Mary Joe Osberger, and Richard Miyamoto. Audition and visual attention: the developmental trajectory in deaf and hearing populations. Developmental psychology, 34(5):840, 1998.
Elizabeth S Spelke. Perceiving bimodally specified events in infancy. Developmental psychology, psycholl 15 (6):626, 1979. 9
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations
for reinforcement learning, arXiv preprint arXiv:2004. 04136, 2020. 1,
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International Symposium on abstraction, reformulation, and approximation, pp. 212-223. Springer, 2002.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning
from reinforcement learning. arXiv preprint arXiv, 2020. 10
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments:
Introduction covariate shift adaptation. MIT press, 2012. 9
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):811-211, 1999.
1
Nadeem Ahmed Syed, Syed Huan, Liu Kah, and Kay Sung. Incremental learning with support vector machines. 1999. 9
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332. 2018. 10
Fumihide Tanaka and Masayuki Yamamura. An approach to lifelong reinforcement learning through multiple environments. In 6th European Workshop on Learning Robots, pp. 93-99, 1997. 3, 9
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. 6
15

---

Published as a conference paper at ICLR 2022
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. 17
Zhi Wang, Chunlin Chen, Han-Xiong Li, Daoyi Dong, and Tzyh-Jong Tarn. Incremental reinforce- ment learning with prioritized sweeping for dynamic environments. JEEE/ASME Transactions on Mechatronics, 24(2):621 632, 2019a.
Zhi Wang, Han-Xiong Li, and Chunlin Chen. Incremental reinforcement learning in continuous spaces via policy relaxation and importance weighting. IEEE transactions on neural networks and learning systems, 31(6): 1870:1887--883, 2019b.
Michael Wertheimer. Psychomotor coordination of auditory and visual space at birth. Science, 134 (4491):692-1192, 1961. 9
William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-awart embeddings. arXiv preprint arXiv: /908: 093577 2019. 5
Ronald J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229 256, 1992. 17
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp. 1015-1022. ACM, 2007. 9
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003 054366 2020. 10
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// github. com/ www.syyaattepptorrrrreeeeeeee
2020. 21, 22
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 11X1v:1910.01711, 2019. 9
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. arXiv preprint arXiv:2102.1 12211, 2021, 4, 5, 10
Wenhao Yu, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy
with online system identification. arXiv preprint arXiv:1002.02453, 2017. 10
Tom Zahavy, Brendan O' o'Donoghue, Andre Barreto, Volodymyr Mnih, Sebastian Flennerhag, and Satinder Singh. Discovering diverse nearly optimal policies withsuccessor features. arXiv preprint NBt::2106.00669) 2021. 10
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. arXiv preprint arXiv: /740, 2019.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008. 17
APPENDIX
A REINFORCEMENT LEARNING
In our continuous-control RL setting, an agent receives a state observation 8, E S from the envi- ronment and applies an action a, E A according to policy In our setting, where the policy is stochastic, the policy returns distribution π (st), and we sample concrete action a ~ (st). The environment returns reward for every action r.. The goal of the agent is to maximize expected
cumulative discounted reward Eeg " -n length T.
E x'r,]
for discount factor ? and horizon
16

---

Published as a conference paper at ICLR 2022
On-policy RL (Schulman al., 2015; Kakade, 2002; Williams, 1992) optimizes by iterating between data collection and policy updates. hence requires new on-policy data every iteration, which is expensive to obtain. On the other hand, off-policy reinforcement learning retains past experiences
replay buffer and is able to re-use past samples. Thus, practice, off-policy algorithms have been found to achieve better sample efficiency (Lillicrap et al., 2015; Haarnoja et al., 2018). For our experiments we use SAC (Haarnoja et al., 2018) as our base RI. optimizer due to its implicit maximization of action distribution entropy, sample efficiency, and fair comparisons with baselines that also build on top of SAC. However, we note that our framework is compatible with any standard off-policy RL algorithm that maximizes the entropy of the action distribution π( either implicitly or explicitly.
Soft Actor-Critic The Soft Actor-Critic (SAC) (Haarnoja et al., 2018) is an off- off-policy model-free RL algorithm that instantiates an actor-critic framework by learning state-action value function Qo. stochastic policy Ty and temperature a over a discounted anfnite-horizonn MDP (x, A. P. R, 7, do) by optimizing a 7-discounted maximum-entropy objective (Ziebart et al., 2008). With a slight abuse of notation, define both the actor and critic learnable parameters by 0. SAC parametrizes the actor policy ne (ag ( via a tanh-Gaussian defined as ag tanh(ug (x)) + ar (xx (e)))), where ~ N ((,0 1), He and dg are parametric mean and standard deviation. The SAC's critic Qo (Zt, at) is parametrized as an MLP neural network.
The policy evaluation step learns the critic Qo(x,, ai) network by optimizing the one-step soft Bellman residual:
LQ(D) = (-D.) ax) - -.) and act10a #++1)
V1 R(x,, ae) + 7 7(Q (#++1, a141) log no ra(+111 |
where DD is replay buffer of transitions, 0' is an exponential moving average of done (Lillicrap et al., 2015). SAC uses clipped double-Q learning (Van Hasselt et al., 2016; Fujimoto et al., 2018), which we omit from our notation for simplicity but employ practice.
The policy improvement step then fits the actor Te Ta(a;)s)) sz) network by optimizing the following objective:
C Ex,- (D (xxpt (x
&
Finally, the temperature a is learned with the loss:
L. (D) = E
an log To (a -PPl) a mm))
where H E IR is the target entropy hyper-parameter that the policy tries to match, which in practice is set to = Al. The overall optimization objective of SAC equals to:
CsAC(D)= = (D) + LQ(D) + (D).
B FURTHER MATHEMATICAN DETAILS
B. EXPANSION AND DERIVATION OBJECTIVES
The objective function F(0), defined in Equation 1, is the source from where we derive incremental objective function, reproduced here.
F(0) I H(A[S,
We can expand the first term in Equation as
T(S H(S) H(S) Z)
17

---

Published as a conference paper at ICLR 2022
by definition of mutual information. Now, once assume Z is discrete variable, the second part of this equation becomes
H(S1 2) = Σ p(zm )H(() Z = 2m)
E
[H(S) Z = 2m)l
Zm pppmm))
And thus we have
(0):2)H((S))
(H(S) = 2m
rpp(mm)
(H(S) - H(( 2m)
Fm (mmpp(2m)
But the term inside the expectation is the definition of information gain (not to be confused with KL divergence), defined by
IG(S: 2m) H(S) H(S =
Thus, we arrive at
T(S: = E Zmp mpmmm
[TG(S = 2m))
Similarly, by definition of conditional entropy, we can expand the second part of the Equation
H(A S, Ep(zm )H((A) S, Z 2m))
E 8m ppp(pmm
[H(A S, 2 2m)
Thus, we can convert Equation! into
F(0) I(S; 2) + H(AAS, Z)
E
[IG = =mm] +
E
[H(A S, 2m
~ppppp( Rm)
rupl 2m)
E
[IG(S = = H(A S, zm)]
we assume a uniform prior over our skills, which is another assumption made by Eysenbach et al. (2018), and also assume trying to learn M skills in total, further expand Equation into:
M Σ M m=
F(O)
Z = + H(A S, 2 = 2m)
Ignoring the number of skills term (which is constant over single skills' learning period) gives us exactly Equation 3, which was:
F(O) = Σ [IG(S 2m) H(A | 2m
Now, under our framework, we assume that skills 1, 2, ... MM - has been learned and fixed, and we ignore the associated Information
formulating objective for the Mth skill. As result, Gain and action distribution entropy terms from skills 1,2, ... M 1, and simplify our objective to be:
F(O) TG(S; = zM) H(A S, 2M = H(S) - H(S 2 = =M) + H(A S, Z = 2M)
which is exactly the same objective defined in Equation 5.
18

---

Published as a conference paper at ICLR 2022
B.2 POINT-BASED NEAREST NEIGHBOR ENTROPY ESTIMATION
In our work, alternate approach, first shown by Singh et al. (2003), to estimate the entropy of a set of points. This method gives us a non-parametric Nearest Neighbor (NN) based entropy estimator:
N Σ
kl 1) N k
Hl, (p)
In
+ Ck.
where is the gamma function, Ck In
(k) T(k)
is the bias correction term, and R,,xXx
| NN x (ax) (Ix)|l is the Euclidean distance between æ; and its kth nearest neighbor from the dataset. X, defined as N (xt).
The term inside the sum can be simplified
2 In In N
AT (g//2++11 In R. K
NTO/2
K
kT kT/222+11
-qln Ri,K,x
NT
kI (9/2
y In ||, - NN:)x
NT
Here, In (g/2 +
Nr is constant term independent of æ;, If we ignore the this term and the bias-correction term C2 and the constant, we get
N
H (p) x
Σ In ||=
Which the formulation we use in this work. This estimator shown to be asymptotically unbiased and consistent in Singh et al. (2003).
B.3 HAUSDORFF DISTANCE
In our work, to compare between two algorithms learning skills on the same environment, we used a metric based on Hausdorff distance. Hausdorff distance, also known as the Hausdorff metric or the Hompeiu-Hausdorf distance, is a metric that measures the distance between two subsets of metric space. Informally, we think of two sets in a metric space as close the Hausdorff' distance every point of either set is close to some point of the other set. The Hausdorff distance is the longest distance one can force you travel by choosing point adversarially one of the two sets, from which you have to travel to the other set. Put simply, it is the greatest of all the distances from a point in one set the nearest point in the other.
Mathematically, given two subsets A and B of a metric space (M, d) we define Hausdorff distance d (A, B) as:
(A,A B) max sup d(a, B), sup d(b, A) BEB
(aeA
Where sup represents the supremum, d(r,, Y),, inf inf wey d(x, v) is the distance between point and another set, and inf represents the infimum.
Given a set of skills, we calculate the diversity of one skill over all other skills by calculating the Hausdorff distance between that skill's trajectory end (z. y) location, and the terminal (z, y) locations of all other trajectories. Intuitively, a skill has high Hausdorff distance if the end states it generates is far away from other skills' endpoints. Similarly, a high average Hausdorff distance for skills from an algorithm means that the algorithm's 's generated skills on average have a high distance from each other, which is desirable property for algorithm which needs to generate diverse skills.
19

---

Published as a conference paper at ICLR 2022
C ENVIRONMENTS
Gym Experiments We derive environments used in the experiments in this paper from OpenAI Gym (Brockman al., 2016) MuJoCo tasks. Namely, v3 environments for 2d locomotion tasks and Swimmer-v3 and Ant-v3 environments for the 3d locomotion tasks (see Figure 2 for the agent morphologies).
the HalfCheetah-v3 and Hopper-
Since we aim to train primitives, we want policies that perform well regardless the agent (global position etc.), only depending on local states (join angles etc.). Thus, we train our agents and each our baselines with a maximum episode length of 100 (200 for Swimmer only). while we test them with a maximum episode length of 500 for static or the block environments and 200 for the broken leg environments.
global states of
As our projection function a, we measured the x velocity of the agent in 2d environments, and the (ar, y) velocity of the agent in 3d environments. We made a (s) available to the intrinsic reward calculation functions both our methods and the baselines.
Block Experiments For block experiment set, implemented the blocks immovable spheres of radius at a distance 10 from origin. We dynamically added 40 blocks at the environment creation, and deleted them with the MuJoCo interface available in Gym. The blocks were all added before the agent took the first step in the environment, and removed over the agents' lifetime as described in Section 4.3. The blocks 2rt
always removed counter-clockwis, following the trajectory over E [0, 7). where is the current timestep and T is the total timestep for of (cos T sin 2mf training.
T
Broken Leg Experiments For our broken leg experiment set, we implemented a broken leg as leg where actions have any effect. We switch which leg is broken every 1M steps, and train all skills for total of 10M steps in both Off-DADS and DISk.
42
3
-22
-42
Figure 6: Skills learned by DISk, evaluated with each of the legs broken. The legs are numbered such that the final leg is numbered #4
Log
-60
-20
Figure Skills learned by Off-DADS at end of 10M steps, evaluated with each of the legs broken. The legs are numbered such that the final leg numhered #4,
Hierarchical Experiments For the hierarchical environments, a goal-conditioned manner. The goals are sampled from [-15, 15]2 uniformly, and the hierarchical agent take 100 steps reach as close the goal possible. At each step of the hierarchical
the Ant-v3 environment in
20

---

Published as a conference paper at ICLR 2022
Log
Leg
Log Log
40
20
-20
-40
Figure 8: Skills learned by Oft-DADS at the end of 9.3M, or 0.3M steps after breaking the final leg, evaluated with cach of the legs broken. Compared to this agent, the agent in Fig. 7 performs worse on all broken leg but the last one, which shows that Off-DADS suffers from instance of catastrophic forgetting.
agent, chooses skill, which is then executed for 10 timesteps in the underlying environment. So, in total, the agent has 1000 timesteps to reach the goal. At every timestep, the agent is given dense reward ~| g/|2, where x is the current location of the agent, and g the location the goal. On each step, the hierarchical agent gets the sum of the goal conditioned reward from the 10 timesteps the underlying environment.
All the hierarchical agents were trained with the stable sttable-baselle linesl package (Raffin et al., 2019). We used their default PPO agent for all the downstream set skills, and trained the hierarchical agent for total 500 000 environment steps.
D IMPLEMENTATION DETAILS AND HYPERPARAMETERS
We base our implementation on the PyTorch implementation of SAC by Yarats & Kostrikov (2020). On top of the implementation, add reward buffer class that keeps track of the intrinsic reward. We provide the pseudocode for our algorithm in Appendix H.
D.1 ARCHITECTURE
For all of our environments, used MLPs with two hidden layers of width 256 actor and critic networks. We used ReLU units as our nonlinearities. For our stochastic policies, the actor networks generated mean and variance value for each coordinate of the action, and to sample an action we sampled each dimension from Gaussian distribution defined by those mean and variance values. For stability, we clipped the log standard deviation between [-5, 2], similar to Yarats & Kostrikov (2020).
D.2 REWARD NORMALIZATION
In learning all skills except the first, we set a and 3 such that both the diversity reward and the consistency penalty have similar magnitude. We do by keeping track of the average of the previous skills' diversity reward and consistency penalty, and using the inverse of that as / and a respectively, which is largely successful at keeping the two terms to similar orders of magnitude.
Another trick that helps the DISK learn using a kunh-shaped annealing curve scale up a, which starts at 0 and ends at its full value. This annealing is designed to keep only the diversity reward term relevant at the beginning of training. and then slowly introduce consistency constraint into the objective over the training period. This annealing encourages the policy to explore more in the beginning. Then, the consistency term kicks the skill becomes more and more consistent while still being diverse. As a note, while DISk is more efficient with this annealing, its success is not dependent on it.
21

---

Published as a conference paper at ICLR 2022
Table 1: DISks list of parperr-arameeess....
Parameter
Setting
Replay buffer capacity (static env)
000 000
Replay buffer capacity (changing env)
4 000
Seed steps
5000
Per-skill collected steps
10000
Minibatch size
256
Discount (?)
0.99
Optimizer
Adam
Learning rate
3 x 10
Critic target update frequency
Critic target EMA momentum (TQ)
0.01
Actor update frequency
Actor log stddev bounds
5, 2]
Encoder target update frequency
Encoder target EMA momentum (Tenc)
0.05
SAC entropy temperature
0.1
Number of samples in To
256
Size of circular buffer BUF
50
k in NN
Table 2: DISks number of steps per skill.
Environment HalfCheetah-v3
(Average) steps per skill
Number skills
Total number of steps
125 000
20
2 500 000
H3pper-v3
50
50
2 500 000
Swimmer-v3
250 000
10
2 500 000
Ant-v3
500 000
10
5 000 000
Ant-v3 (blocks)
500 000
10
5 000 000
Ant-v3 (broken)
500
20
10.000 000
D.3
FULL LIST OF
D.4 LEARNING SCHEDULE FOR ANT-V3 ON DISK
We noticed that on the static Ant environment, not all learned skills have the same complexity. For example, the first two skills learned how to sit in place, and flip over. On the other hand, later skills which just learns how to move in different direction could reuse much of the replay buffers of the earlier, complex skills which learned to move, and thus they did not need as many training steps as the earlier complex skills. As result, they converged at different speed. A schedule that most closely fits this uneven convergence time is what we used: 5 000 000 environment steps unevenly between the skills, with the skills getting 250 000, 250 000, 1 000 000, 1 000 000, 500 000, 500 000, 500 000, 500 000, 250 000, 250 000 steps respectively. On all other static environments, we use regular learning schedule with equal number of steps per skill since they are simple enough and converge in roughly equal speed.
D.5 BASELINES
DIAYN We created our own implementation PyTorch based on Yarats & Kostrikov (2020), fol- lowing the official implementation of Eysenbach et al. (2018) 8) method to the best of our ability. We used the same architecture as DISk for actor and critic network. For the discriminator, we used an MLP with two hidden layers of width 256 that was optimized with cross-entropy loss. To make the comparison fair with our method, the discriminator was given access the (s)) values in all cases. Each of the DIAYN models were trained with the same number of skills and steps DISK, as shown in Table 2.
While the performance of DIAYN in complex environments like Ant may seem lackluster, we found that be in line with the work by Sharma et al. (2019h) (see their Fig. 6, top right.) We believe
22

---

Published as a conference paper at ICLR 2022
big part of the seeming gap is because of presenting them in [-125, 125]² grid. We believe this presentation is fair since we want showcase the abilities of Off-DADS and DISK.
We used the official implementation of Sharma al. (2020), with their standard set of parameters and architectures. We provide 000 000 steps to each environment except otherwise specified, and evaluate, we sample the same number of skills DISK from the underlying skill latent distribution. For the hierarchical experiments, we sample the skills and fix them first, and then we create a hierarchical agent with those fixed skills.
Off-DADS
D.6 COMPUTE DETAILS
All of our experiments were run between a local machine with an AMD Threadripper 3990X CPU and two NVIDIA RTX 3080 GPUs rumning Ubuntu 20.04, and cluster with Intel Xeon CPUs and NVIDIA RTX 8000 GPUs, on Ubuntu 18.04 virtual image. Each job was restricted to maximum of GPU, with often up to three jobs sharing GPU.
E CONSISTENCY OF SKILLS
HalfCheetah
Hopper
Swimmer
Ant
Algorithm
Diak
DADS DAANN
Veriance
0.6
Random
0.0
200
200
200
400
200
400
200
400
Evaluation steps
Evaluation steps
Evaluation steps
Evaluation steps
Figure 9: Mean normalized variance over skills of diffcrent agents on four different environments (lower is better). On average, the initial position so has high normalized variance since the agent has little control over it, and the subsequent steps has lower normalized variance the skills lead to consistent states.
To ensure that the diversity of the agent as seen from Section 4.2 are coming from entirely random skills, we run an experiment where we track the average consistency of the skills and compare them measure the variance of the agent location normalized
our baseline. To measure consistency, by the agent's distance from the origin for DISK, each of our baselines, and a set random skills as shown in the Figure 6.
We see from this experiment that our method consistently achieves much lower normalized variance than random skill, thus showing that our skills are consistent. Note that, on the Hopper environment, random skills always crash down onto the ground, thus achieving a very high consistency. While skills from the baselines sometimes achieve higher average consistency compared to DISK, we believe that it might caused by the higher overall diversity in the skills from DISK.
F HIERARCHICAL BENCHMARK
Note that in the fast block environment Off-DADS is able to achieve nontrivial performance; this is because as the circle of blocks opens up fast, the agent learns to move slightly left to avoid the -oe-ee block take advantage of the first opening. In the same environment, DISk suffers the most because the environment itself changes too much while single skill is training, but it is still able learn a decent of skills (see Fig. 2.) As result, their plots seem similar.
G ABLATION STUDIES
In this section, discuss the details of the ablation studies discussed Section 4.5.
23

---

Published as a conference paper at ICLR 2022
G.1 REUSING THE REPLAY BUFFER
Since we use an off off-policy algorithm learn our reinforcement learning policies and compute the rewards given the (s, a, s') tuples on the fly. it is possible for us to reuse the replay buffer from one policy the next. In this ablation, we examine the efficacy of reusing the replay buffer from previous skills.
parallei-DISk
Reuse R
Mcan Hausdorfi Distance
175
Mean Hausdorff distance
80
150
70
125
(last [ skills)
60
100
75
50
Intuitively, if using the replay buffer is ef- fective, it makes more sense that the impact of reusing the replay buffer will be more noticable when there a large amount of experience stored in the replay buffer. Thus, for this ablation, we first train six skills on an Ant agent for 4 000 000 steps total while reusing the replay buffer, and then train four further skills for 250 000 steps each. In first case, we reuse the replay buffers from the first six skills. On the second case, we do not reuse the replay buffer from the first skills and instead clear the replay buffer every time we initialize new skill. Once the skills are trained, we measure their diversity with the mean Hausdorff distance metric.
Figure 10: Ablation analysis of DISK on parallel skill discov- ery and replay R skill relabelling.
As have seen on Figure 10, the four skills learned while reusing the buffer much diverse than the skills learned without reusing the replay buffer. This result goes to show that the ability to reuse the replay buffer gives DISK large advantage, and not reusing the replay buffer would make this unsupervised learning method much worse in terms of sample complexity.
G.2 PARALLEL TRAINING WITH INDEPENDENT SKILLS
We trained our DISK skills incrementally, but also with independent neural network policies. We run this ablation study to ensure that our performance is not because of the incremental only.
To run this experiment, we initialize 10 independent policies on the Ant environment, and train them in parallel using our incremental objective. We initialize a separate circular buffer for each policy. For each policy, we use the union of the other policies' circular buffer as that policy's 's T. We train the agent for 5 000 000 total steps. with each policy trained for 500 000 steps. To train the policies in parallel, we train them in round-robin fashion, with one environment step taken per policy per step in the training loop.
As we have seen in Section 4.5, this training method does not work - the policies never learn anything. We hypothesize that this failure to learn happens because the circular dependencies in the learning objective does not give the policies a stable enough objective to learn something useful.
G.3 BASELINE METHODS WITH INDEPENDENT SKILLS
To understand the effect of training DISK with independent skills, ideally, we would run an ablation where DISK uses shared parameter network instead. However, is nontrivial do so, since DISk necessarily incremental and we would need to create a single network with incremental input or output space to properly do so. Still, to understand the effect of using independent skill, run ablation where we modify some of our baseline methods to use one network per skill.
Out of our baselines, Sharma et al. (2019a) uses continuous skill latent space, and thus it is not suitable to have an independent network per skill. Moreover, its performance degrades if we use discrete latent space as shown in Sharma et al. (2019a) Fig. 5. So, we take our other baseline, Eysenbach al. (2018), which uses discrete skills, and convert use one policy per skill.
We see a better performance with Disjoint-DIAYs agent compared to the DIAYN agent, which implies that some of the performance gain DISK may indeed come from using an independent policy per skill. The reason why disjoint DIAYN performs much better than simple DIAYN may also stem from the fact that it is much harder for the skills in DIAYN "coordinate" and exploit the
24

---

Published as a conference paper at ICLR 2022
160
Algorithm
40
Random
DIAYN
120
DIAYN-DDsjoint
20
Off-DADS
100
DISK
80
-20
40
-40
-60
Env
50
Figure 11: Training DIAYN with one policy skill improves its performance. On the (left) we show the comparison of mean Hausdorff distance of DIAYN with independent policies alongside method and other baselines, and the (right) we show trajectories from running skills from the Disjoint-DIANN agent.
diversity based reward in distributed settings where the skills only communicate through the reward function.
G.4 VALOR-LIKE INCREMENTAL SCHEDULE FOR BASELINES
In VALOR(Achiam al., 2018), to stabilize training options based on variational objective, the authors present an curriculum training method where the latents were sampled from small initial distribution, and this distribution was extended every time the information theoretic objective was saturated. More concretely, if с is a variable signifying the option latent and T trajectory generated by then every time their discriminator reached (iidsiii (c r)] = PD. where pD is set constant s 0.86, they expanded the set of option latents. If there were latents when the objective saturates, it was updated using the following formula:
(1
3K +
K = min
Kmax
Where K is hyperparameter that signifies the maximum number of options that be learned.
DIAYN Ant w/ VALOR-scheduling
DIAYN Ant Block w/ VALOR-scheduling
60
40
20
-20
-20
-40
-60
Figure 12: Training DIAYN for 4M steps with curriculum scheduling similar to VAL.O(AAchiam et al., 2018) results in better performance than vanilla DIAYN but suboptinal performance compared to Fig. 11. On the (left) we show the trajectories from learned skills Disjoint-DIAYN with added curriculum training on static Ant environment. On the (right), show the learned two skills that wasn't available the environment initialization; however other skills are degenerate.
the dynamic Ant-block environment, where the agent
Adapting this schedule for Off-DADS is difficult since it has nominally infinite number of skills throughout training. However, implemented this scheduling for DIAYN by sampling the skill
25

---

Published as a conference paper at ICLR 2022
from the distribution Uniform(K). starting with K = 2, max 25 and updating K with Eq. whenever discriie 7)] R pp threshold was reached, where pD is hyperparameter. We ran this experiments top of the disjoint DIAYN (Sec. G.3) baseline and searched over possible pn values, since we found it necessary to have DIAYN converge to any meaningful skills. We chose the pD values from the set n+T v n. S 10), and found that the best performance was around 6 S < 8, or PD E [0.83, 0.87J, which lines up with the findings in the VALOR paper.
As we can see on the figures Fig. 12, in complicated static environments like Ant, adding curriculum scheduling to DIAYN performs better than vanilla DIAYN with shared weights, but may actually result in more degenerate skills than Disjoint-DDi -DAYN. We hypothesize this may be caused by the arbitrary nature of the threshold E[Passemm (c r) which is also mentioned by Achiam et al. (2018).
On dynamic environments like Ant-block, adding skills successively does allow DIAYN learn one skill that was not available at the beginning, unlike vanilla DIAYN. However, we believe because of the downsides of curriculum scheduling. it also results in lot of "dead" skills, which slow down the addition of new skills, which results in few useful skills overall.
26

---

Published as a conference paper at ICLR 2022
H DISK PSEUDO CODE
Algorithm 2 Pseudocode for DISK training routine for the Mth skill.
envizonment
Stochas: 1c policy network
3 di Projection function on
8 E: State dimenaion
aimension after projecticn
Current skill aumber
P: Number of collected states Irem each past skills
max steps: Number of steps to train the policy for
* replay_ buffer: Standard replay buffer for off off-policy algorithms
buE: Circuler replay buffer
a, 8: Hypesparameter for normalizing consistency penalty and diversity reward respectively
a Training Loop
the beginning training skill, collect using learned policies
Tun: m (1,"",M M- T sample states (projested) visited from each of the M-1 previous skills, (x-1) []
for m in (1, 2, .... 11: cos__coolleccion_. oollected leteed_rojjeetert www.pooooeee_staeeee. while cotal_callese cotal_callletiin_staa
=
env.reset
done False
while not done:
total sal_collletion_tles 1-
(5m (X))
next, reward, done anv.stap(a)
Add profected state to the col collected states
(x_next)
x_next
tatal _stel_ =
while total_ateps max_steps:
env.reset
while not done:
total_ateps +=
xx(xxx
x_rext, reward, dona Env.stop(a) replay_buffer.e
a, x_next]
x x_next
Fush projected state to the circular buffer but.push (x_next)]
Update phase
for i in range (update_steps).
sample minibstch
transitions without reward from the zeplay buffer (er Bey Xtsil: state (BxU), action (axIAl), next state (BXD)
(eer 84+1) = sample (uffiay
compute entropy based reward using the next projected stace (8441)
It compute
(8411)
train exploration Rh agent on an augmented mintbatch transitions (Xi, BG Tar (+1)
update_ (2s,
X+t1) i standard state-based SAC
Entropy based
agnestic zeward computation
at state (BxD,)
a bi di wers' ty reward batch size
daf compute revarda (5, b 256):
Ts uniform_sampla(7,
sample diversity rewerd condidates
Finding nearest neighber for each sample in s (BxD,) over the candidates buffer Ts. bxDs)
i rind pairwise 12 distances (Rxhl between and Tj
dista norm(al:, None, topk_dists, idiatss k=3, dimel, "argest=ralse) compute sopk distances (8x3)
[None, f,, dims-l,
Siversity Fewerds (8x1) are defined as L2 distances to the k-nearest neighber from To diversity_teward - topk_distsl:, 1:]
Finding k-nearest neighhor for each sample in a (5x0,) over the circular buffer buf, UI buclxD,1
rind pairiise 12 distannes (Bx (exlbufl) between and but
dists norm(al;, None, buf [Noe,, : :), dimm-l, p=2)
tisk_dists, topkidists,
dimel, "Largest=aaldd Consistency penalties (Bxl) are defined as 12 distences to the & nearest neighbor from but
compute copk distances (Bx3)
consistency_pena topk_dista[t, -1:]
reward Budiveralty_rewee axconsistency__ennt
returr reward
27

---

Published as a conference paper at ICLR 2022
I DISK SKILLS LEARNED IN THE STATIC ANT ENVIRONMENT IN SEQUENTIAL ORDER
Skill
Skill 2
Skill
100
-100
50
-50
-100
Skill
Skill
Skill
100
-100
Skill
All skills
50
-50
-100
-100
-50
50
100
-100
-50
50
100
-100
-50
50
100
Figure 13: All 10 skills learned by DISK in static Ant environment in the order learned. Even though it seems like skill and did learn anything, skill learns stand perfectly still in the environment while skill learns to flip over terminate the episode.
28


Reviews:
[{"review_id": "hijVTpRhmrt", "reviewer": "Reviewer_bA27", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives**\n\n- The paper considers a setting that is both novel and relevant. Providing agents with the ability to continuouslly acquire new skills in evolving environments is an important research direction.\n- It is well written and easy to follow. The introduction and related work sections properly put the work in context.\n- Results are strong and show that DISk can discover more diverse locomotion behaviors than DIAYN and Off-DADS, both under fixed and changing environment dynamics.\n\n\n**Concerns**\n\n- My main concern is related to the significance of the domains used to evaluate the different methods. I understand that these locomotion environments where agents need to discover skills that walk/run in different directions, so that a hierarchical architecture can then leverage these skills to solve more complex navigation tasks, have been a standard benchmark in the literature for at least 5 or 6 years. However, if we want to truly develop agents that can discover skills autonomously, we need to start considering more complex environments. I cannot agree more with the first sentence of the paper (*Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive.*). Unfortunately, the paper then considers environments where it is very simple to design reward functions that produce the desired behaviors. Note that previous works have considered unsupervised pre-training on Atari (e.g. VISR, APT). Another interesting domain that should be easier to integrate in the current codebase is the Fetch environments in Mujoco, potentially modified to include more objects that could lead to the emergence of manipulation skills.\n- It is not surprising that the baselines fail in the presence of changing dynamics (e.g. overfitting to the last final leg). I wonder if authors could introduce slightly stronger baselines. For instance, a version of DADS where the policy is checkpointed before every change to the dynamics. The discovered set of skills would then be the combination of all the checkpoints (e.g. if there are N checkpoints with M skills each, this version of DADS would return $N \\times M$ skills). This would show whether the problem with DADS is lack of adaptation or catastrophic forgetting.\n\n\n**Other comments and questions**\n\n- Please cite [SNN4HRL (ICLR 2017)](https://openreview.net/forum?id=B1oK8aoxe). This is a very relevant work that combines an intrinsic reward proportional to the magnitude of the speed of the robot with a mutual information regularizer that makes skills distinguishable from each other.\n- How does DISk behave when the the environment dynamics are changed by placing new obstacles instead of removing them? For instance, by making a wall similar to the one in Figure 1 [here](https://arxiv.org/pdf/1712.06560.pdf) spawn after a certain number of iterations?\n- Figure 9 (right) shows the MHD after a certain number of steps. It would be more insightful to see a plot showing how the MHD evolves as a function of the number of steps.\n- I was surprised to see that Skill 2 in Figure 10 does not move at all. Could authors please provide their intuition for why this happens?\n- Please provide a reference for the following sentence in the introduction:\n> Not only that, but the agents also generalize poorly to any changes in the environment\n", "summary_of_the_paper": "The paper introduces DISk, an information-theoretic skill discovery method that discovers skills in a sequential fashion instead of all at once. This provides better adaptation to changes in environment dynamics. Evaluation is performed on standard continuous control benchmarks under changing and constant dynamics, and DISk compares favorably to baselines like DIAYN and Off-DADS.", "summary_of_the_review": "Overall, I lean towards accepting the paper but I have serious concerns about the adequacy of the evaluation protocol. While they clearly prove the advantages of DISk, I'm not convinced that by making advances in these toy environments we are truly getting closer to developing agents that can acquire useful skills autonomously. I acknowledge that the paper mostly follows standard practice in the community, which is why I still recommend acceptance, but the submission would be much stronger if authors could provide results in other settings (e.g. Fetch, Atari).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dKA600ZNyXz", "reviewer": "Reviewer_N9w1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The paper combines two interesting and relevant problems: unsupervised learning of skills and learning in continually changing environments.\n\n- The writing of the paper is clear and the method easy to understand. While a few details are missing (see questions below), the overall approach is simple and can be easily grasped from the explanations. It is also put well into context of prior works.\n\n- The figures visualize the diverse learned skills in an intuitive and easily understandable way.\n\n\n\n## Weaknesses\n\n### Conceptual Weaknesses\nIn comparison to prior works in unsupervised skill discovery the idea of incrementally learning skills with separate networks has a few drawbacks:\n\n- First off, the problem and goal of learning skills in continually changing environment does not seem clearly defined to me: what do we want to final set of learned skills to look like? Should they all work on the final training environment? Should they all work on all environments seen during training? Or should some work on a subset of environments while others work on another subset? What environment would we evaluate the downstream learning in? The paper does not clearly articulate these goals.\n\n-  The requirement to learn a full new network per skill, while feasible in the simple tested environments, seems not scalable to large skill sets. At the same time it seems wasteful to train each new skill from scratch. Some form of parameter sharing between the skills seems required but the very concept of *not* sharing parameters seems to be at the heart of the proposed method and thus in conflict with these considerations.\n\n- The iterative nature of the proposed approach requires us to define a schedule at which we start training new skills. This schedule seems to be fundamentally environment dependent and linked to the rate of change in the environment and thus requires additional human design effort which prior works that learn skills jointly do not require.\n\n- In the setting with environment variations the formerly learned skills are not updated to work on new environments later in training. While this is not a problem in an *expanding* environment (since all skills will still be useful) it becomes an issue in environments that change in different ways, eg by disabling different legs of the ant agent. Now for a downstream task I would need to know which of the skills are still executable in this unseen downstream environment (eg which leg of the ant is currently disabled) as all other skills will not function properly. Priors works will continuously try to adapt all skills to the current environment and thus do not need this additional skill filtering supervision.\n\n\n### Execution Weaknesses\nThe paper's execution has a few weak points, in particular in terms of the experimental evaluation:\n\n- The main quantitative evaluation metric used is some distance between skill's end states in a projected state space. I would argue that this does not necessarily captures the diversity of skill learned to reach these states and a more objective metric is the performance on downstream tasks (since this is what we ultimately care about). While the paper does evaluate downstream learning performance, it is not evaluated on all tasks and in general the experiment seems rushed: it is only presented in the appendix and seemingly only evaluated on a single seed, which is bad practice for RL experiments. I would suggest to instead evaluate all experiments with RL on (multiple) downstream tasks and report those result as the main quantitative evaluation.\n\n- It is unclear to me why the baselines perform that badly on the block environments in Figure 2, particularly the Off-DADS approach. I would expect it to learn still somewhat diverse trajectories in the 2D plane it is optimized to diversify (like it does in Fig 4), but instead it learns skills that always walk in the same direction. The text does not explain this phenomenon clearly. Maybe I am not fully understanding the Off-DADS objective?\n\n- The paper argues that the baselines work better than the proposed method in the cheetah environment since they share parameters, which improves learning efficiency. If this is the case (A) shouldn't we be able to achieve the same performance by training DISk longer? (B) shouldn't parameter sharing also be beneficial for learning efficiency in the harder tasks?\n\n- The method section of the paper does not clearly explain *when* new skills are added. The conclusion alludes to some \"convergence criterion\", but it would be good to add this to the main section (please correct me if I missed it in the text).\n\n\n### Further Comments\n- The paper follows prior works and uses a hand-defined projection of the state space into a lower-dimensional space in which it measures diversity (eg here 2D velocities for the ant agent). To me this seems like a significant assumption that can not easily be translated to more complicated tasks (what would a simple space for measuring \"interestingness\" of driving skills look like?). Since this is an assumption that many papers in this line of work make, I won't count it as a weakness for this particular submission.\n\n- The paper's contribution of learning skills sequentially instead of jointly seems more largely applicable than just in the context of continual learning (although continual learning is an obvious and intuitive application). If there is a good explanation for why the incremental learning works better than the joint learning even on static environments, it might strengthen the paper's scope and clarity to present it as a full (improved) alternative to DIAYN / DADS and show the continual learning results only as one additional application that was impossible to solve with prior methods.\n\n\n## Questions\n\n- Do the baseline approaches that learn skills jointly with weight sharing get the same overall (1) number of environment interactions and (2) number of trainable network parameters?\n\n- I wonder whether the \"greedy\" approach to skill learning in the proposed method (ie learning one skill after the other as opposed to all jointly) can lead to optimization issues where the first discovered skill is somehow suboptimal (eg by combines two primitives into one skills that are frequently used independently of each other in different parts of the skill space and should thus be learned as separate skills) and this suboptimal skill leads to all subsequently found skills also being suboptimal. To put it another way, I am wondering whether the problem of discovering the most optimal skill set can be solved greedily or whether it requires to be solved as a whole in order to optimally divide the behavior space into skills. I am uncertain about this question, but would be interested to hear the authors thoughts on it.\n\n- The idea of learning separate neural networks in a sequential fashion seems applicable to continual learning problems outside RL -- are there other works that apply this idea beyond RL? It might be useful to add more discussion of approaches that train separate neural networks for continual learning in the related work?\n\n- The paper proposes to use a non-parametric estimate for the state entropy. How does this compare to priors works who also needed to measure entropy? Could we instantiate the proposed method with the techniques used in prior works or in turn instantiate prior works with the used non-parametric estimate to understand the relative contribution of this design choice?", "summary_of_the_paper": "The paper proposes an approach for unsupervised learning of diverse skills by maximizing an intrinsic reward function. In contrast to prior works (eg DIAYN, DADS) the proposed approach learns skills incrementally using a separate neural network for each skills. In experimental evaluations on OpenAI gym environments this leads to (a) improved diversity in discovered skills and (b) the ability to learn skills in continually changing environments, where prior approaches struggle.\n", "summary_of_the_review": "In summary, I think the paper proposes an interesting problem along with a simple solution that is clearly explained. As listed above I do however have some major concerns, both conceptually and in terms of the experimental evaluation, and thus cannot recommend acceptance of the submission in its current form.\n\nPost Rebuttal\n============\nAfter considering the authors' rebuttal I have increased my score to recommend (weak) acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2ibuGlp8s2", "reviewer": "Reviewer_HdqY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros:\n1. The paper addresses an important problem in RL: unsupervised skill discovery in stochastic or evolving environments. Dropping this simplifying assumption brings it closer to real-world environments. As far as I'm concerned, this is the first work in this space. To me, the problem itself is real and practical.\n2. The proposed framework of incremental skill discovery (DISk) is novel for separating the learning of individual skills.\n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework.\n\n### Cons:\n\n1. One thing I think was missing is the time-wise comparison between the results of DISk vs baselines in the main body of text. How long did the training take for DISk, DIAYN and Off-DADS in terms of both environments steps and wallclock time on the results presented in Section 4.2 - 4.4? DISk needs to train many neural networks (instead of one like in baselines). How slow does this make them compared to other methods (i.e., in terms of time when trained on the same number of environment interactions)? This is especially relevant for Section 4.4: transferring skills to the downstream task. I hope the authors can elaborate on this.\n2. Also, I'm not particularly convinced by the ablations performed in the paper. The authors try to examine whether \"the performance gains in the static environment are primarily due to the independent policy part\", isn't the natural test for this training shared policy in parallel, rather than individual ones?\n3. Have the authors considered running a version of baselines where the network weights aren't shared among the skills? Authors themselves wonder whether \"the performance gains in the static environment are primarily due to the independent policy part\", after all. This additional experiment might shed light on the question.\n\nSome typos:\n- Page 4: paragraph 3: \"used\" → \"using\"\n\n## Post-rebuttal discusion\n\nI thank the authors for the detailed response. Based on the clarifications that the authors provided, I've decided to increase my score by 1.", "summary_of_the_paper": "The authors have presented DISk, an unsupervised skill discovery method that incrementally learns a sequence of skills. This allows DISK to adapt to changes in the environment or agent dynamics during training. The authors demonstrate experimentally that in both evolving as well as static environments, DISk outperforms existing skill discovery methods on both skill quality and the ability to solve downstream tasks.", "summary_of_the_review": "To me, the problem is real and practical. The methods that the authors propose is novel. Authors provide comprehensive experiments to supports the proposed method's superiority compared to existing baselines. Nonetheless, I have several concerns regarding fair comparison of the DISk vs existing methods. Specifically, I'd learn more about its computational efficiency and see additional experiments and ablations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-nWF9HgcgNw", "reviewer": "Reviewer_eZE6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\nThe authors present three features that each might be useful for the unsupervised skill learning literature. Their skill visualizations are well-presented.\n\nWEAKNESSES\n\nThe current version of this paper suffers from unclear motivation, missing baselines, and missing ablations. I do believe there are good ideas here, however the paper is not ready for publication.\n\n**Unclear motivation**: The authors motivate their approach by describing skill learning in a slowly evolving environment, where it is important to remember skills for every version of the environment along the way. This strikes me as a quite artificial setting. If the world has evolved, why wouldn't we care about the agent's performance just in the most recent \"current\" world? If instead the focus is indeed on performing well in many different environments, why not randomly sample one each episode? Depending on the answer to this question, it would e.g. be interesting to rerun the experiment in Figure 3 but with the broken leg resampled every episode, rather than every 1M steps.\n\nIncremental skill learning is a good idea; so good in fact that it's been done before.\n\n**Missing baseline #1: VALOR**: Achiam et al 2018 (https://arxiv.org/abs/1807.10299) introduced a curriculum for learning skills which introduced new skills based on discriminability performance on current skills, rather than at preset intervals. It is an important baseline as previous work, but also to allow the authors to compare performance-based and fixed-interval skill expansion.\n\n**Missing baseline #2: DSP**: Zahavy et al 2021 (https://arxiv.org/abs/2106.00669) introduced Diverse Successive Policies, which incrementally learns new skills that maximize diversity in terms of successor features. This alternative take on fixed-interval skill expansion is also an important baseline.\n\nThe approach the authors propose simultaneously introduces three features to unsupervised skill learning, but the current experiments don't tease apart which matter. Ideally, I would like to see *each* combination of these features tried with each baseline, however that be a lot to ask and for example how exactly to adapt DIAYN to learn skills incrementally involves a lot of non-trivial choices. However, one of these features is very easy to try with each baseline.\n\n**Missing baseline #3: all other baselines with one network per skill**: It is an unfair advantage to DISk to get to use a separate network per skill, while other algorithms do not. Each other baseline should be run as two versions - one with a shared network, and one with a network per skill. As the authors rightly identify, it may be that one or the other approach is preferred in specific environments. Additionally, the authors should emphasize more clearly that despite repeatedly using the motivation of life-long learning, an algorithm whose parameter use grows linearly in time seems problematic for that setting.\n\nI would also like to clarify one existing baseline and one feature of DISk.\n\n**Question: why is DIAYN so bad?** DIAYN seems to perform implausibly bad, especially in the Ant environments. For example, if DIAYN is performed on velocity-based features, wouldn't the policies in Figure 2 move? My suspicion is that the authors use velocity-based features *in addition* to the environment-provided features, rather than *instead of* them. If true, this is now an unfair comparison, since of course the DISk skills that are *only* based on velocity diversity will learn to move, while the DIAYN skills can instead manifest as different joint configurations. Thus DIAYN should be rerun on velocity-based features alone. If my suspicious is wrong, however, hopefully the authors can provide an alternative intuition for what is going on.\n\n**Question: how is the skill expansion interval chosen for DISk?** It seems like the interval at which new skills were introduced was hand-crafted to match the interval at which the environment evolves, or in the static environment case, was carefully tuned to a complicated variable-interval schedule (Appendix D.4) to get interesting skills. This again seems quite artificial and unfair to the baselines. How sensitive is DISk to this schedule? Seems like an important ablation. Additionally, how do the authors imagine DISk being used in practice in environments where the user doesn't control the environment dynamics?\n\nIn addition to the ablation suggested in the previous question, there is another one missing.\n\n**Missing ablation: shared network**: in section 4.5, the authors rightly state \"Compared to prior work, DISk not only learns skills incrementally, but also learns independent neural network policies for each skill.\" However, the authors then only ablate the first feature, and not the second. The authors should test a version of DISk that uses a single shared network of similar architecture to the baselines.\n\nAdditional more minor comments and suggestions:\n1. Another interesting approach to compare to is that of Groth et al 2021 (https://arxiv.org/abs/2109.08603), which uses a curiosity bonus alone to drive exploration and then periodically freezes copies of the policy as skills. This approach shares with DISk a gradually evolving continuous development of skills that is targeted to the life-long learning setting. The work seems to have been published after the ICLR deadline, however, so it would be unfair to ask for it as a baseline here. That said, it is obviously a baseline of interest for future versions of this work, and would make for good discussion in the present related work section.\n2. The notation of section 3.2 seems to confuse random variables with the probability distributions they are drawn from. Additionally, the union operation in equation 6 doesn't seem well-defined to me. I think what the authors actually mean is: S_m ~ p(s|pi_m), S~(1/M)sum_m p(s|pi_m), and the first entropy term in equation 6 should then be simply H(S).\n3. The authors motivate two of DISk's features (incremental skill learning, independent skill policies), but not the third - why do the authors use a seemingly complicated new entropy estimator, rather than for example using the variational (discriminator-based) approach of DIAYN (i.e. breaking down I(S;Z) in the other direction)?\n4. Why isn't DIAYN included in Figure 3?", "summary_of_the_paper": "The authors introduce a new unsupervised skill learning algorithm that combines three features: 1) incremental skill learning (i.e. learns one skill after another), 2) independent networks per skill, and 3) a nearest neighbor entropy estimator. They compare their approach to DIAYN and off-DADS in both static and dynamic MuJoCo environments in OpenAI gym. Comparisons involve skill visualizations, state coverage metrics, and reusing the skills for HRL on a goal-seeking task.", "summary_of_the_review": "Adapting skill learning approach to dynamic worlds and learning skills incrementally are good ideas, however the current set of experiments do not sufficiently distinguish the author's approach from existing literature, nor pinpoint what is important about their setup. With the baselines, ablations, and clarifications above included, I think this will be a strong paper that I look forward to rereading.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is about an unsupervised method to learn new skills in non-stationary environments by maximizing an intrinsic reward function. Experimental evaluations on OpenAI gym environments show that the proposed approach improves the diversity of the learned skills and is able to adapt to continuously changing environments.

This paper is borderline. After reading each other's reviews and the authors' feedback, the reviewers discussed the pros and cons of this work. Even if the reviewers have pointed out that the paper has some limitations, they agree that the paper represents a valuable contribution and have appreciated the improvements implemented by the authors during the rebuttal, thus reaching a consensus towards acceptance. 
The authors need to update their paper according to what they have proposed in their response and they have to take into serious considerations all the reviewers' suggestions while they will prepare the camera-ready version of their paper.
Decision: Accept (Poster)