Open-Ended Reinforcement Learning with Neural Reward Functions
Authors: Robert Meier
Asier Mujika
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised skill discovery
Reinforcement learning

Abstract: Inspired by the great success of unsupervised learning in Computer Vision and Natural Language Processing, the Reinforcement Learning community has recently started to focus more on unsupervised discovery of skills. Most current approaches, like DIAYN or DADS, optimize some form of mutual information objective. We propose a different approach that uses reward functions encoded by neural networks. These are trained iteratively to reward more complex behavior. In high-dimensional robotic environments our approach learns a wide range of interesting skills including front-flips for Half-Cheetah and one-legged running for Humanoid. It is the first skill discovery algorithm that can learn such skills without relying on any form of feature engineering. In the pixel-based Montezuma's Revenge environment our method also works with minimal changes and it learns complex skills that involve interacting with items and visiting diverse locations.


Open-Ended Reinforcement Learning with Neural Reward Functions
Robert Meier Department of Computer Science
Asier Mujika Department of Computer Science ETH Zürich
ETH Zürich
Zürich, Switzerland romeierêias ethz. ch
Zürich, Switzerland asierm@inf.em wtw.iii...eehc.....
Abstract
Inspired by the success of unsupervised learning in Computer Vision and Natural Language Processing, the reinforcement learning community has recently started focus more on unsupervised discovery skills. Most current approaches, like DIAYN or DADS, optimize some form of mutual information objective. We pro- pose different approach that uses reward functions encoded by neural networks. These are trained iteratively to reward more complex behavior. In high-dimensional robotic environments our approach learns wide range of interesting skills includ- ing front-flips for
and one-legged running for HUMANOID. It is the first skill discovery algorithm that can learn such skills without relying on any form of feature engineering. In the pixel-based Montezuma's 's Revenge environment our method also works with minimal changes and it learns complex skills that involve interacting with items and visiting diverse locations. The implementation our approach can be found here
1 Introduction
Deep reinforcement learning (RL) has proven to be very successful in many challenging tasks (Mihh et al., 2015; Silver al. 2017b Berner al., 2019). These considered intractable just few years ago. However, current methods require enormous amounts of compute to achieve great performance in individual tasks. Most of the time, previous models can be utilised when new tasks are considered, even if the environment does not change. This was also the case in Computer Vision and Natural Language Processing. However, recently unsupervised learning has been shown to be very effective in both fields. By using task agnostic pre-training schemes (Tenney et al. 2019 Brown al.) 20200 Caron et al.) (202T), unsupervised models can solve most tasks with minimal or even no mee-tunnng Unsupervised reinforcement learning aims to bring similar successes to the reinforcement learning community.
Agents which learn a wide range of unsupervised skills may be able to leverage those to solve new tasks faster and with minimal fine-tuning.
Here, we propose a new method for open-ended, unsupervised skill discovery.
We devise iterative process which creates pairs of neural reward functions and policies. The policy optimizes the corresponding reward function. Each them corresponds a skill that the agent has leamned The neural reward function is neural network that maps the current observation to scalar reward. In each iteration, the neural reward function is modified differ from the previous one. This results in increasing the complexity of the encoded task. Then, a new skill is trained to optimize this reward function. We devise several techniques to transfer the knowledge from previously learned
"qqul contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

skills. These mechanisms enable learning the most complex reward functions that our method creates. In fact, we show that some of the functions are impossible learn from scratch.
We empirically test our framework in a diverse set of environments. First, we apply it to a simple 2d navigation task. This lets us perform experiments quickly and gain a solid empirical understanding of the different components of our method. Then, we apply it to three robotic environments where we have continuous high-dimensional observations and actions. In contrast to previous methods, our approach can deal with high-dimensional input directly without feature engineering. Finally, we apply on the challenging Montezuma's 's Revenge Atari game which has visually rich pixel based observations and discrete actions. Despite their differences, our method manages to learn useful and interesting skills in all of them. This shows that neural reward functions are equipped to encode meaningful tasks in diverse environments. We highlight the main contributions of this paper:
Propose the first skill-discovery algorithm work in high-dimensional environments without prior expert knowledge.
Solve a 2d maze that cannot be solved by using random exploration.
Acquire complex skills including performing front- and back-flip in the running in all directions in ANT and HUMANOID, standing and jumping on one leg in HUMANOID.
Learn to run as fast in HUMANOID as a supervised agent trained for tens of millions of steps.
Achieve a higher Particle-Based Mutual Information Metric (Gu et al. 2021 than approaches that explicitly optimize this metric.
Collect the first key and reach several rooms in Montezuma's 's Revenge.
2
Related Work
Our work fits best into the unsupervised skill discovery literature (Mohamed & Rezende 2015
Gregor al.] 2016 Florensa et al. 2017a Achiam al., 2018 Eysenbach et al. 2018 Sharma et al. 2019] et al. (021). Compared to DIAYN (Eysenbach et al_]8001) and similar approaches, we do not fix the number of skills to be learned at the beginning training. This allows us learn new skills in an open-ended fashion. On top of that, maximizing mutual information can lead to degenerate behaviors in digh-dimensional environments. By manipulating small subset of the dimensions, a lot of information can be encoded, without exploring the rest of the state space. Due to this issue, previous methods only consider hand-picked subset of the dimensions perform well.
Open-ended learning (Srivastava et al. 2012
Wang et al.) 2019 2020 Campero et al. 2020 Dennis
al. 2020 ECoffet al.] 2021 Stooke al.
2021 is closely related unsupervised skill discovery.
However, most approaches require either a parameterizable environment (Wang et al.
2019 2020),
some fixed encoding of tasks (Stooke et al. 2021) self-competition (Silver et al.)
2017a: Baker
iet al. 2019). This limits the applicability to environments that are engineered with these restrictions in mind. In contrast, neural networks are universal function approximators and thus, our approach
can encode any possible task in any possible environment, as long as the input is chosen appropriately, Similar to our method, the Go-Explore algorithm (Ecoffet et al. 202T) also explores around the frontier of the known. It uses handcrafted feature map to group similar states. Instead, we use neural reward functions to group them as skills. This does not use any expert knowledge.
Another related line of research to our approach is intrinsic motivation (Stadie et al. 2015 Bellemare let al. 2016 Pathak al. 2017 Burda et al. 2018b Pathak et al. 2019 Raileanu & Rocktäschel 2020). These approaches have managed great success in hard-exploration Atari games. However, these approaches do not learn multiple skills by default. Additionally, their goal is to reward all novel states, which leads to general policies that visit many states. In contrast, skill discovery algorithms try to reward narrow regions of the state space to achieve controllability of meaningful dimensions.
Encoding reward functions as neural networks has also been considered in the literature. Compared to our work, they trained with supervised signals (Abbeel & Ng) 2004 Fu al. | et al. Li et al. 202T). Other approaches train auxiliary rewards with meta- leaanig (Zheng et al.
2019
2018
Du al.
2019 Veeriah et al. 2019) enhance the learning of the original reward function.
Another approach to train multiple behaviors is goal-conditioned learning (Kaelbling) 1993 Schaul
ct al. 2015 Andrychowicz et al. 2017 Rauber et al. 2017 Nair et al. 2018 Veeriah ct al. 2018

---

Warde-Farley et al. 2018 Pong et al. 2019 Choi et al. (2021) ) In automated curriculum learning
(Bengio et al. 2009; Florensa al. 2017b; Forestier et al. | 2017 Graves al. 2017 Sukhbaatar
et al. 2017 Torensa et al. 2018 Matiisen et al. 2019] Narvekar et al., 2020 Portelas et al. 2020a bj Z.hang al. 2020), a sequence of goals is created such that each of them is not too hard nor too easy for the current agent. These approaches mostly rely on low-dimensional goal embeddings. When dealing with high-dimensional observations, they must use dimensionality reduction techniques. These techniques can introduce instabilities or destroy relevant information from the input. Our approach, on the other hand, can deal with high-dimensional inputs directly. On top of that, goals encode narrow region of the state space, while each of our reward functions can be rewarding in large region. This speeds-up up the exploration in 'easy' regions the state space.
Method
We introduce method that performs open-ended, unsupervised skill discovery. It iteratively creates pairs of neural reward functions Rg and policies Te trained to maximize the corresponding Rgg Our proposed method alternates between increasing the complexity of the reward function R.g and leveraging the previously learned skills to learn policy Ty that can solve the new Rg. This yields a general learning procedure that learns complex skills in diverse set of environments. See Figure] for a high level overview.
Train my to opeimine Rtp
to collect poeltive and sogutive
Updace R2 with collected
3.1 Increasing the complexity of Rsg
The reward function in fully observable Markov Decision Process (MDP) is by definition function of the current observation, the next one and the action that was performed. However, in many cases this can be reduced function of just the current observation. Because of this, opt for these simpler reward functions as the basis of our Rg. In our method, the reward function Re is neural network which takes observations o, input and outputs single scalar value, the instantaneous reward r!
Figure I: Main steps of our algorithms open-ended training loop.
Assume that we have policy that can reach states in the MDP which are rewarding under Rig. To increase the complexity of the reward function, we do the following:
Decrease the reward of states which are visited by Te, already have policy that reaches these states. We create a data set Oneg of such states. We refer to these states as negative samples.
Increase the reward of states that can almost be reached by the current policy. This allows to leverage Ty to learn the new reward function. We create data set Opos of such states. We refer to these states as positive samples.
To generate the negative samples, we run Fæ for a given number of steps (ideally until it reaches rewarding states) and store the visited states. To then generate positive samples, we change to performing random actions for a fixed number of steps. To ensure that the new reward function is different from all previous ones, also keep track of all the negative samples that we have collected for all skills in a data set Oneg all-
Finally, we set target values a and -a for the positive and negative samples respectively, and train the reward network using standard supervised learning on the following loss:
+
(R a)2
(R aa)
Σ DEO..
Σ
Σ
C.
1Onegl
1Onegatt
1ppoe
DEOneo.att
oEO, you
This loss ensures that positive samples that have never been seen before have positive reward in the next Ry. while all other samples that have been seen before decrease their reward. In the reinforcement learning phase clip rewards the [0, a] range. This ensures that the agent seeks positive samples, rather than less negative ones.
In MDPs with discrete actions we sample actions u.a.r. and in the continuous case, we keep the mean of ne and increase the standard deviation.
3

---

3.2 Forward transfer for Te
Given the procedure presented Section 3.1 we create increasingly complex reward functions. While this is great for open-ended learning, it eventually leads to skills that are too complex and cannot be learned from scratch. In order to learn these skills, we must leverage previous knowledge about the environment. In this section we present several forward transfer mechanisms that are necessary for the most complex skills.
In principle, our method can be combined with any standard reinforcement learning (RL) technique. But here, we focus on actor-critic methods like Advantage Actor- eetor-Critic (Mnih et al. 2016) or Proximal Policy Optimization (Schulman et al.) 2017) for learning Ty for several reasons. These methods have a value network that is separate from the policy which allows us use different transfer mechanisms for the value and policy network. Also, these approaches work for both continuous and discrete action spaces. This allows us to use the same technique for the robotic environments and for the 2d navigation tasks. Finally the learned policies are stochastic which increases the diversity of negative samples and speeds up the skill discovery process. See Section 4.1.2
We present our three forward transfer mechanisms below. All of them exploit the similarity between consecutive reward functions to ensure that even very complex reward functions can be solved by the RL agent in a reasonable number of environment interactions.
Initialize the value network to the final value network of the previous agent. While two consecutive reward functions are different, both still reward close-by regions of the state space. Thus, by keeping the previous value function, the policy network will be nudged towards that region of the state space from the very first gradient updates.
Value Reuse:
Policy Feature Reuse: Initialize the policy network to the final policy of the previous agent but setting the weights of the final layer to O. This keeps the previously learned features, but outputs a uniform policy over all actions (or mean 0 and a fixed standard deviation in the continuous case) which allows for proper learning and exploration
Guiding Policy: Act with the previous policy for random number of steps at the beginning of each episode. This heavily simplifies the exploration problem. The agent will start exploring from states that are much closer to the rewards defined by Rgg. This is because the previous policy could already solve the previous reward function. In contrast to the other two mechanisms, this one does not rely on initialization. Thus, this the most effective in sparse reward functions that need many parameter updates to be learned.
In Section 1.1
we individually evaluate these three techniques and show that their combination is necessary in complex environments.
Putting everything together we get algorithm which learns reward functions that encode increas- ingly complex behaviors and learns RL agents that solve those reward functions. Figure SeggureTillna iillustrates the main steps of our training loop and see Algorithm ii the appendix for more detail.
4 Experiments
We now proceed to experimentally test method. First, in Section 4.1 different components of our model in a 2d navigation task. This task allows us to verify the function of each component and also explicitly visualize what each reward function is encoding.
thoroughly test all
Then, in Section 4.2 we move to BRAX robotic environments (Freeman et al., 202Tp. These have a lot of flexibility and thus allow the agent to learn very complex tasks. In these tasks, we evaluate the complexity of our skills by measuring their zero-shot transfer ability to the environment rewards. In the HUMANOID environment, our unsupervised skills outperform supervised agents trained for tens of millions of time steps. We also compute the one dimensional particle-base metric that has been proposed in the literature before (Gu al. (2221) and show that method outperforms previous approaches, even when other approaches only consider handcrafted feature dimensions in their objective.
mutual information
PPoiccess at the end of learning can be very deterministic which slows down or completely stops learning of new reward functions.

---

we apply our method to Montezuma's 's Revenge. We show that the learned reward functions keep getting increasingly complex and we are mostly limited by the amount of compute that it takes to learn each new reward function.
the current Finally, in Section 4.3
4.1 2D Navigation Task
The task consists of 32 by 32 maze with several walls and "danger zone'. The observation is given as 32x32x1 image with all values set to O, except position of the agent. The agent starts in the top left corner and can perform actions, either move in one the car- dinal directions or stay in the current position. If the agent moves into a wall it stays at its current position instead. If the agent is in the 'danger zone' and moves up, moves down or stays, the episode is terminated and the agent is moved back to the starting position. Figure 2 shows the layout of the maze. The 'danger zone' ensures that random exploration will not work reach many parts of the environment and lets us easily test both the increasing complexity of the reward functions and the importance of forward transfer. We computed the expected number of steps to reach the bottom right corner with a random walk using Dynamic Programming. In expectation, 7 1027 episodes are needed to do so. This shows that this maze is difficult to navigate.
The black bars represent the walls and the dotted line represents the danger zone in the 2d maze. 40 skills dis- covered by our method are shown. For each skill, the circles represent the av- erage position Colors change from early skills in purple to late skills in red. Consecutive reward functions
Figure 2:
locations visited often.
connected by line. The agent learns increasingly complex skills, until it reaches the bottom right corner of the maze; the hardest part to reach in the whole maze.
To train agent (A2C) (Mnih et al. (016) algorithm. To learn the rewards use the full algorithm presented in Section 3 We use the same architecture for the reward, policy and value net- works. but do not share any parameters. The architecture is a ReLU network with 2 convolutional layers followed by 2 fully connected layers. For the exact hyper- paremeeassssssss see Table in Appendix locations for each skill of one run of our algorithm. The first few skills visit points near the origin, later skills start moving harder to reach parts of the state space. After roughly 40 iterations they reach the bottom right part. As stated before, this would take unreasonably long when using only random exploration.
the Advantage Actor Critic
fractioo of vontee
co
B Figure plots the most visited
0.013 0.023 0.23 0.45 0.023
adactive
entropy regularizzaion
Figure 3: Total number of states visited by the learned policies after genera- tions as a function of the entropy regu- larization used by the underlying A2C agent. The adaptive entropy model man- ages to visit all states all runs.
Inspired by the BRAX library (Freeman et al. 2021), we implemented both the environment and an A2C agent inside single JAX (Bradbury al., 2018) compiled func- tion. By doing this, the computation graph of the environment and agent are optimized jointly and both run on the GPU. This eliminates the need to send data between the CPU and GPU which is one of the main bottlenecks in RL. Using just one NVIDIA RTX 3090 GPU, the training process runs at over one million frames per second which enables training of agents in just a few seconds. This allows us to experiment quickly and at lower economic and environmental cost. We believe this code useful for the RL. community on its own and provide the supplementary material.
4.1.1 Forward transfer of skills
As pointed out in Section 3.2 our reward functions become too complex to be learned from scratch with random exploration in a reasonable number of steps. When this happens, our agent must rely on transferring knowledge from previous generations. Our navigation task is specifically designed to test this transfer ability, as random exploration would never reach the bottom right corner (7 10 episodes in expectation).
We experimentally evaluate the three forward transfer mechanisms proposed in Section 3.2 Value
reuse, Policy feature reuse and Guiding policy. The reward functions from Figure 2 serve tasks,
5

---

sorted according to creation order. We train ablations of the three mechanisms sequentially on these tasks. This allows us ignore the skill discovery process and only measure the forward transfer of skills. We repeat each experiment three times. The agent with all mechanisms always manages to solve all reward functions. On the other hand, the Policy, Value and Guiding ablations fail to learn after solving 29.3 + 9.5, 18.7 7 + 11.6 and 19 + 8.5 reward functions, respectively.
4.1.2 Speeding-up deep exploration
E
One key parameter when training actor critic methods entropy regularization. method, policies with a lot of entropy generate a diverse set of negative samples. Diverse nega- tive samples lead to reward functions that evolve more in each generation. This especially ben- eficial in environments where many steps necessary to reach certain states, like in this 2D navigation task Montezuma's 's Revenge. We empirically verify this claim by training a set of agents with varying levels of entropy regulariza- tion. Figure 3 shows the coverage of the state space after 60 generations a function of the entropy regularization. We observed that higher entropy leads space up to a certain threshold. However, too much entropy leads policies that do not learn to reach the rewarding states when these are far away from the origin. Observe that this problem arises independent of the training procedure as entropy regularization changes which policy is considered optimal.
our
(a) Fixed 0.0025
(b) Fixed 0.035
(c) Adaptive
(d) Reward Function
faster coverage of the state
Figure States visited by policies trained solve a reward function (d) that was created in the train- ing process of Figure We trained two policies with a fixed entropy regularization of 0.0025 (a) and 0.035 (b). (c) shows an agent trained with an entropy regularization of 0.0025 in non-rewarding states and of 0.035 in rewarding ones.
Hence, want have policies that gather diverse samples change the reward function but can also be deterministic in parts of the trajectory that already have no reward. To achieve this, we use adaptive entropy regularization. That is, apply a small entropy regularization term everywhere. We increase the entropy regularization in states where the reward function is positive, i.e. states in which we want decrease the reward function. See Figure 4 for visualization of the impact of the different entropy regularization strategies. With this, the method manages to consistently visit all states the maze. This technique was necessary create the reward functions from Figure
4.2 Robotic Environments
Quantitatively measuring unsupervised reinforcement learning progress remains open problem and active area research (Gu et al. 2021). Because of this, it is still important to visualize and qualitatively study the learned skills. We do this in Section 4.2.1 In the next two Sections, 4.2.2 and 4.2.3 we quantitatively measure the performance our algorithm on downstream task performance and with the so-called particle-based mutual information metric.
In contrast to the previous experiments, the action space continuous and the input modality is now a vector of features, like relative position, angle and speed of the different joints. We do not use adaptive entropy here, as we need deterministic policies to maximize the downstream task performance and the mutual information metric. Given that the environments do not have far away regions reach, did not implement the guiding algorithm here. We do ablations for the other two transfer mechanisms. Note that the agent does not see the x-y position. Details about typer-parameter and architecture can be found in Appendix(C
In addition to skill discovery methods, we also compare our methods to RND BBurda et al. 2018h) and 'Disagreement' (Pathak et al.) 2019), two of the most prominent works intrinsic motivation. By default, these methods only create single final policy. We extract multiple skills by taking
We consider reward function solved if the agent manages to find positive reward.

---

checkpoints of the policy regularly during training To our knowledge, this has not been done before and we believe this showcases how closely related intrinsic motivation and unsupervised skill discovery are. We tuned the hyper-parameters parameters of both methods extensively to maximize the Mutual Information metric (c.f. Section 4.2.3)
4.2.1 Qualitative Analysis
In Figure we illustrate a selection of particularly interesting skills. Videos of these and addi- tional skills can be found here In Figure 6 we see how the velocity of learned skills in ANT and HUMANOID evolve over training. See Appendix for the other runs. One can see that con- secutive skills slightly change the direction and speed of moving. However, it is important to realize that the observations contain tens hundreds of dimensions. Thus, the skills can encode much more complex behaviors than speed and direction of movement. For example, Figure shows that the skill involves keeping one leg the air on top of moving the right direction.
We also show scatter plots for RND and 'Disagreement" in Appendix E
While the two methods also work well in the HUMANOID environment (c.f.
Figures 14 and [5]. they do not find any useful behavior in the ANT en- vironment (c.f. Figures [T and T2). These approaches reward all novel states, they lead to policies that cover large regions of the state space, as can be seen in ANT. We believe that the inherent instability of HUMANOID forces the policies to be more specific.
Figure 5: Visualization of four learned robotic skills. One of the legs is tracked with red circle when it is visible. The HALF-CHEETAN learns to do a front-flip. The ANT does a partial rotation around its torso and then starts running. The upper HUMANOID also does a partial rotation and then runs backwards. The lower HUMANOID jumps moving forward.
4.2.2 Zero-Shot Transfer
After discovering 50 unsupervised skills, we identify the skill that aligns best with the reward given by the en- vironment. We report this zero-sho performance Table T As a base- line, train agent from scratch and measure how long it takes it to reach equivalent performance. We use the Optuna hyper-parameter op- timizer (Akiba al., (219) to find hyper-parameters which maximize the speed of learning beforehand. The exact procedure can be found in Ap- pendix D
leg while
ANT
HUUANOID
10
The simplest environment, HALF- CHEETAH, benefits the least from supervised learning, while the hardest one, HUMANOID, benefits the most. We report the zero-shot performance of RND, 'Disagreement' and the ablations in Table 7 in the Appendix. As already seen in Sec- tion 4.2.1 the intrinsic motivation methods do not find any interesting behavior in ANT. Conversely, they perform well on the HUMANOID environment, the latter one even slightly outperforming our method
Figure 6: Scatter-plots of the x-- and y-velocity of the states visited by the first 13 skills of one run in ANT and HU- MANOID. The second half of the trajectory is shown. Colors change from early skills in purple to late skills in red.
'We thank anonymous reviewers 'Wytf' and 'uUdP' for suggesting this baseline.
Teechicaaly,, we query the true reward function to identify the skills, but do not perform any training with it. NNote that the tuning RND and 'Disagreement" for the r-velocity Ml-metric promotes locomotion.
7

---

4.2.3 Particle-based Mutual Information metric
Measuring how well the agent can control rel- evant state dimensions is another way to track progress in Unsupervised reinforcement learn- ing. This is measured using the mutual informa- tion between state dimensions and skills. While high- high-dimensional estimation of mutual infor- mation is an active area of rescarch, sampling can be an effective form of estimation in the 1- dimensional case, c.f. Algorithm 1 in (Gu et al.] 2021). We report the particle-basee mutual in- formation for the z-velocity in Table 2 using the same bucketing strategy as in (Gu et al.
Table 1: Zero-shot environment reward of our al- gorithm and the number of steps a supervised PPO agents needs to match it. Both columns averaged over 10 repetitions.
Task
Zero-shot reward
Steps from scratch
Cheetah
1094 = 1130
340K 50K 1.2M + 0.24M
Ant Humanoid
2506 + 511
9092 + 1063
55M 27M
202T
Our method heavily outperforms DIAYN and DADS, when both methods look at the complete observation space. DIAYN and DADS achieve diversity by learning set of skills that can be correctly labeled by a neural network. In high-dimensional spaces this is casy to do by relying small subset of all state dimensions. This leads to non-diverse behaviors across most state dimensions. Even when expert knowledge about relevant dimensions is supplied to other methods, Le. only taking the and y-velocity into account, our method still fares well. Particularly, in the most complex environment, HUMANOID, our method performs best. We believe that the iterative increase in complexity leads better coverage of hard reach regions of the state like high speed running backwards. With all this, our approach achieves greater controllability of the v-velocity than DIAYN and DADS without any kind of feature engineering.
Surprisingly, our nove bbseelines, RND and 'Disagreement', also achieve better performance than DIAYN and DADS in the HU- MANOID environment, but not in ANT. As already discussed in Section I we believe that the larger instability of HUMANOID compared to ANT is the reason for this difference in performance.
Table 2: Particle-based mutual information metric for the x-- velocity. Results are averaged over 10 runs. Algorithms with feature engineering only consider x-y velocities. Baselines taken from (Gu et al. 202T)
Task
Method
Feature
MI(8, z)
Engineering
Cheetah
Ours DIAYN
1.40 0.21
Cheetah Cheetah Cheetah
X
0)49 + 0.16
DIAYN,
1.82 + 0.20
GCRL
V
1.63 + 0.16
4.3 Montezuma's Revenge
Ant
Ours
X
1.33 + 0.11
To show the generality of our ap- proach we evaluate the notori- ously hard Montezuma's Revenge Atari game. this game, the agent controls character in complex 2d world with several rooms. Ap- pendix G shows the initial room and various items with which the agent may interact. Same in (Mnih et al. 2015). the observation stack of the last 4 frames. This gives the agent information about speed and direction of movement. This is done for all networks, that is, the value, policy and neural re-
Ant
RND
X
0.26 0.15
Ant
Disagreement
X
0.08 0.03
Ant
DIAYN
X
0.32 0.07
Ant
DADS
X
Ant
DIAYN,
+
Ant
GCRL
1.22 + 0.19
Humanoid Humanoid
Ours
X
1.29 + 0.25
RND
X
0.94 + 0.17
Humanoid
Disagreement
1.05 0.14
Humanoid Humanoid Humanoid
DIAYN
X
0.07 + 0.01
DADS
X
0.24 0.06
DIAYN P
0.93 + 0.13
Humanoid
GCRL
0.77 + 0.15
We split the dimension in 1000 bins in the [-10, 10] range. As in Gu et al. 2021 we only take the second half of each trajectory. The initial state is independent of the skill and thus the beginning of the trajectory does not anything about the gained controllability.
'The approaches are not new, but using them to generate checkpoints that can be used as skills has not been done before to the best of our knowledge.

---

ward networks. We use the same simple CNN architecture as in (Mnih et al. 2015) for all three networks. For exact learning details see Appendix
Our algorithm uses finite episode lengths because once reaches a rewarding state, the agent can stay there forever. Because of this, reset the environment every 500 steps.
One of the main difficulties when dealing with Montezuma's Revenge is that it cannot be simulated as fast as the other studied environments. On top of that, agent can learn hundreds of different skills without ever leaving the first room. Finally, skills learned by our agent evolve from simple to very complex and extended in time. In the beginning, the agent just needs to stay close to the initial position. By the end of training, the agent learns to collect a key, open the door, avoid several enemies and visit four different rooms. In the most complex skills, it takes the agent several hundred steps to reach a rewarding state. This means that experiments can take several days to visit a different room.
In order to save computation, we adapt the num- ber of training steps to ensure the agent has learned each skill. See Appendix F for details.
We found that the first thing the agent learns is to get the life counter to O. Then it starts exploring the room, with lives already. This is because losing life happens very easily during random exploration and because once the agent reaches 0 lives, all future positive samples will have lives. This has two side effects. On the one hand, the initial 100 - 200 steps of an episode are spent losing lives. On the other hand, exploration is harder, as the agent will reach a terminal state very easily, Because of this, cut out the part of the image that shows the remaining lives, see Appendix G
(a)
(b)
Figure 7: (a) The position of the agent in the most rewarding state of each skill. The skills are num- bered by the creation order. (b) The rewards re- ceived by skill 29 during part of its trajectory. This shows the long term nature of the discovered skills.
Figure (() visualizes different skills. The first observations which get the maximum possible reward are overlaid. All the shown skills go the bottom and kill the skull first. Then, they move to specific location. It can be seen that later skills explore states further and further away from the initial state. Figure 7(b] shows that the skill receives no reward for many steps. Only after the agent has killed the skull being the ladder becomes rewarding 10 This shows that both the reward network and the agent have learned lot of the concepts that are necessary to tackle Montezuma's 's Revenge. This includes controlling the agent in the 2d environment, killing enemies and interacting with different objects. All of this without ever accessing the original reward function. Eventually, after learning a few hundred skills, the neural reward function pushes the rewarding states all the way to the fourth room. This vidco shows a run of this skill.
Limitations
Our approach has three main limitations: the randomized exploration strategy, the re-learning necessary for each new skill and the depth-first nature of the search of skills.
We have shown that the random exploration allows for discovering diverse skills. However, it can lead to slow progress in complex regions of the state space. Using more sophisticated exploration policies could lead to faster evolution of the skills. A potential avenue for improving this is using previously discovered skills as exploration policies.
we use multiple techniques to make the learning of the next policy more As stated in Section 3.2
efficient. Still, the vast majority of our compute is used for learning these policies. Using more advanced transfer learning or even meta-learning algorithms, could make the policy learning more efficient. Another possibility would be to relabel previously seen trajectories with ideas similar to (Rauber et al. 2017).
Using the ladder is needed to kill the skull, but the reward function gives no reward when the agent is on the ladder before killing the skull.

---

Finally, our approach only uses the current skill to create the next one. On a high level, it uses single search point at each time. Then, the iterative increase in complexity of the skills leads DFS-like exploration. This may not always be optimal. For example, in Montezuma's 's Revenge it causes the agent to explore only one of the two possible exits of the first room. On top that, our algorithm can sometimes get stuck in places were backtracking is complicated or impossible, see Figure 9 in the Appendix. This also happens in Montezuma's 's Revenge when the life-counter is not removed. Figuring out smart back-trackin strategies keeping multiple search points could address this problem and make our algorithm even more efficient.
Conclusion
We have presented an unsupervised reinforcement learning algorithm that uses reward functions encoded by neural networks. Our algorithm alternates between increasing the complexity of the reward function and transferring previous knowledge learn a new skill that finds rewarding states. This allows it to learn an unbounded number of skills.
We have thoroughly tested the different components of our model in 2d navigation task. This has allowed us to better understand our method in practice. We have shown that our method works both with high dimensional feature inputs, in robotic environments, and pixel inputs, in Montezuma's 's Revenge. Our algorithm learned a diverse set of skills in both settings. HUMANOID and Montezuma's 's Revenge, skills found by our method achieve zero-shot performance that takes millions of steps to learn in the classical reinforcement learning setup.
We believe our algorithm is one step in a direction that may one day allow reinforcement learning agents to fully understand an environment without making use of any predefined reward function. Just like in Computer Vision and Natural Language Processing, this will lead to agents that need very few labels from the task at hand to be able to solve it and will drastically expand the applicability of reinforcement learning.
Acknowledgements
We would like to thank Xun Zuo for helping with the implementation of the maze environment, on top of many great discussions. We would also like to thank Frederik Benzing and Yassir Akram for many fruitful discussions. Finally, we also thank the anonymous reviewers for the thorough and insightful reviews and discussions. Robert Meier and Asier Mujika were supported by grant no. TRSII5_17377 of the Swiss National Science Foundation.
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firs international conference on Machine learning. pp. 2004.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery 2018
algorithms. arXiv preprint
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-2631. 2019.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbecl, and Wojciech Zaremba. Hindsight experience replay. arXiv 2017.
preprint
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv: arXiv: 1909.0728. 2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information processing systems, 29:147 1-7499999479 2016.
10

---

Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06600, 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http::/githhie
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04333 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /8/0. /2894, 2018b.
Andres Campero, Roberta Raileamu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint (rXiv:2006.12122 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2/04. 14294, 2021.
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment as representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01404, 2021.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096, 2020.
Yali Du, Lei Han, Meng Fang, Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. 2019.
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(78447:580-586 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 1802.0007, 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. arXiv preprint arXiv: /04.::/70403012.. 2017a.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017b.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Sébastien Forestier, Rémy Portelas, Yoan Mollard. and Pierre-Y Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint arXiv: 02190 2017.
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. 2021.
11

---

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 1rX1v:1710.11245: 2017.
Alex Graves, Mare Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference machine learning, pp. 1311-1320. PMLR, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.0755 2016.
Shixiang Shane Gu, Manfred Diaz, Daniel C Freeman, Hiroki Furuta, Seyed Kamyar Seyed Ghasemipour, Anton Raichuk, Byron David, Erik Frey, Erwin Coumans, and Olivier Bachem. Brax- lines: Fast and interactive toolkit for rl-driven behavior engineering beyond reward maximization. arXiv preprint arXiv:21 2021.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint WwXiv:/4/2.6990 2014.
Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-awant rewards for outcome-driven reinforcement learning. In International Conference Machine Learning, pp. 6346-6356. PMLR, 2021.
Tambet Matüisnn, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9):3732-3740, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Shakir Mohamed and Danilo Jimenez. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. arXiv preprint arXiv: (10.:v::5599087550 087319 2015.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl. Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807.04744 2018
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. arXiv preprint arXiv:2003.04960, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2788 PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062-5071. PMLR, 2019.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self superviee reinforcement learning. arXiv preprint 0rXiv/9933003666 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference Robot Learning, pp. 835-853, PMLR, 2020a.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmaan, and Pierre-Yve Oudeyer. Automatic curriculum learning for deep rl: short survey. arXiv preprint arXiv:2003.0448 04664, 2020b.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. arXiv preprint arXiv:2002.12299) 2020.
12

---

Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Juergen Schmidhuber. Hindsight policy gradients. arXiv preprint arXiv:1711.06000 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprini 0:XXvv:907.0165) 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.018855 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354 359, 2017b.
Avi Singh, Larry Yang. Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854 2019.
Rupesh Kumar Srivastava, Bas R Steunebrink, Marijn Stollenga, and Jürgen Schmidhuber. Continu- ally adding self-invented problems to the repertoire: first experiments with powerplay. In 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL),
pp. 1-6. IEEE, 2012.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 157.0004, 2015.
Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trehacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2 /2808, 2021.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self self-play. arXiv preprint 17X1v:1703.054077 2017.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05995 2019.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. arXiv preprint 09605, 2018
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. arXiv preprint arXiv: (ate::9999999944 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint wwXhv:1901.01750 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley, Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference Machine Learning, pp. PMLR, 2020.
David Warde-Farlcy, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.1133 2018
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. arXiv preprint 2xxv:2006.0941 2020.
13

---

Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. arXiv preprint arXiv:184400445, 2018.
14

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss limitations and future directions of research in the Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [No] We developed a new approach for open-ended learning. While we hope that our work moves the understanding of unsupervised open-ended processes forward in the long term, we do not expect any immediate societal impacts in our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theory
(b) Did you include complete proofs of all theoretical results? [N/A] No theory
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Included code, we will release it under the Apache 2 license or similar
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "ykWrjbSYuJD", "reviewer": "Reviewer_Wvtf", "summary": "The paper studies the problem of unsupervised reinforcement learning, where an agent aims to explore its environment without a reward signal and learn diverse skills. The proposed approach amounts to learning a reward function which encourages exploration, and a policy with A2C to maximize this reward. The exploration reward function itself is essentially a classifier trained to predict visited states (negatives) from states near the boundary of the visited states (positives). The policy is trained to maximize the score (go near the boundary). \n\nTo recover \"skills\", the policy checkpoints at various stages of training are used. At each phase, the reward function then policy are trained using the newly collected data. To stabilize learning, the policy/value function are reused from the previous skill in learning the new skill. \n\nExperiments show this can explore a maze environment as well as ant/humanoid environments. The proposed approach outperforms DIAYN in terms of mutual information between skill and state-distribution. ", "questions": "- If the policy at different stages of training are taken as \"skills\", why are all of the listed intrinsic motivation papers not valid comparisons?\n- How does the proposed method for state coverage differ/compare to the relevant work? There should be comparisons to at least some recent work which also does exploration+skill discovery by optimizing for state coverage.", "limitations": "Limitations are not really discussed.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The paper studies an important problem in unsupervised RL and skill discovery. \n- Expanding the policy to cover the state space is a well motivated objective for unsupervised RL.\n\n*Weaknesses*\n- First, the way the proposed method decomposes \"skills\" is a bit unclear. From what I can tell, the method is training a reward to explore the boundary of states, and a policy to maximize that reward, both being updated online. Then \"skills\" as denoted in this work are just the policy/reward at different checkpoints in training. So it is less that the method learns a multi-skill conditioned policy, but rather it is a collection of checkpoints (which get manually selected during downstream 0-shot adaptation). \n\nIf this is the case, then the claim in the related work that intrinsic motivation papers are not comparable is not quite accurate.\n\"Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Raileanu & Rocktaschel, 2020). These approaches have managed great success in hard-exploration Atari games. However, these approaches do not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks.\"\nAny of these works could also produce skills via the same checkpointing procedure used here and thus are valid comparisons. \n\n- Moreover, the core exploration objective proposed in this work is not particularly new compared to prior work. Theres a large literature of works that use state-space coverage as their exploration+skill learning objective, including model error/disagreement [1,2,3], pseudo counts [5,6], state distribution matching [7], and exploring the boundary of states [4,8] to name a few. While certainly not all are necessary, the paper should at least compare to some of these baselines, and currently none are. And the paper should do a better job of explaining how exactly their instantiation is different from what is being proposed in these works.\n\n1. Pathak et al. Curiosity driven Exploration by Self-Supervised Prediction. \n2. Pathak et al. Self-Supervised Exploration via Disagreement. \n3. Sekar et al. Plan to explore vis Self-Supervised World Models\n4. Mendonca et al. Discovering and Achieving Goals via World Models\n5. Tang et al. # exploration: A study of count-based exploration for deep reinforcement learning.\n6. Burda et al. Exploration by Random Network Distillation.\n7. Lee et al. Efficient exploration by state marginal matching.\n8. Bharadhwaj et al. LEAF: Latent Exploration Along the Frontier. \n\n- Finally, there are a number of design choices in the proposed approach that are unclear. For example, why does random exploration from the current policys states produce a good set of positive examples for exploring beyond it. I can think of cases (like going through a narrow gap), where such exploration will not hit the states relevant for pushing the boundary. Approaches like those used in [4,7,8] seem like they could be more effective are identifying the boundary to explore.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TgQvw8hdmKd", "reviewer": "Reviewer_tSXg", "summary": "The authors propose a method for unsupervised skill discovery via reinforcement learning. The method involves training a sequence of neural network-based reward functions, and using RL to learn policies that optimize these reward functions.  The reward functions are trained using positive and negative examples, where negative examples are the states reached by current policy, and positive examples are states reached by executing random actions. This encourages learning skills that cannot be visited by the current policy (and all previous policies). The authors apply this method for learning dozens of skills in simulated physics environments (half-cheetah, humanoid etc.), and hundreds of skills in Montezuma’s revenge. The skills often (but not always) recover interesting behaviors, including a humanoid hopping on one leg and the agent in Montezuma’s revenge being able to explore new rooms (beyond the first room). ", "questions": "Questions\n- “In expectation, 7 · 10^27 episodes are needed to do so. This shows that this maze is difficult to navigate.” How did the authors arrive at - this number? \n- The ablations were only performed for the 2D navigation domain. Is there a reason why they were not done for the robotics domain? Since the two domains are fairly different, it is not necessary that all conclusions from the 2D navigation experiments will hold. ", "limitations": "The paper lacks section on limitations (there is maybe only one or two lines in the conclusion that mention limitations, and they only do so in the context of future work that could be done). I would encourage the author to come up with a more comprehensive discussion of limitations (I mention some in the weaknesses above, but I would guess the authors are probably aware of more limitations). ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Simple, straightforward method. \n- Impressive results in a few different domains. \n- Clear writing and presentation. \n\nWeaknesses\n- The authors only show results on zero-shot transfer (i.e. how well a learned skill performs on some standard reward function), and there are no experiments on using the learned skill(s) for quickly learning a new task (i.e. improving beyond the pre-trained skill).\n- The method is prone to learning uninteresting skills / getting stuck in local minimas. For example, in the experiments for Montezuma’s revenge, the authors noted that one of the first skills learned by the agent was to get the life counter to zero, which made it very difficult to do any further exploration (since the episode can terminate very easily once the life counter is at 0). This led the authors to manually intervene in the learning process and hide the part of the screen that shows remaining lives.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S6cSnxiJBg", "reviewer": "Reviewer_uUdP", "summary": "This paper presents a method for unsupervised skill discovery in reinforcement learning environments. The authors propose a method which iteratively learns policies for new skills by defining new reward functions for each new skill. New reward functions are constructed by taking the visited states by the previously learned skill, as well as all previously learned skills, and setting a low reward at those states. Rewards are higher for states which are close to being reached by the previous policy. The method takes advantage of some policy and value function transfer techniques to make learning more efficient. Experiments are conducted using a 2D navigation environment, robotics environments simulated using BRAX, and Montezuma’s Revenge.\n", "questions": "- Is the memory consumption of the method limiting? The reward function fitting seems to require keeping track of *all* of the previously seen negative states for all skills, which would grow as training continues. Additionally, as the number of negative states grows, does the objective in the supervised loss for the reward function need to be rebalanced for positive and negative classes?\n- For Figure 2, I don’t quite understand what the meaning of the colored circles is. My understanding is that for each skill, a policy is learned which brings the agent from the top left further and further through the maze. But “The circles represent the average position of the most visited locations for each skill.” Does this mean that for a particular skill, the agent actually goes further into the maze than its circle is, and the circle just represents the average position for states visited during that trajectory?\n\nNitpicks/typos:\n- 268: Unsupervised -> unsupervised\n- Left quotes in several places in the manuscript are flipped\n- Can the number of samples used to learn for Montezuma’s revenge/wall clock time be reported?\n", "limitations": "The limitations are mentioned in the conclusion, but it could be nice if the authors addressed the limitations of the current method more explicitly. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of the paper:\n- The method is straightforward and appears to make fewer assumptions than other methods like DIAYN do in terms of the x-y position prior, which is a very nice property. \n- Evaluations are performed on sensible tasks and can obtain favorable results compared to DIAYN and GCRL, which is quite impressive given that the method is conceptually simple. \n- Generally the paper is clearly written; I note some suggested small changes below, but there are some other minor grammatical errors which do not impact the clarity of the text but could benefit from another round of proofreading.\n\nWeaknesses/clarifications:\n- It seems that this work falls into the category of exploration with intrinsic curiosity. The reason is that while this method does generate pairs of discrete skills and policies whereas most exploration algorithms do not, the skills learned in this work aren’t necessarily disentangled. And while the authors note that “However, these approaches do not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks”, it’s not clear from this work how the discrete skills learned here can be composed or fine-tuned for completing a new task. The zero-shot transfer task investigated in the experiments requires that a *single* learned skill can solve the entire task. \n- As a follow-up to the previous point, it would be helpful to see more comparisons in the experimental section not only to additional unsupervised skill learning methods like DADS, but also intrinsic exploration methods like those mentioned in the related work section. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LLsbYkdKD4o", "reviewer": "Reviewer_GsFo", "summary": "The paper introduces a method for unsupervised skill discovery using learned reward functions. Pairs of reward function and policy pairs are iteratively learned. The reward function evolves by predicting negative rewards for states the agent has visited and positive rewards for states just outside the agent's reach. The policy is then transferred to the new reward using three forward skill transfer methods.\n\nThis new method is the first unsupervised skill discovery approach to work in high-dimensional environments without expert knowledge. The paper first analyzes a 2d maze task. Next, the paper shows results in continuous control tasks and compares to 3 baselines. Finally, the paper shows results on Montezuma's revenge from pixel inputs.", "questions": "* Why use a regression loss for training the reward function? Isn't this a classification problem for distinguishing the already visited negative states from the new positive states?\n* I do not understand the justification for clipping the rewards to [0,a]. Could the negative rewards be important penalties for avoiding exploring parts of the state space that were already visited? \n* How does the training compare to the baselines from Gu et al. 2021? Do these baselines also do such extensive hyperparameter tuning to make this comparison fair? \n* What is the strategy used to adapt the number of training steps to ensure each skill has been learned in Montezuma's revenge?", "limitations": "The authors mention limitations of the work with the potential for better forward transfer methods and using a population of skills rather than the narrow deep search strategy. \n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* To the best of my knowledge, learning a reward function for unsupervised skill discovery is novel. I believe this is an important step forward from prior work that relies on more hand-crafted reward objectives. \n* Unlike other skills discovery methods like DIAYN, the proposed method does not require engineering only part of the observation space as input for good performance.\n* Method demonstrates superior performance over relevant unsupervised skill discovery algorithms both in terms of the MI metric.\n* Qualitative analysis demonstrates the behavior of skills learned in all three benchmarks.\n* The algorithm does not need a specified number of skills ahead of time. New skills can continuously be added.\n\nWeaknesses:\n* While this weakness was also acknowledged in the limitations section, the proposed method learns skills in a depth first rather than breadth first way. This narrow search of skills could result in lacking skill diversity. Imagine a version of the 2d maze environment where there are multiple possible paths. I think the proposed method would find skills that only go along one path, but never discover skills down the other path because the skills progressively build off each other.\n* Several design decisions are not quantitatively evaluated. The impact of adaptive entropy regularization and the three forward transfer methods are not ablated. Furthermore, there are several arbitrary hyperparameters related to the guiding phase like the duration of the guiding phase and the time spent collecting negative actions.\n* Only the continuous control results include a comparison to baselines. It is unclear if the method helps in other tasks like maze navigation or Montezuma's revenge. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept