Unsupervised Skill Discovery via Recurrent Skill Training
Authors: Zheyuan Jiang
Jingyue Gao
Jianyu Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Reinforcement Learning
Unsupervised Skill Discovery
Deep Learning
Unsupervised Reinforcement Learning

Abstract: Being able to discover diverse useful skills without external reward functions is beneficial in reinforcement learning research. Previous unsupervised skill discovery approaches mainly train different skills in parallel. Although impressive results have been provided, we found that parallel training procedure can sometimes block exploration when the state visited by different skills overlap, which leads to poor state coverage and restricts the diversity of learned skills. In this paper, we take a deeper look into this phenomenon and propose a novel framework to address this issue, which we call Recurrent Skill Training (ReST). Instead of training all the skills in parallel, ReST trains different skills one after another recurrently, along with a state coverage based intrinsic reward. We conduct experiments on a number of challenging 2D navigation environments and robotic locomotion environments. Evaluation results show that our proposed approach outperforms previous parallel training approaches in terms of state coverage and skill diversity. Videos of the discovered skills are available at https://sites.google.com/view/neurips22-rest. 

Unsupervised Skill Discovery via Recurrent Skill Training
Zheyuan Jiang!
Jingyue G
Jianyu Chen 4.3
Institute for Interdisciplinary Information Sciences, Tsinghua University Department of Computer Science, Tsinghua University
3 Shanghai Qizhi Institute
Abstract
Being able to discover diverse useful skills without external reward functions is beneficial in reinforcement learning research. Previous unsupervised skill discovery approaches mainly train different skills in parallel. Although impressive results have been provided, we found that parallel training procedure can sometimes block exploration when the state visited by different skills overlap, which leads to poor state coverage and restricts the diversity of learned skills. In this paper, we take deeper look into this phenomenon and propose novel framework to address this issue, which we call Recurrent Skill Training (ReST). Instead training all the skills in parallel, ReST trains different skills one after another recurrently, along with a state coverage based intrinsic reward. We conduct experiments on number of challenging 2D navigation environments and robotic locomotion environments. Evaluation results show that our proposed approach outperforms previous parallel training approaches in terms of state coverage and skill diversity. Videos the discovered skills are available at https:: //s::///sies gowgle..........oo com/ view/neurips22
Introduction
Recent advances deep reinforce-
ment learning have shown its promis-
ing performance in domains ranging
from game playing [2] 3. robotics [4]
and recommender systems 6 These applications of reinforcement learning rely on task task-specific reward functions for the agents to success- fully accomplish the tasks. However, automatically explore the environments and learn di- useful skills in the absence of ex- ternal supervision. Such ability is ben- eficial in variety of situations. For tasks where rewards are non-trivial to design where the reward signal is sparse, unsupervised skill discovery approaches can provide intrinsic re-
(a) Explored
(b) Converge
(c) ReST (ours)
Figure 1: Skills discovered in a 2D navigation environ- ment.(a) shows the explored states of skills during the train- ing phase of DIAYN [I where two skills (red and blue) have successfully passed through the bottleneck and explored the states on the right room. (b) shows the states covered by the converged skills of DIAYN, which does not reach states the right, indicating that the two skills (blue and red) visiting the same state in (a) discourages both skills from visiting that state later. Our proposed method, as shown in (c), success- fully reached the states the right after convergence.
intelligent creatures
"Equal Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

wards to help accomplish tasks. Moreover, in hierarchical control problems, unsupervised skill discovery can serve as low-level policies for downstream tasks
Although existing works have shown great potential in discovering diverse useful skills in an unsu- pervised manner 3 one of the key problems of such methods, observed in preliminary experiments and some recent works [ To.. is that they might suffer from poor state coverage, which may lead failures in learning desirable useful skills. For instance, in robotic locomotion environments, previous approaches tend to learn "posing' skills instead of dynamic, far-reaching skills M. A straightforward explanation for this phenomenon would be the lack of exploration [9]. However, we argue that this is not the only reason for the poor state coverage. Counterintuitively, we observed that the skills after convergence may avoid visiting certain states even if they are explored during training. For instance. as shown in our preliminary 2D navigation experiments, the discovered skills fail to cover the states passing through the bottleneck to the right (Figure [b, even if they have been explored during training (Figure
a
We call this phenomenon Exploration degradation.
In this paper, we take a deeper look into the above phenomenon and show that it mainly caused by the parallel training paradigm, which is a common choice for most existing works [1 7 g I0]. When multiple skills trained in parallel have visited the same state, such state will be prevented from being visited again. Detailed analysis is provided in Section 3.1 Based on the analysis, we propose Recurrent Skill Training (ReST), an unsupervised skill discovery algorithm that addresses the exploration degradation issue. Instead of training all the skills in parallel, ReST trains the skills one after another in a recurrent fashion, along with an intrinsic reward that discourages covering frequently visited states of other skills. A preliminary result of using ReST is shown in Figure Ic where the exploration degradation problem is eliminated and the converged skills are able to visit the space in the right room. Evaluation results on complex 2D navigation and robot locomotion tasks show that our approach can achieve better state coverage and skill divergence compared to baselines.
Our contributions are summarized as follows:
We discover a new phenomenon that reduces state coverage called exploration degradation, which indicates that some certain states are discouraged from being visited by the learned skills, even if they have been explored during training.
We show that the main reason causing exploration degradation is that multiple skills visiting the same states can reduce the MI reward in the parallel training paradigm. We then propose Recurrent Skill Training (ReST), recurrent training paradigm along with a state coverage based intrinsic reward, which prevents multiple skills from visiting the same states and alleviates the exploration degradation issue.
We conduct experiments various 2D navigation tasks and robot locomotion tasks. Evalu- ation results show that our method achieves better state coverage and divergence compared baseline methods. Moreover, ReST learns diverse meaningful robot locomotion skills that have not been shown in previous works.
2 Preliminaries
2.1 Markoy Decision Process
Markov decision process (MDP) can be used to find a reward maximizing policy, It is represented by tuple (S, A, P, R, 7, 1),, where S, A are the state and action spaces. P: x x S [0, 1] is the transition dynamic that maps the state and action into a probability distribution over the next A IR is the reward function. y is the discount factor for the reward function while S - [0, 1] is the initial state distribution. The expected discounted cumulative reward can be formulated as (T) Ex 7" R(s,, a.)]. Thus the overall optimization problem can be written as xt arg max, JR(T).
state. R:
2.2 Unsupervised Skill Discovery
Generally speaking, unsupervised skill discovery aims to find a family of skills conditioned on latent 2, which results in a latent-conditioned policy rra 2) that maximizes the mutual information between state and latent:

---

= - = -((S) - H(SIZ)
H(Z[s)
(2)
where E S, a E A, 2 the state, action and latent respectively. Let (S p(s. 2) as the random variables the state distribution and the latent distribution. As suggested by [11]. variational lower bound for Equation I be derived
[( 2 B(z.s))p(z..) [log go((2)))))))))))))))))))(()))g [()]
where (2(2)) () is a learned discriminator approximating p(z]s). Such lower bound also exists for Equation
T(S: 2) 2 ((px)) (s(==) - p())
(4)
where (s(s))) a learned function approximator of p(s(2).
Recurrent Skill Training
Uniisisd states
Both visitd Janox
Vieiel by skill
Visiced by skill
1
1
0
1
2
3
3
2
3
(a) Explored
(b) Parallel
(c) Recurrent
Figure 2: Grid world example. The orange grids are states explored by both the two skills and the green and yellow grids denotes states explored by skill and respectively. Blue grids are unvisited states. (a) shows the state visitation map during exploration while (b) and (c) shows the state visitation of the final converged policies of parallel and recurrent training paradigms respectively.
In this section, we first introduce the exploration degradation phenomenon of previous skill discovery approaches with parallel training paradigm. Then we present our proposed recurrent skill training method that addresses this issue
Update
Update
Update
Update
Convergod
Trajectory skill
Trajectory of skill 2
Trajeccory of skill
Estiinated state visination distribution of skill
Estimated state visitution distribution of skill
Eatimated distribution of skill 3
Figure 3: Intuitive illustration of the ReST algorithm. Each color represents skill. The lines in the figures represent trajectory rollouts of skills whereas the shaded areas indicate the estimated state distribution of the skills. The figure on the left demonstrates the initialization of the three skills, where all skills' trajectories are entangled together. Then ReST starts with training skill by optimizing the reward function in Equation 7 which encourages the trained skill to stay away from the estimated state visitation distributions of other skills. ReST then continues to optimize skill 2, and goes back to optimize skill I recurrently. After convergence, the skills would be spreaded out to cover diverse states as shown in the figure on the right.

---

Algorithm Recurrent Skill Training
Initialize: random initialized skill x, RND networks f and f
for i = to N do
Set skill Ty = IT
Set RND network f
Collect on-policy samples with RT
Update RND network ft by minimizing loss in Equation 5 end for
repeat
for = 1 to N (N is the number of skills) do
for j to M (M is the number of training epochs for each skill) do
Collect on on-policy samples with Tj:
Calculate reward using Equation 7
Update skill policy т, using any RL algorithms;
Update RND network fú by minimizing loss in Equation S samples; end for end for until convergence
using the latest on-policy
3.1 Exploration Degradation
As we observed in our preliminary experiment shown in Figure I previous unsupervised skill discovery approaches might suffer from the exploration degradation problem, such that some explored states (e.g, states near the bottleneck and in the right room in Figure Tap are prevented from being visited by the learned skills. We now provide simplified analysis to show that this phenomenon is mainly caused by the parallel training paradigm commonly used in previous works.
Let us focus on discrete latent case with N different skills. Consider at state s, skill k visited state with p(au|s) probability, which is modeled by the discriminator go ( s). Assume state so is only visited by skill i whereas $1 is visited by multiple skills, which means p( # (so) = and p( p(zu(s)) < 1. As long as the discriminator qs can model the difference and output go Zi Su) > go (0( (1)) which is not a strong assumption, the intrinsic reward used by MI approaches would encourage the visitation of state 80 by skill and discourage visiting 81. This means MI based approaches end up with skills that prefer exploiting states only visited by themselves in the previous training epochs, causing exploration degradation.
For clarity, we further explain this issue with a toy example. Consider 2 x 2 grid world as shown in Figure 2 There are four states 0, 3 and at each state, there are three actions to take: go to the two adjacent states or stay where it is. For simplicity, we use the number of the next state to denote the action, for instance, if the agent in state and chooses to go to state 1, then the action will be denoted as 1. We use the mutual information described in Equation (I) and assume that the discriminator is perfect: (2(s) Px (2|s). Consider a case when the number of skills N and during the first collection of samples, skill visited {0, 2, 3} while skill 2 visited {0, 1,3). as shown in Figure 2a Since we have perfect diseriminator, for state the discriminator would output (2 (3) (2g = 0.5, which results in 0 reward for both skills. Therefore, the optimal converged policy for skill would generate trajectory {0, 2, 2} while skill 2 would generate trajectory {0, 1, 1). shown in Figure 2b State 3 is not covered, which is undesirable. This example indicates that even in such an extremely simple case, the exploration degradation phenomenon still exists.
3.2 Recurrent Skill Training
Instead of training all skills in parallel, we propose recurrent training paradigm, along with a state coverage intrinsic reward. We now introduce the details of our proposed method.
Recurrent Training Paradigm. As analyzed in Section 3.1 the main reason causing the exploration degradation issue is that the same states are visited by multiple skills in the parallel training paradigm. A natural way to alleviate this issue is train the skills one after another recurrently. In this way,

---

we can encourage the latter trained skills to avoid entering the same states covered by the previous skills. Figure iillustaate the recurrent training paradigm. Starting with N randomly initialized skills (N = 3 in this case), the recurrent training paradigm trains one skill at each epoch whereas the parallel training paradigm trains the skills together. Furthermore, recurrent training paradigm updates each skill for M epochs (M = 2 in this case) before switching to another skill. In this work, we use N independent neural networks to parametrize the N skills, which can be considered as discrete latent conditioned policies.
order to improve convergence, the
State Coverage Intrinsic Reward. When implementing the above recurrent training paradigm. the latter trained skill needs to avoid visiting the states frequently visited by other skills. In order to accomplish this objective. we need identify how frequently each state is visited by each skill, or equally, how novel a given state is to a skill. In this paper, we adopt random network distillation (RND) [12]. a simple yet scalable novelty detection approach, to estimate the novelty of state to a specific skill.
Random network distillation (RND) mainly involves two networks for state novelty detection: randomly initialized target neural network whose parameters are kept fixed throughout the whole training process and predictor network trying fit the target neural network. The target network f: S + R where k is the dimension of the output vector, maps a state to a vector while the predictor network f S - R& is trained to minimize the expected mean square error between the output vector and the target output vector: [If(s) f(s) which distills the target network into the predictor. Therefore, the predictor should have low prediction error on the data it is trained on while have high prediction error other states. We call such target network and predictor a pair of RND networks.
We assign each skill with a pair of RND networks. Each skill's predictor network is trained on the the on-policy data it visited during the last rollout, which means for each skill with latent 2j E 2, where E {1, 2, ..., N} is the index of the skill, with state distribution 8 ~ p(s]2;), we can train the predictor network using gradient decent by minimizing the mean square error loss Li
Ls = Egrp(eles ( [
(5)
As mentioned above, the learned predictor network's prediction error [[f{()) fi(s) can indicate visitation frequency of a certain state 8. Intuitively, a higher prediction error indicates higher uncertainty of f. on this state, which further implies its higher novelty. Since we need to avoid visiting states visited by other skills when training certain skill, straightforward solution is to define a reward function for skill i with state 8t and action de as:
min T0 0x) jE(1,2,...N Ji - (+(+1))]
such that states frequently visited by any other skills are less desired. However, this minimum operator would make the reward landscape rugged, which might lead to poor convergence property. To stabilize the training process, we introduce a soft version of the minimum operator in 6
rs(3;; az) = -lg Σjel NJJ4 Ejst e(-a-1l (( {1( MIP)
where a is task-specific temperature parameter.
Practical Algorithm. We summarize our algorithm in Figure 3 and Algorithm [ Firstly, we randomly initialize policy network and pair of RND networks fü and fü for each skill E {1, (1,2.... .N.. Before training skills, ReST first collects on- on-policy samples for each initialized skill and train their RND networks. Then we train each skill's policy network and the corresponding RND networks for M times during each training epoch. The skill networks are trained recurrently until convergence. For maximizing the intrinsic reward, the ReST algorithm can be combined with an arbitrary reinforcement learning algorithm. this paper we choose Proximal Policy Optimization (PPO) advantage estimation (GAE) [4]
[K] with generalized
5

---

Grid World Example. We further illustrate the effectiveness of the proposed algorithm using the
2 x 2 grid world example with the same setting as Section 3.1 As suggested by our previous analysis, the parallel training paradigm will end up with two skills that neither of them visits state 3. As mentioned in Section 3.1, the initial two skills visits (0, 2,3 and {0, 1,3) respectively. Here we assume that the RND networks f. and fi perfectly obtain the state visitation frequency, which means:
[lfi(s) -(())) f(())]1 {
if state & is visited by skill otherwise
(8)
where 0 is the prediction error. ReST starts by training the corresponding RND networks of each skill and then recurrently train different skills to maximize the intrinsic reward. For skill 1, visiting state would gain reward while visiting state and 3 would get zero rewards. Therefore, the optimal policy for skill would be visiting {0, 2, 2} in sequence. After updating the RND networks of skill correspondingly. for skill 2, visiting state and 3 would gain reward r while visiting would get zero rewards. Therefore, the initial skill 2 is already the optimal policy and the training converges. This way, the optimal trajectory for skill is {0. 2, 2] while for skill 2 is {0, 1, 3). as shown in Figure 2c Therefore, the state space of the 2 x 2 grid world is fully covered by the two
skills.
Experiments
ReST (ours)
DIAYN
DADS
DIAYN-
DADS-i
SC
MI
044
6.1
(a) DoorMaze
0A
CenterMaze
622
(c) 4RoomMaze
(d) 9RoomMaze
Figure 4: Results for 2D navigation experiments. The left five columns of the figure show qualitative results of skills discovered by different algorithms in the corresponding navigation environments. We trained 10 skills for each algorithm where 20 trajectories are rendered for each discovered skill. The right two columns of the figure show the quantitative results of different algorithms. SC stands for state coverage while MI stands for mutual information. SC and MI are calculated based on converged models of each algorithm with three different random seeds. We calculate the SC and MI metrics three times for each converged skill to draw the histogram. Detailed analysis of the evaluation metrics can be found in the Appendix
conduct experiments on several challenging 2D navigation environments and several robotic locomotion tasks. We compare our proposed approach with two of the most popular unsupervised

---

(a) HalfCheetah flipped (b) HalfCheetah rolling (c) HalfCheetah rolling (d) HalfCheetah running running forward
forward
backward
backward
(c) Hopper hopping (f) Hopper crawling for- (g) Hopper hopping back- ward
(h) Hopper backflip
ward
ward
MAITIILLT MAtHHUI
(i) Walker2d trotting for- (j) Walker2d dashing for- (k) Walker2d ward
digging (1) Walker2d walking backward
ward
backward
Figure 5: Visualization of skills discovered using ReST. The proposed ReST algorithm discovers several high-quality, diverse skills for each robot, including running, flipped running, backflip,
dashing, etc. The arrows the bottom of each sub-figure show the moving direction of each skill.
RcST (ours)
DIAYN
DADS
DIAYN-i
DADS-i
330
100
(a) Half Cheetah
22
20
5/0
150
1000
251
352
1192
255
500
1000
251
533
3391
335 1005
Hopper
30
12
(c) Walker2d
Figure 6: Semi-quantitative results for robotic locomotion tasks. The horizontal axis represents the time-step during evaluation while the vertical axis represents the position of the robot at each time step. Each color represents a discovered skill and we rollout three trajectories for each skill. Generally speaking, baseline methods discover 'posing' skills instead of dynamic locomotion skills whereas ReST discovers dynamic, far-reaching locomotion skills.
skill discovery approaches, DIAYN ① and DADS 7 Both of these approaches the parallel training paradigm to optimize different skills. Since our proposed approach parameterizes each skill with an independent neural network, we also compare our proposed approach with DIAYN and DADS using independent neural networks, which we call DIAYN-i and DADS-i respectively, for a fair comparison. We present our empirical results both qualitatively and quantitatively. The quantitative results introduce state coverage metric and a mutual information metric. Our proposed approach significantly outperforms previous skill discovery approaches terms of state coverage while staying comparable with previous approaches in terms of mutual information. Qualitative results include visualizations of the converged navigation experiments, the state visitation of robotic locomotion tasks, and rendered videos of novel skills discovered using our proposed approach, which can be found in our project website implementation can be found in the Appendix.
wwwees.gool..coomm con/view// http://vwwweeeeeeiiieee neur ips22- pss22-rett rest Details of
7

---

4.1 2D Navigation Tasks
Environments. We conducted experiments on several challenging 2D navigation environments. The agent is a point mass navigating in a 2D plane with boundaries [0, 1] x [o. 11. The observation space of the environment has 4 dimensions, including the x E [o, 1] and E [0. 1] position on the plane and the corresponding velocity Vy and By belonging to the velocity space {(e,: uy) + +g < 0.013. The action space has 2 dimensions, including the acceleration a, E [o,, 0.1] and ay E [0, 0.1] of the point mass agent. There are walls inside the plane and the agent cannot go through the walls. We designed diverse placements of the walls with increasing difficulties for exploring the environments, which could be used to test the effectiveness of our proposed approach.
Evaluation Metries. We introduce two metries to compare our proposed approach with our com- parison baselines. The first one is the state coverage metric. State coverage matters since not covering enough state space might result in failures in learning desirable useful skills. We evaluate the state coverage on the X-Y plane by first decomposing the environment into cells and then testing whether each cell is successfully visited. We roll out 1 trajectory for each skill and use the visited states to calculate the state coverage metric. The percentage of cells visited by at least one of the skills the state coverage. Moreover, merely covering the state space is not enough. The skills need to be informative about which states they are going to visit in the environment so that the skills are meaningful. We use mutual information between skill latent & and the corresponding covered states to quantify how informative the skill is. Similarly, we decompose the state space into multiple cells and roll out 20 trajectories for each skill to record the probability distribution of states p(s) for all the states visited by the skills. Then record the probability distribution of states visited by each skill p(s{z) and calculate the entropy (S) and H (S))Z)) Since the skill distribution p(z) is uniform distribution for all skills, we can calculate the mutual information as 2) HS) + (ZiE(1......)) (S().))
Qualitative Results. We train N = 10 skills for each of the algorithms and rollout 20 trajectories to obtain the qualitative results. We conduct experiments on 4 challenging 2D navigation mazes, which we call the DoorMaze, CenterMaze, 4RoomMaze and 9RoomMaze, which are shown in Figure 4 The left 5 columns show the qualitative results. Each rolled-out trajectory is rendered as a curved line and different colors represent different skills. The results indicate that our proposed ReST algorithm can reach more diverse states in the environments whereas baseline approaches like DIAYN or DADS can only reach a small portion of the environments. Usually, the baseline approaches cannot pass through the 'bottlenecks" in the environment. The results support our insight that the parallel training paradigm is one of the causes of previous skill discovery approaches' state coverage issues. Moreover, when using independent neural networks to parameterize different skills, we observe that the performance is not so different from the intentcconditionel parameterization version.
Quantitative Results. Beyond qualitative results, we quantify the state coverage and the informative- ness of different algorithms using the aforementioned two metrics. As shown in the right two columns of Figure 4 our proposed ReST algorithm significantly outperforms baseline approaches with parallel training paradigm in terms of state coverage and is comparable with baseline approaches in terms of mutual information I [S; Z). The above results indicate that our proposed approach can cover diverse sets of states without too much sacrifice on informativeness. Quantitative results also evidenced that the difference in performance is not because of the different parameterization since DIAYN and DADS using independent neural networks are not so different from the original ones.
4.2 Robotic Locomotion Tasks
We present qualitative results of robotic locomotion skills discovered using our proposed ReST algorithm in MuJoCo (15]. Generally speaking, our proposed ReST algorithm can discover dynamic, far-reaching robotic locomotion skills whereas DIAYN and DADS tend to discover 'posing' skills. Figure 5 shows visualizations of parts of skills discovered using ReST. There are several novel skills discovered, such as Hopper backflip. that have not been presented in previous works to the best of our knowledge. More rendered results can be found in the Appendix and our project website http::://sites.googl.. wwtpv/ewwccoiviie/eeeeeeee wwwpee/wwwwrrrrieneeeeee
Moreover, we provide semi-quantitative results of the proposed approach. As shown in Figure t we draw the agent's position over timestep. using the skills discovered by ReST and the comparison
8

---

baselines. The results are evaluated on HalfCheetah, Hopper and Walker2d tasks. We use the OpenAI Gym [T settings of the three tasks, where HalfCheetah is trained with fixed episode length whereas Hopper and Walker2d terminate when the agents fall during training. The resulting timestep-a curve indicates that our proposed approach learns skills that are more far-reaching. diverse, and dynamic than baseline methods. This also evidenced that our recurrent training paradigm outperforms the parallel bhining-based baselines by alleviating their state coverage issues.
5 Related Work
Unsupervised Skill Discovery. Previous unsupervised skill discovery approaches mainly focus on maximizing the mutual information [ Z) to obtain meaningful skills. As discussed in Equation ① and Equation Q the mutual information has forms. Several works follow the Equation [ VIC 17 uses the mutual information between the final states and the skill latent as the intrinsic reward and optimizes it via reinforcement learning. DIAYN (1] fixes the prior p(z) and uses the mutual information between the skill and its visited states as its intrinsic reward to learn meaningful skills, which has superior performance compared VIC. VALOR N further improves DIAYN by replacing the state-based objective with a trajectory-based objective. DADS M. on the other hand, uses the Equation 2 form of mutual information and learns a transition model 4s (5'ss,))) and uses model predictive control to solve downstream tasks. EDL
i provides insight into why previous approaches suffer from lack of exploration and proposes an algorithm using fixed state prior p(s) that to alleviate the issues. IBOL [10] tries to relieve the difficulty of reaching diverse states by introducing low-level controller. Besides learning a set of skills, SMM [X] formulates skill discovery as state marginal matching problem and optimizes the KI divergence between the expected state distribution and the current policy's state distribution. MUSIC [9] improves previous unsupervised skill discovery algorithms by adding the mutual information between the surrounding state and the agent state. DDL 20 learns one skill that maximizes the dynamical distance functions of the previous skill. Plan2Explore (2] makes use of self-supervised reward signals to optimize one policy to efficiently gather state-covering dataset. In contrast, ReST optimizes multiple skills and maximize online state coverage. LEXA [2 learns
condiiond policy using self-supervised reward. RcST, on the other hand, does not require the agent to achieve specific goals, which is more general formulation. Besides the above approaches, there are also other unsupervised skill discovery methods 22
24 25 26
Intrinsic Reward. Another stream of works related to this work is intrinsic reward. In our proposed approach, we use intrinsic reward as an objective to help learn set of meaningful skills. Intrinsic reward can also help with cases where rewards are sparse by augmenting them to the original reward function. Count-based exploration [27 29 30 uses pseudo count to identify the frequently visited states and the less frequently visited ones and adds the count-based bonus as intrinsic rewards to accelerate exploration. Prediction error exploration methods [3] 2 prediction errors as intrinsic rewards based have higher novelty. Intrinsic reward can also benefit novelty sceking tasks 355 36]
33 KE!
make use of
insight that states with high prediction error should where RSPO [35] made use of an iterative paradigm when training different strategies, which is similar recurrent training paradigm in this work. Other works augment an information-theoreiic intrinsic reward with
extrinsic rewards that encourage information gain about the environments 37 38 39
6 Discussion
In this paper, we proposed a novel, effective yet simple algorithm called Recurrent Skill Training (ReST). We began by introducing a new phenomenon called exploration degradation which reduces state coverage of the learned skills. We found the key reason for this phenomenon is the parallel training paradigm commonly used in previous skill discovery approaches. such that the same states visited by multiple skills are discouraged from being visited again. Instead of training all skills in parallel at each epoch, ReST trains different skills one after another recurrently. This recurrent training paradigm is supported by an effective prediction error- error-based intrinsic reward inspired by novelty detection methods. We then conducted experiments on 2D maze navigation to continuous robotic control tasks. Both qualitative and quantitative results show that ReST is able to discover more diverse skills with better state coverage compared to baseline algorithms. Moreover, we demonstrated several novel and dynamic robot locomotion skills that have not been presented in previous works.
9

---

There are also some limitations of the proposed algorithm. First of all, compared with previous approaches, our proposed approach has worse sample complexity during skill discovery since only one skill is trained at each epoch, as shown in Figure E Moreover, the computational complexity higher than approaches like [ or [TO] since ReST needs to compute intrinsic reward based on all other skills' prediction errors. Finally, due to the recurrent training paradigm, ReST is currently not scalable to continuous latent, which is in general a better choice as a low-level controller for hierarchical control in downstream tasks. Future works include addressing the above limitations and apply ReST to downstreammhhicraccicaal tasks.
Acknowledgement
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), We thank Wei Fu and the anonymous reviewers for many insightful suggestions which improved the manuscript.
References
[1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint arXiv: (rXiv:/802.06000, 2018.
[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-553 2015.
[3] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre,, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):4844488 2016.
[4] OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-200 2020.
[5] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference Robotics and Automation (ICRA), pages 8248-8254 IEEF, 2019.
[6] Guy Shani, David Heckerman, Ronen Brafman, and Craig Boutilier. An mdp-based recom- mender system. Journal of Machine Learning Research, 6(9), 2005.
[7] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint 1rXiv:1907.01657 2019.
[8] Joshua Achiam, Harrison Edwards. Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 1.X.xv1807.11299 2018
[9] Víctor Campos. Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[10] Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2 106. 14305, 2021.
[11] David Barber Felix Agakov. The im algorithm: variational approach to information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
[12] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.1288 2018
10

---

[13] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:170.00637, 2017.
[14] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018.
[15] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[16] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint Ttff::6606.11500000 2016.
[17] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 161 I. .07507 2016.
[18] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint
arXiv:1906.05227..215 2019.
[19] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprini arXiv:2103. 0000000077...1 2021.
[20] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for supervsuee and unsupervised skill discovery. arXiv preprint arXiv:1907.0822: 2019.
[21] Ramanan Sckar, Oleh Rybkin, Kostas Danillidis, Picter Abbecl, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In International Conference Machine Learning, pages 3583- PMLR, 2020.
[22] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379-24397 2021.
[23] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736-6747, PMLR, 2021.
[24] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551 2021.
[25] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- non- panaamametrii discriminative rewards. arXiv preprint arXiv: 1.... 2018
[26] Pierre-Alexandre Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:21 2021.
[27] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
(28) Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pages 2017.
[29] Justin Fu, John D Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning, arXiv preprint 1rXiv:1703.01260 2017.
[30] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information processing systems, 29:1471-1477, 2016.

---

[31] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model buidd-nu neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222-227, 1991.
[32] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 1507::1507000/111 00814, 2015.
[33] Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint 17Xxv:7033000
2017.
[34] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference machine learning, pages 2778- 2787. PMLR, 2017.
[35] Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint arXiv:2204.02246, 2022.
[36] Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In International Conference on Machine Learning, pages 7483-7492. PMLR, 2019.
[37] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. arXiv preprint arXiv: 0967,,9 2016.
[38] Navneet Madhu Kumar. Empowerment-driver exploration using mutual information estimation. 2018. 2018.
arXiv preprint
[39] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. arXiv preprint arXiv:15090083// 2015.
[40] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 1 for details.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URLY? [No] The code would be released immediately after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes) See the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] We did not use existing assets.
12

---

(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes] See http::/sites: google. com/ www/voew//c////ccc www.://wwwwps//wr..... rest
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rQCWv-az2G", "reviewer": "Reviewer_nWfC", "summary": "An ideal scenario of unsupervised skill discovery is to acquire a set of skills that has large state coverage as well as diversity. To this end, this paper suggests optimizing these two criteria by (1) using the state visitation-based reward for exploration and (2) iteratively training skills in that each skill is different from other skills. \n\nThe paper identifies that the prior unsupervised skill discovery work discourages skills visiting the bottleneck states due to low intrinsic reward on the bottleneck states $r \\sim p(z_i \\vert s)$, resulting in the bottleneck state never visited by any skill eventually, which is called the \"exploration degradation\" problem. This paper claims that this problem comes from simultaneous skill training, and thus proposes to train skills one by one, not in parallel.\n\nThe empirical results demonstrate that the proposed method acquires skills that are diverse and identifiable as well as cover wider states, whereas prior works, DIAYN and DADS, only show limited state coverage.", "questions": "- Given that RND's prediction networks will become more accurate over time, the reward signal may become less informative about the state visitation statistics. Have the authors experienced this issue for experiments? Is there any way to prevent this?\n\n- Is there any stochasticity in the environment or skills?\n\n### Minor suggestions\n\n- Iteratively sounds more intuitive than recurrently.\n\n- In the abstract, \"parallel training procedure inherently discourages exploration\" is not intuitive. It would be great to briefly provide an intuition.\n\n- Figure 1 clearly explains the motivation of the paper. However, it does not have enough information to get the intuition without reading until Section 3. A bit more explanation about \"two skills (blue and red) visiting the same state (a) discourages both skills to visit that state later (b)\" and \"colors of trajectories represent skills\" would help understanding it.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The proposed iterative skill learning scheme is novel under the unsupervised skill discovery setting.\n\n- Experiments on the 2D navigation environments are intuitive and well demonstrate the advantage of the proposed method in terms of diversity and coverage of the learned skills, compared to the baseline methods (in Figure 1 and Figure 4).\n\n- The paper clearly explains its motivation, approach, and experiments. The paper is easy to read.\n\n- Both qualitative and quantitative results are well designed and explained.\n\n### Weaknesses\n\n- There is another direction of work for acquiring a goal-conditioned skill policy in an unsupervised manner [1, 2], which demonstrates its wide state coverage and applicability to complex locomotion as well as manipulation environments. Comparisons to these baselines would be necessary as these works have the same goal of covering a wider range of states but are not limited to a fixed number of discrete skills.\n\n- The proposed method mainly involves two changes from prior work: (1) iterative training for the diversity of skills and (2) the state visitation-based reward for exploration. To verify the effect of each component, it would be helpful to have results about the parallel training version of the proposed reward.\n\n- The proposed method is designed only for discrete skills, and scaling to continuous latent skills seems not trivial given the nature of iterative skill training.\n\n[1] Sekar et al. Planning to Explore via Self-Supervised World Models, ICML 2020\n\n[2] Mendonca et al. Discovering and Achieving Goals via World Models, NeurIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "rPuPHssHA-B", "reviewer": "Reviewer_kKQH", "summary": "Parallel training of skill discovery methods results in failure of learning some skills. This happens because certain states are never visited by the learned skills. Authors propose recurrent skill training, which involves training a particular skill for some epochs followed by the next skill for the next few epochs. \nThey further shape the reward according to the state coverage. This prevents multiple skills from visiting the same states. Further experiments on various 2D navigation tasks and robot locomotion tasks show the new method achieves better state coverage and divergence compared to baseline methods. \n", "questions": "How does one decide how many epochs to train a particular skill?\n\nGiven the current weakness it is difficult to accept the paper. If the authors provide good reasons why this work is important despite the weakness, I am willing to change my score. \n", "limitations": "Please address catastrophic forgetting when number of skills gets large. ", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper finds a clear and important weakness in the current skill discovery methods. And solves it with a \"somewhat\" novel solution. \n\nThe paper is well written and clearly explained. The experiments are also well done. \n\nWeakness: \nThe paper does come up with a solution to a problem. But the proposed solution creates more problems of its own. \n1) Proposed method has worse sample complexity compared to other baselines (while other methods have less state coverage)\n2) Does not work with a continuous latent\n3) Does not address catastrophic forgetting of skills as number of skills grow\n4) Needs at least 2*N networks, where N is the number of skills\n\n\nPost-Rebuttal:\nWhen we compare ReST against current methods in unsupervised skill learning, ReST falls short in multiple accounts.\n\nHas higher sample and computational complexity compared to other methods\nCannot handle continuous latent (while other comparative methods can)\nToo many moving pieces compared to other methods (Multiple networks for skills, for novelty detection)\n\nThus, it is difficult for me to recommend acceptance of the paper. I thank you for your efforts and response.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KVMPLpJZLFl", "reviewer": "Reviewer_UvQx", "summary": "This paper presents an algorithm, ReST for unsupervised skill discovery. ReST trains different skills in sequential, instead of training all of them in parallel. ReST can address the exploration degradation problem proposed by the authors. The efficiency of the proposed approach is demonstrated in various tasks.", "questions": "1. I think the baselines already encourage the diversity of learned skills. Can we develop better approaches to avoid exploration degradation without sacrificing sample efficiency?\n2. If the new skill is forced to visit the novel states, it might not able to visit some existing good states, which might be harmful to the exploration. What's your opinion on this?\n3. Can the learned skills help hierarchical reinforcement learning?\n4. Will the code be released?", "limitations": "The authors give a good explanation of the limitations part. However, the broader impact section is not provided.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I tend to give an accept at this moment, and I might change my opinions after reading the rebuttals.\n# Strengths:\n1. This paper proposes a novel problem in unsupervised skill discovery: the exploration degradation problem. I think this is an interesting problem and is worth an oral presentation for all the researchers in this field to study and understand.\n2. The ideas are well introduced and the motivations are quite convincing. I'm convinced by the toy examples provided by the authors. Figure 1 and Figure 2 are very convincing and the ideas are simple and clear.\n3. I believe in the effectiveness of sequential skill learning because it can reduce collisions of the learned skills as introduced by the authors.\n4. The provided videos of the discovered skills are convincing. They can show that the ReST can learn different skills efficiently.\n\n# Weakness:\n1. The proposed videos https://sites.google.com/view/neurips22-rest are not compared to the videos of any baselines. So I cannot evaluate the advantages of the ReST by seeing the video. I wish to see more comparison videos in the rebuttal phase.\n2. The proposed recurrent algorithm may consume much longer time than the baselines. So the comparison might be unfair to the baselines. I notice this in the limitation part, so I do not penalize this too much.\n3. The quantitative results are not enough. I only see some comparisons on the right of Figure 4. And the comparison cases are quite toy (e.g. in Maze scenarios). The authors should compare to more realistic cases (robotics) and list the numbers for ReST and the baselines. This could benefit future approaches.\n4. The writing quality can be improved. Refer to the following section.\n5. The code is not provided so it might affect the reproducibility of this work. Although code should not be a must, in the field of reinforcement learning, reproducibility is worth concern without code. \n6. The broader impact is not included in this paper. In the standard of Neurips submission now, (expecially this paper is concerned with robotics), this paper needs a broader impact section. \n\n# Writing Suggestions:\n1. FIgure 3's text and others are too small for publication quality.\n2. Figure 3 does not illustrate the algorithm well. It is too complicated and the authors should walk through Figure 3 (a) in the caption. In general, the ReST algorithm is simple and the authors should provide a simple diagram with clear examples as Figure 3. \n3. Eq. 3 to Eq. 7 are a waste of space. You can only leave one form of the information bound and remove other equations.\n4. Eq.1 is also a waste of space because it's too trivial.\n5. What are N and M in algorithm1? It should be self-contained. You can replace them via \"the number of epochs\", etc.\n6. Some parts are hard to follow. For example, in Ln 131->152, the authors are trying to adopt RND to learn novel skills. However, the descriptions are not organized well so it is hard to understand here if the readers do not know RND before. One suggestion is to discuss the function of the networks here, then dive into details. Another question is, why two networks are used? This question is briefly answered in Ln 147. But I'm not quite convinced by the function.\n7. Eq 10 is not illustrated well and there is no ablation study to support this equation. \n\n-------------------------------------\nTo colleague kKQH,\nI agree with you on the strengths parts. For the weakness part, you mainly point out some engineering drawbacks to implementing this paper. I'd like to share my opinions on your concerns:\n1. The sample complexity is not that important in this sort of problem. This should be a pure research project and we do not need to deploy the training on a real robot/ industrial application at this moment. The longer training epochs would not cause a longer inference time.\nBy the way, I think somehow you need that long complexity if you wish the skills are \"orthogonal\" or \"different\". Recall the algorithm of Standard Orthogonal in Linear Algebra, we need to find each orthogonal vector at each time to ensure that they are fundamentally different.\n2. The continuous part is another long-standing challenge. I don't think they can be solved easily in one framework and many so-called general algorithms can only work well for discrete or continuous problems. This should not be a reason for rejection.\n3. I don't quite understand the \"catastrophic forgetting of skills\", do you mean that the skills can hardly be diverse if you increase the number of skills? This should hold true for all algorithms in this domain, not necessarily a limitation of the proposed approach.\n4. The number of networks is fine and one may compress or prune them into one network. This should be an engineering problem. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept