AUTOMATIC CURRICULUM FOR UNSUPERVISED REIN- FORCEMENT LEARNING
Authors: Yucheng Yang
Tianyi Zhou
Tianhong Dai
Meng Fang
Mykola Pechenizkiy
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recent unsupervised reinforcement learning (URL) can learn meaningful skills without task rewards by carefully designed training objectives. However, most existing works lack quantitative evaluation metrics for URL but mainly rely on visualizations of trajectories to compare the performance. Moreover, each URL method only focuses on a single training objective, which can hinder further learning progress and the development of new skills. To bridge these gaps, we first propose multiple evaluation metrics for URL that can cover different preferred properties. We show that balancing these metrics leads to what a “good” trajectory visualization embodies. Next, we use these metrics to develop an automatic curriculum that can change the URL objective across different learning stages in order to improve and balance all metrics. Specifically, we apply a non-stationary multi-armed bandit algorithm to select an existing URL objective for each episode according to the metrics evaluated in previous episodes. Extensive experiments indifferent environments demonstrate the advantages of our method on achieving promising and balanced performance over all URL metrics.

Under review as a conference paper at ICLR 2023
AUTOMATIC CURRICULUM FOR UNSUPERVISED REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent unsupervised reinforcement learning (URL) can learn meaningful skills without task rewards by carefully designed training objectives. However, most existing works lack quantitative evaluation metrics for URL. but mainly rely on visualizations of trajectories to compare the performance. Moreover, each URL method only focuses single training objective, which can hinder further Iearn- ing progress and the development of new skills. To bridge these gaps, we first propose multiple evaluation metries for URL that cover different preferred properties. We show that balancing these metrics leads to what a "good" trajec- tory visualization embodies. Next, we these metrics to develop automatic curriculum that can change the URL objective across different learning stages in order to improve and balance all metrics. Specifically, we apply non-stationary aumti-armed bandit algorithm to select an existing URL objective for each episode according to the metries evaluated in previous episodes. Extensive experiments in different environments demonstrate the advantages of our method on achieving promising and balanced performance over all URL metrics.
INTRODUCTION
Reinforcement learning (RL) has recently achieved remarkable success in autonomous control Ki- umarsi et al. 2017) and video games (Mnih et al.] (2013). Its mastery of Go (Silver et al.) 2016)
and large-scale multiplayer video games (Vinyals al.J 2019) has drawn growing attention. How- ever, primary limitation for the current RI. is that it is highly task-specific and easily overfitting to the training task, while is still challenging to gain fundamental skills generalizable to different tasks. Moreover, due to the sparse rewards in many tasks and poor exploration of the state-action space, RL can be highly inefficient. To overcome these weaknesses, intrinsic motivations (Oudeyer & Kaplan 2009) have been studied to help pre-train RL agents in earlier stages even without any task assigned. The so called "unsupervised RL (URL)" does not rely on any extrinsic task rewards and its primary goal is to encourage exploration and develop versatile skills that can be adapted to downstream tasks.
Although URL provides additional objectives and rewards train fundamental and task-agnostic skills, it lacks quantitative evaluation metrics and yet mainly relies on visualizations of trajectories demonstrate its effectiveness. Although it can be evaluated through downstream tasks by their extrinsic rewards (Laskin et al. (0221)). this requires further training and can be prone to overfitting or bias towards specific tasks. A key challenge in developing evaluation metrics for URL is how to cover different expectations or preferable properties for the agent, which usually cannot be all captured by single metric. Recently, IBOL (Kim et al. 202T) introduced the concept of disen- tanglement to evaluate the informativeness and separability of learned skills. However, it does not consider other characteristies such as the coverage over the state space. addition, how to balance multiple metrics in the evaluation is open challenge. Therefore, it is critical to develop set of metrics that can provide a complete and precise evaluation of an URL agent.
In this paper, we take first step towards quantitative evaluations of URL by proposing set of evaluation metrics that can cover different preferred capabilities of URL, e.g., on both exploration and skill discovery. In case studies, we show that URL achieving balanced and high scores all the proposed metrics fulfills our requirements for a promising pre-trained agent. However, excelling on only one metric cannot exclude certain poorly learned URL policies.

---

Under review as a conference paper at ICLR 2023
In contrast to the ambiguity of evaluation metrics for current URL, the existing intrinsic rewards for URL are quite specific and focused, c.g., the noveltyluncerrainty of states (Pathak et al. 2017
Burda et al.,
2019; Pathak al. 2019), the entropy of state distribution (Lee et al. 2019 Mutti
let al. 2021
Liu & Abbeel 202 la), and the mutual information between states and skills (Gregor Eysenbach et al. 20T9b), which are task-free and can provide dense feedback. For most URL methods, each one only focuses on learning with single intrinsic reward. They mainly differ on implementations, e.g., how define the novelty, how to estimate the state entropy or mu- tual information. However, the quality of inplementations significantly depends on the modeling of the environment dynamics, which cannot be always accurate everywhere if the exploration only guided by single reward. For example, shown later, agents learning with single intrinsic reward for exploration could hindered from further exploration since its novelty approximation is lim- ited to local regions. Moreover, in order to achieve consistent improvement on multiple evaluation metrics and balance their trade-offs, training with a single intrinsic reward is not enough. Hence, it is necessary in URL to take multiple intrinsic rewards into account.
et al. 2017
In this paper, we leverage multiple existing intrinsic rewards and aim to automatically choose the most helpful one in each learning stage for optimizing the proposed multiple evaluation metries. This produces a curriculum of URL whose training objective is adjusted over the course of train- ing keep improving all evaluation metrics. Since the intrinsic reward is varying concurently with URI. on the fly, we apply a multi-objective multi-armed bandits algorithm to address the exploration exploitation trade-off, i.c., we intend to select the intrinsic reward (I) that has been rarely selected before (exploration) or (2) that results in the greatest and balanced improvement over all the metrics in history (exploitation). Specifically, we adopt Pareto UCB (Drugan & Nowe 2013 to optimize the multi-objective defined by the metrics and then extend it to capture the non stationary dynam- ies of curriculum learning, i.e., the best intrinsic reward may change across learning stages. This assumption is in line with our observation that single intrinsic reward cannot keep improving all metries and URL may stop exploration and ends with sub-optimal skills.
To the best of our knowledge, our work is among few pioneering studies focusing on developing evaluation metrics for URL. While automatic curriculum learning (ACL) has achieved success in deep RL (Portelas et al. 2020), it has not been studied for URL, though adaptively changing intrinsic motivations is a natural human learning strategy in exploring an unknown In experiments, we challenging URL environments. Our method consistently achieves better and more balanced results over multiple evaluation metrics than SOTA URL methods. Moreover, present thorough empirical analyses to demonstrate the advantages brought by the automatic curriculum and the multi-objective for optimizing the curriculum.
evaluate our approach
2 RELATED WORKS
Unsupervised Reinforcement Learning. Intrinsic rewards are used for training URL. For ex-
ploration, intrinsic motivations can be based on curiosity and surprise of environtal dynam- ics (Di Domenico & Ryan| 2017), such as Intrinsic Curiosity Module (ICM) (Pathak et al. (017).
Random Network Distillation (RND) (Burda et al. 2019), and Disagreement (Pathak et al. 2019
Another common way to explore is to maximize the state entropy. State Marginal Matching (SMM) (Lee et al. 2019) approximates the state marginal distribution, and matching it to the uni- form distribution is equivalent to maximizing the state entropy. Other methods approximate state entropy by particle-based method MEPOL (Mutti et al. 2020). APT (Liu & Abbeel 2021a), Pro- toRL YYarats et al. (021). APS (Liu & Abbeel][022 2021b). Mutual information based approaches have been used for self superviid skill discovery, such as VIC (Gregor et al. 2017). DIAYN (Eysenbach) et al. 2019a). VALOR (Achiam et al., 2018). VISR (Hansen et al. 2020) also optimizes the same ojective, but its special approximation brought successor feature (Barreto et al. | 2016) into unsuper- vised skill learning paradigm and enables fast task inference. APS (Liu & Abbeel b02lb) combines the exploration of APT and successor feature of VISR.
Automatic Curriculum Learning. Automatic curriculum learning has been widely studied. It al- lows models to learn in a specific order for learning harder tasks more efficiently (Graves et al. 2017 Bengio et al.] 2209). In RL, lot of work considers scheduling learning tasks (Florensa et al. 2018
2017 Fang et al. 2019 Matiisen et al. 2019 Schmidhuber 2013). In URL, handcrafted currculum
used by EDI. (Campos al. 2020) and IBOL. (Kim et al. 2021 EDL first explores, then assigns
2

---

defined as Under review as a conference paper at ICLR 2023
the discovered states to skills, and finally learns to achieve those skills. IBOL also explores and assigns skill after a specific linearzer learning phase. Both of them are not automatic curriculum, and the number of training steps for each training phase needs to be specified before training. In addition, VALOR (Achiam et al.) 2018) mentioned curriculum learning, but their curriculum is just gradually increasing the numher of skills.
PRELIMINARIES
MDP without external rewards In Markov Decision Process (MDP) M = (S,A, without external rewards, and A respectively denote the state and action spaces, and p(#++ (8t, a+) the transition function where 8t, 8t-1 E S and E A. Given a policy [((e(s)), a trajectory (80, do...., ST) follows the distribution We formulate the problem of unsupervised skill discovery as learning skill-conditioned policy [ay]s:, where
p(T) p(50) II: I ((ae]ss)p(s; (,, at).
represents the latent skill. The latent representations of skills 2 can be either continuous 2 E Fd or discrete E {21, 22, 2N (-) and [(=-) denote entropy and mutual information, respectively.
Intrinsic Rewards for URL Intrinsic Curiosity Module (ICM) (Pathak al. 2017 defines the intrinsic reward as the error between the state prediction of a learned dynamics model and the obser- vation. Not good at handling alcatoric uncertaintics in the dynamics: r(s, a, s') o llg(s' ss a) s/
Disagreement (Pathak al. 2019) is similar to ICM but instead trains ensemble of for- ward models and defines the intrinsic reward as the variance (or disagreement) among the mod- els. Better at handling aleatoric uncertainties since high stochasticity in the environment will sult high prediction error but low variance if it has been thoroughly explored. rss, a)
(8 a.)]; i = N.
State Marginal Matching (SMM) el. 2019) learns parametric density function qo to approx- imate state distribution. It tries match the state distribution to uniform prior p" that maximizes (S):: r(s) = log (s)) (s) log (s)..
Active Pre-training (APT) (Liu & Abbeel 2021a) utilizes a particle-based estimator that uses K nearest-neighbors to estimate state entropy in a non-parametric way. It is defined as r(s) x Σ EjErandom log || syll == -....,,.
DIYAN [Eysenbach et al. 2019b) optimizes the mutual information between states and skills. It learns a parametric disciminator qe. and optimize -H(Z)S) by the intrinsic reward: r(s, 2) oc log qu 2|5) const.
4 METHODOLOGY
4.1 OVERVIEW OF AUTOMATIC CURRICULUM FOR URL
Instead of keep using one intrinsic reward for training an URL agent, we allow the agent choosing one reward among multiple candidates in each learning stage for improving multiple evaluation metrics. This generates an automatic curriculum for URL whose goal is to find sequence of intrinsic rewards that optimizes multi-objectives each corresponding our proposed automatic curriculum method is illustrated in Fig I Our curriculum adds an outer loop outside the conventional URL framework (i.e., the interaction between RL agent and environment). The curriculum has reward selection module that selects an intrinsic reward for each learning stage based on multiple evaluation metrics computed on the replay buffer. By allocating the intrinsic reward that can result in the greatest improvement on multiple metries in each stage, the curriculum aims to find an optimal sesquence rewards to keep improving the URL loop and optimize all the metrics.
metric. The framework of
Given previous work in URL, we still need to address two primary new challenges in building the curriculum: (1) what the evaluation metries? and (2) how to select the intrinsic reward for each learning stage? We propose our solutions to these two problems in Section 4.2 anddn respectively. In Section 4.2 propose multiple metrics to evaluate the capability of URL exploration and
3

---

Under review as a conference paper at ICLR 2023
Intrinsic reward function
RL agent
reward selection module
Environment
Replay bulfer
Evaluation metrics
Figure 1: Block diagram of our proposed method
skill learning. These metries not only include existing ones also cover other preferred properties. trade-off for award selection in URL and
In Section 4.3 we discuss the extend an multi-objective multi multi--ameed bandits algorithm to make non-stationary decisions on the reward used for each learning stage's 's URL.
4.2 MULTIPLE EVALUATION METRICS
In the following of this section, Our goal developing general and consistent metrics to evaluate the process of exploration and skill learning. We start describe our methods with toy examples.
(a)
(c)
Figure 2: Examples of tree maze trajectories
In Fig. 2 the 3 subfigures show the trajectories different agents: (a) an unconverged agent, (b) an APT agent and (c) agent by proposed method. Each agent has 4 different skills. Each skill has trajectories with its own color. Although the trajectories in Fig. 2a seem to have covered more state space than others, the agent's 's skill is not well learned. Trajectories of different skills in Fig. 26 are more separable but they not well balanced because the yellow skill covers more states than others. Fig. 2c seems have explored the state space well and learned balanced skills.
Previous works rely on visualizations of trajetories to compare the performance of exploration and skill learning (Campos et al. (2020) (Kim al.) (021) (Park al.) (2022) (Eysenbach et al.) (0196). By visualization, we might consider the agent in Fig. 2c to be the best overall, but we do not have quantitative reason for it. Next, we will define general metries that quantitatively evaluate ex- ploration and skill learning. And the evaluation results using the proposed metrics for Fig. 2 can be found in Section 5.2
State Coverage (SC) This metric evaluates how much of the state space the agent can cover.
Similar to previous methods that approximate the state entropy using particle-based methods, our metric is also based on particle-based entropy. By (Singh et al., 2003) the particle-based entropy estimation should be sum of the log of the distance between each particle and its k-th nearest neighbor, defined as
Hpu (S) log |ss
8 (k))
(1)
4

---

Under review as a conference paper at ICLR 2023
For robust and stable implementation, we use the modified version from APT (Liu & Abbeel 202 see Appendix A We find that with large numbers of n and k, this estimation can reflect what good state coverage in visualization, we apply it for large number of recent states in the buffer to evaluate the learning progress of state coverage.
Remark. Our purpose is to have an automatic curriculum that trying to optimize evaluation metrics by selecting intrinsic reward. For this state coverage metric, one question could be why not use itself as the intrinsic reward and directly optimize it with RL algorithms APT did. The main reason is that APT agent tend to expand its trajectory locally instead of reaching to new areas. Combining with other intrinsic rewards help to escape this local optimum. In fact, the agent in Fig, 2b is an APT agent and the one in Fig. 2c] learns by a curriculum. What's s more, we find it expensive to estimate with large n and is for every RL update, so the entropy estimation in APT could be less accurate.
Particle-based Mutual Information (PMI) Mutual information between state S and latent skill Z is an essential objective for URL to learn a skill conditioned agent. Intuitively, [Eysenbach hEysebbbhyyEyysebacchh et al. (2021) showed that, under some assumption, maximizing the objective T(S; Z) initialize the agent to be optimal for certain downstream tasks. Mutual information between state and skill can be expanded as
((S = -S) H(
(2)
It is one entropy subtracting another, so we propose to implement particle-based entropy to approx- imate them and obtain the mutual information.
I(S; Z) x HPp ())
The approximation bias of particle-based entropy in Eq. [ depends on h and so with the large sample numbers of and is for both Hpp (S) and Hp (S|Z). from this substraction.
get accurate approximation
Mutual information for individual skill When optimizing the mutual information objective, it is possible to learn some skills that are less informative than others. So it is necessary to evaluate the informativeness of an individual skill. There have not been any of this kind of metrics in URL yet. To fill this gap, we define novel metric as T(S; 1z), where 1 is the indicator of Z = and is random variable. The estimation of this metric is shown Appendix B
We find that I(S; 1.) measures not only the informativeness of this skill, but also the separability. In the URL skill learning setting, separability between one skill and others means that the states inferred by this skill should share little overlaps with the states inferred by others. Therefore, it means that the states should be certain to be inferred by one single skill, thus meaning low H11,
Because 1.) H(S) H (SS 1.), for situations with the same state and skill distributions, the one with larger (S; 1.) would lower in H (S]1.). ). since H (1,]SS) = ((S) + HH)), lower H()) means also low in (1_ thus better separability.
Disentanglement IBOL (Kim et al. 2021) first introduced URL with the concept of disentangle- ment, motivation for this metric from a theoretical perspective is in Appendix. C This concept includes two aspects: Informativeness and separability. Informativeness here means information shared between a skill and its inferred states. Separability in representation means that there should be no information shared between two latent dimensions, while here it means the trajectories inferred by two different skills should be separated from each other.
They evaluate disentanglement with metrics such SEPIN@K and WSEPIN (Do & Tran 2019) to measure disentanglement. SEPIN @k is the top k average of I(S, Zz44) and WSEPIN is the average of I(S, Z, Z++) weighed on I Z.).
However, because SEPIN @k and WSEPIN were originally used for representation learning, 2, was meant refer to independent representations. For skill-conditioned RL, skill Z is single random variable, so for URL setting, it is not suitable to directly use I(S, Z Z44), which requires Z, and Zyz to be two individual random variables.
With I(S 12) to measure the informativeness and separability of skill 2, we can define metrics for the disentanglement of learned skills. Again, because SEPIN@k and WSEPIN are defined for
5

---

Under review as a conference paper at ICLR 2023
representation learning. where the more informative representations valued, while the lesser ones are ignored. how SEPIN @k and WSEPIN average over the representations is not suitable for the skills in the URL setting. Instead of ignoring the ones with lower informativeness, all learned skills of an URL agent have an impact on the agent's behavior, so we care about the median and minimum of the informativeness of the skills.
We defined Median SEParability and INformativeness (MSEPIN) as
MSEPIN med [(5 12),
(4)
where med the median over skills 2, and Least SEParability and INformativeness (LSEPIN) as LSEPIN = min (,)
(5)
4.3 AUTOMATIC CURRICULUM FOR URL
Existing URL methods only use single intrinsic reward to train their policies. Since the accuracy of the reward depends on the agent's modeling of the environment dynamics, whose quality heavily relies on the data collected through the agent's ' exploration, URL keeping using the same reward may stop exploration earlier with a sub-optimal policy. In this paper, we instead use multiple intrinsic rewards because different rewards can benefit the learning process from different perspectives. This shown in Section [.3] where curriculum combining two intrinsic rewards made them comple- mentary to each other and thus performs better than single-reward URL. Moreover, it also showed that a random sequence of intrinsic rewards could not be better than a single one, so the order of intrinsic rewards matters in URL. Therefore, our goal is to find curriculum, i.c. sequence of intrinsic rewards selected from a candidate set.
The curriculum should also be unsupervised, so no prior knowledge extrinsic metric is allowed. the mechanism of choosing the next intrinsic reward for the agent should only be based on the historical information collected from environmental interactions. In order make better choices, it also needs to try different intrinsic rewards and evaluate the improvement they bring to URL. trade-off process, e.g., multi-armed bandit algorithm, is critical to the curriculum development.
Therefore, an exploration-exploiiae
Since our goal is an agent excelling on multiple evaluation metries, the curriculum should take all these multiple objectives into account when selecting a reward for the next stage training. Hence, we formulate as culti-objeectiee multi-armed bandit problem and adopt empirical Pareto UCB (Dru- gan & Nowe 2013) algorithm because of its easier implementation and tighter regret bound. In addition, due the non-stationary dynamics curriculum, the best intrinsie reward may change across learning stages and we need non-stationary extension of Pareto UCB capture the changes.
We consider automatic curriculum learning based on Empirical Pareto UCB to learn task selection function D H. U where H can contain any information about previous interactions and 2/ is the finite candidate set intrinsic rewards as tasks. The goal of D is to minimize the regret with respect to the Pareto optimal for these objectives:
max ED
P!J max E.
PP..
P? L=
ED
PM
where 7' is the total number of selections in the curriculum and P is vector with each entry corre- sponding to each evaluation metric. p is the dimension of P and is the number of metrics, p the äth entry of P. H, E H includes the (P,, D(H(-1)) tuples for all í < t. The metric Pt quantifies the agent's behavior on task D(H,) after a certain number of training steps.
For Empirical Pareto UCB, task uniform randomly chosen from the Pareto action set
(n(t Ni(o)
In(t (n(t Ne(u)
a Vu EU U, (1(v) +0
+ pr(u) +
where u, E 24 is the intrinsic reward selected by UCB, and p(u) is the weighted sum of the past P by training agent in the intrinsic rewards of the task N (()) is the number of times u has been chosen.

---

Under review as a conference paper at ICLR 2023
K is empirical number that upper bounds the Pareto optimal set of arms. means non-dominant. We say that æ is non-dominated by y/s y yf I; if and only if there exists at least one dimension j for which y < xd [Zitzler et al. 2003).
This nonstationary multi-arm bandit, because the intrinsic reward resulting in best learning progress might change along with the agent's learning process. There adapt the UCB algorithm for nonstationary situations. One way to do this is with discounting and another way use a sliding window (Lattimore Szepesvári, 2020). We find that discounting has better performance in experiments. Let E (0, 1) be the discount factor, and define
common ways to
(u)
Σ 7 P,T(Us =u),
(8)
8=0
and
(1) = Σ
I(u u
(9)
==0
When using discounting for nonstationary Empirical Pareto UCB, the Pareto action set becomes
In(E wey (w) VpK)
y (7) t In EweE N7 (w) vpk)
Vu U. (0) t
(7)
N7
(10)
5 EXPERIMENTS
In this section, we conduct experiments to evaluate our method. We first analyze how the curricu- lum combines the advantages of candidate intrinsic rewards, providing intuition for benefit in exploration and skill learning. Then, we compare the proposed method to baselines such as single intrinsic reward and random curriculum.
5.1 SETUP
The environments consider are the 2D maze environments of (Campos al. 2020), which modified from (Trott et al., 2019). These environments are meant to provide intuitive comparison between visualizations and our proposed metrics. More experimental details and more results in high-dimensional Mujoco environments are included in Appendix. D and Appendix. F
Baselines are two categories:
Conventional URL with single intrinsic reward. It is also the minimal implementation of previous URL approaches.
Random curriculum. We consider curriculums with randomly selected intrinsic rewards. By comparing our proposed method to this random curriculum, could validate whether the bandit algorithm for selection is necessary.
5.2 ANALYSIS OF EVALUATION METRICS
Recall that in Fig. 2 there are traditional visualization results for three agents. Correspond- ingly, we provide evaluation results with proposed metrics in Table The particle-based met-
Table 1: State Coverage in Fig.
SC
PMI
LSEPIN
MSEPIN
(a)
1656
44
0
(b)
1386
881
145
275
(c)
1471
966
271
308

---

Under review as a conference paper at ICLR 2023
rics estimated with 4000 and k 2000. The state coverage of (a). (b) and (c) is [1656, 1386, 1471]. It is obvious that Fig. 2a has best state coverage, so this metric is consistent with the visualizations. The PMI of Fig. is [4, 881, 966). It is also consistent with the visualiza- tions in which both Fig. and Fig. 2c show overall informative skills. The T(S: 1:) for Fig[b/j is (blue: 145, yellow: 509, green: 324, red: 225). is in accordance with the visualizations, in which yellow is the most informative skill. For the blue skill, its ;(S] is high because it has a high H (s|Z A 2) close to H (s). This is shown in visualization in which the state coverage without blue is close to the state coverage with blue. Intuitively, the blue skill is not as informative as others because does not contribute to the total state coverage as much as other skills, so blue has low [S; 1;). The [(S; 12) for Fig 2c is (blue: 345, yellow: 271, green: 258, red: 369]. The skills learned are better balanced, with red being the most informative and green slightly less informative than others. T(S; 1 for every skill in Fig 2alis close to 0, because the skills are inseparable and uninformative. For the skills in Fig. 2b the disentanglement metrics are |MSEPIN: 275, LSEPIN: 145). and they are (MSEPIN: 308, LSEPIN: ] for the skills in Fig. 2c As expected from the visualizations, the skills of the agent in Fig. 2c are more disentangled than in Fig. 2b
5.3 ADVANTAGES OF CURRICULUM URL
We use the two intrinsic rewards, including APT and ICM, and evaluate our curriculum with them on the tree maze environment to show how good combination compliments each other's disad- vantages. In our method, we use APT and ICM as two arms. For the tree maze environment, we consider a single objective of state coverage as defined in Section 4.2 We run APT, ICM, Random curriculum and method with the same series of 5 random seeds. The total number of reward selections are 100.
Table Two Arm State Coverage in Tree Maze. (SO indicates single objective.)
State coverage
APT
ICM
Random
Ours (SO)
mean std
1397.32
1215.26
1391.09 114.82
1474.78
54.12
304.28
48.99
[shle the means and standard deviations of the state coverage. It's clear that Ours(SO) dom- inates both in mean performance and performance variance. The baseline of the random curriculum could not be better than the single APT, so the order of intrinsic rewards in the curriculum matters and Ours(SO) is capable of finding a good curriculum.
(a)
(b)
(c)
(d)
Figure 3: Tree maze trajectories of (a) APT, (b) ICM, (c) Ours (SO), and (d) Ours (MO).
By first looking at the trajectories learned by APT and ICM shown in Fig. 3a and 3b the results show that APT prefers to make the agent expand its state coverage locally. The trajectories are getting thicker but stay near to the starting state. ICM reaches to further areas, but its trajectories are thin, possibly because its reward rely on a prediction model of environmental dynamics. When this model is accurate in large part of the state space, the intrinsic reward might lead the agent to go only along where the model is not as accurately approximated. We found that as learning progresses, Ours (SO) prefers to choose APT more. The number of times APT is chosen in the later half of training is on average 32.6% higher than the first half. This is in agreement with an intuition that for better state coverage, the agent should first reach further and then expand its trajectories.
8

---

Under review as a conference paper at ICLR 2023
5.4 ADVANTAGES OF MULTI-OBJECTIVE FOR CURRICULUM
For the same tree maze, we consider multiple objectives: SC, PMI and LSEPIN. We use three intrinsic rewards, including SMM, ICM and APT. Table shows the results that we compare the multi-objective Pareto UCB to UCB with only SC objective. Overall, MO works better than SO. means that multiple objec- tives improve the agent's learning. Fig. Bl (c) and (d) show two agents learned by Ours (SO) with single objective of SC and Ours (MO) learned by multi-objective of {SC. PMI, LSEPIN) respec- tively. The SO agent has skill on the left side of the tree, while the MO agent has trajectories of all skills separated. This an example of why Ours (MO) has better disentanglement on average.
1995
SPM
Figure Multi-metrie
Table 3: Comparison between Ours (MO) and Ours (SO). (MO indicates multiple objectives.)
SC
PMI
LSEPIN
MSEPIN
Ours (MO)
1393.59 1460.23
921.34
198.61 179.08
274.66
Ours (SO)
887.01
232.91
We also evaluate our method on larger crazy maze environment. This is an environment much more difficult to explore than the simple tree maze.
(a) APT
(b) ICM
(c) SMM
(d) Ours (MO)
Figure Trajectories samples the crazy maze
In Fig. we compare our method with individual intrinsic rewards and the random curriculum base- line. As we can see from Fig. 4 our proposed method achieves well-rounded result as expected.
Also, the quantitative result inturtively accords with visualizations in Fig. For example by Fig. SMM has better SC metric but less disentanglement (MSEPIN and LSEPIN than APT. SMM covered more to the upper right of the maze, but its blue and purple skills seem to be entangled with each other.
CONCLUSION
We proposed quantifiable and general evaluation metrics for URL. Our proposed metrics can stably measure the state coverage for exploration, as well as mutual information and disentanglement for skill learning. This helps to enable evaluation of URL without specific downstream tasks. Further- more, proposed an automatic curriculum to select intrinsic rewards based on the agent's learning progress. This automatic curriculum does not require prior knowledge of the environment or its intrinsic reward candidates. It is nonstationary Pareto UCB that utilises historical evaluations for decision making, and tries to train the agent to be well-rounded at all aspects of the considered metrics. Our experimental results have demonstrated the effectiveness of our method.

---

Under review as a conference paper at ICLR 2023
ETHICAL STATEMENT
Our work does not suffer from discriminatioobbiiafaainess concerns.
REPRODUCIBILITY STATEMENT
The implementation code will public after our paper is accepted.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 1807 2018.
André Barreto, Rémi Munos, Tom Schaul, and David Silver. Successor features for transfer in rein- forcement learning. CoRR, 0hbs/1606.0533 2016. URL BRKLLttt.:/aaawv..or///////////
05312
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp- 41-48, 2009.
George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):
374, 1951.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nicto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, 2020, URL
https://arxet org/ abs 2002. 03647
Stefano I. Domenico and Richard M. Ryan. The emerging neuroscience of intrinsic motivation: A new frontier in self-determination research. Frontiers in Human Newroscience, 11, 2017. ISSN
1662-5161. doi: 10.3389/mhhmm22000 2017.00000211000003333 URL
articles/I0 10. 3389/388 fwwwmmn.mmmmmmmm 2017 7 00145
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations, 2019.
Madalina M Drugan and Ann Nowe. Designing multi-objective multi-armed bandits algorithms: A study. In The 2013 International Joint Conference on Neural Networks (IJCNN, pp. 1-8. IEEE, 2013.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- fions, 2019a
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In JCLR, 2019b.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of un- supervised reinforcement learning, 2021. URL httpp/////xxxvoorr/aaa//.
Meng Fang, Tianyi Zhou, Yali Du Lei Han, and Zhengyou Zhang. Curricllu-guiieed hindsight experience replay. Advances in neural information processing systems, 32, 2019.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR. 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In Inter- national Conference on Learning Representations, 2017.
Steven Hansen, Will Dabney, André Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- lional Conference on Learning Representations, 2020.
Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning, 2021.
http:///aaax....o////////////// 14305
Bahare Kiumarsi, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L Lewis. Optimal and autonomous control using reinforcement learning: survey. IEEE transactions on neural networks and learning systems, 29(6):2042-2062 2017.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark, 2021a. URL
aittps://arxte org/ abs attg///b///1.1111/33
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 2021b.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhut-
dinov. Efficient exploration via state marginal matching. CoRR, abs/1906.00277 2019.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551 2021a.
Hao Liu and Picter Abbeel. APS: active pretraining with successor features. In International Con- ference on Machine Learning, 2021b.
Tambet Matissn, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE transactions on neural networks and learning systems, 31(9):3732-3740 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task- agnostic exploration. CoRR, hns/2007.04644) 2020. URL htttps://arxiv.org
2007.00 04640
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task-agnostic exploration. In Conference on Artificial Intelligence, 2021.
Pierre-Y Yvers Oudeyer and Frederic Kaplan, What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Scohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery, 2022. URL.
00914
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, 2019.

---

Under review as a conference paper at ICLR 2023
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey, 2020.
Julia Robinson. An iterative method of solving a game. Annals of mathematics, pp. 296-301, 1951.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers psychology. 4:313, 2013.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484-445 2016.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 23(3-4):301-321, 2003.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances Neural Information Processing Systems, 32, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamnekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, 2021.
Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M Fonseca, and Viviane Grunert Da Fon- seca. Performance assessment multiobjective optimizers: An analysis and review. IEEE Trans- actions evolutionary computation, (22::117-132,200 2003.
12

---

Under review as a conference paper at ICLR 2023
A MODIFIED PARTICLE-BASED ENTROPY
Liu & Abbeel 2021a) find averaging the distance over all nearest neighbors leads to a more robust and stable result.
(sB(s)
Σ
n
log
c+
1 k
Σ
1s
(3)
(11)
(N
where (+(O) denotes the k nearest neighbors around particle, e is a constant for numerical stability (commonly fixed to 1).
B APPROXIMATION OF [(S; 1.)
It can be approximated by
I(S:1.) x HS) (S
(12)
The second term of Equation 12 is the state entropy conditioned on knowing whether skill equals 2, and it defined as
H(S|1.) = 2) H
=
(13)
The first term of Equation 13 can be estimated by sampling from the states generated with skill 2. The second term little more tricky for parametric methods that try to approximate log p(s z). but for particle-based entropy it can still be conveniently estimated by sampling states from all skills not equal to z.
C MOTIVATION OF DISENTANGLEMENT FROM A THEORETICAL PERSPECTIVE
Eysenbach al. 2021 analysed that maximizing the mutual information objective T(S, Z) learns skills are all optimal for some downstream state-dependent reward functions (a common family of downstream tasks). However, in practice, (S, 2) cares about only the expected informativeness of skills, only optimizing (S, 2) could result in skills that are far less than the average. By adding dis- entanglement metrics MESPIN and LSEPIN, we can evaluate how well individual skills are learned, in order ensure that all skills are potentially the best skill for certain downstream tasks.
From a information geometry perspective, this paper claims that the learned skills must lie at ver- tices of the state marginal polytope, and for every state-dependent reward function (corresponding downstream task), among the set of policies that maximize that reward function is one that lies at a vertex of the state marginal polytope. But the learned skills can cover all vertices. Therefore, by adding disentanglement metrics, we care about the separatibility of skills. We want the skills to be further from each other and not to cluster only in a small region of vertices.
D EXPERIMENTAL DETAILS OF 2D MAZE
The experimental settings follow mainly Campos al. 2020 g and the 2D maze environments are modified from Trott et al. 2211). It is a continuous 2D environment, so the implemented RL algorithm is DDPG. The reward selection module selects new intrinsic reward function every 500 episodes. The buffer only stores state action transitions. When DDPG update, the intrinsic rewards are calculated by the sampled states and current intrinsic reward function.
For baselines, we mentioned that we use the minimal implementation of existing URL algorithms. For example, ICM and APT can operate the space of latent representations with additional mod- ifications, while we implement them directly in the state space. Besides, SMM mentioned that exploration with SMM can be considered as a game between the density model of the state distri- bution and the agent, fictitious play Brown equilibrium in finite time (Robinson) 195
can be implemented to help find the Nash To capture this advantage of SMM, we implemented
1951p p
SMM with 4 historical density models.
13

---

Under review as a conference paper at ICLR 2023
E
MORE RESULTS FOR 2D CRAZY MAZE
Fig.
6 show sample trajectories on the the crazy maze environment, which is a more complicated 2D maze environment than the maze. Our method works better than others. Table shows that the well-rounded performance of our proposed method is consistent.
(a)
(b)
(c)
(d)
Figure 6: Trajectories samples of the crazy maze: (a) ICM, (b) SMM, (c) APT, and (d) Ours(MO).
Table 4: Comparison between Ours (MO) and baselines.
Method
SC
PMI
LSEPIN
MSEPIN
Ours (MO)
1552.81
1076.98
251.59
345.36
Random
1338.36
883.22
148.64
284.97
APT
1457.11
848.73
188.74
266.67
SMM
1279.97
764.52
137.35
229.73
ICM
1375.10
647.51
83.27
228.65
F. EXPERIMENT ON MUJOCO BENCHMARK
In the setting of URLB Laskin al. 2021a la
the normalized performance of 4 downstream tasks in
Walker domain for existing methods is shown the radar plot Fig. 7a We evaluated their pretrained model by proposed metrics are shown in Table
XT
47
9M
0/4
301
M0W
(a) Walker
(b) Quadruped
(c) Jaco
Figure 7: Radar plots for benchmark downstream performance. Each plot shows the normalized performance in each domain of 4 downstream tasks.
In URLB setting, ICM and APT only explore and do not learn skills, so they do not optimize the mutual information skill learning objective and the skills metrics not applicable to them. The performance shown in Fig. is from pretraining the agents with corresponding URL algorithm in
corresponding domain for le5 steps.
As we can see by comparing Fig. 7 and Table 5 there is a strong correlation between overall down- stream performance and the proposed state coverage metric. This shows that in more complicated
14

---

Under review as a conference paper at ICLR 2023
Table 5: Metrics for Benchmark domains
Domain Walker
Method
SC
PMI
LSEPIN
MSEPIN
ICM
6328.43
APT
6521.84
SMM(W MI)
5861.81
179.66
1.24
60.69
DIAYN(W MI)
4367.92
245.43
98.99
119.38
quadruped
ICM
6365.39
APT
6357.98
SMM(W MI)
6509.69
38.23
26.84
38.82
DIAYN(W MI)
6135.50
8.55
7.43
8.66
Jaco
ICM
3202.43
APT
3155.22
SMM(W MI)
3663.76
41.16
13.28
20.59
DIAYN(W MI)
3434.12
28.99
2.28
11.33
environments, our proposed metrics can also evaluate the properties of the pretrained agents, and these properties are correlated with downstream task performance.
A high mutual information value means that the state space is better partitioned by the skills, so the 3kill-conditoone policy of DIAYN with high PMI metric covers specific region of the state space. However, these downstream tasks are not just simple partitions of the state space. Tasks such as Walk, Jump, and Run are trajectories with specific state and action sequences, so reasonable that the simple mutual information objective T(S, Z) is not capable of assigning task-specific trajec- tory sequences with skills, while the high PMI metric means that the explored state space is better partitioned by skills, meaning that individual skill-conditioned policy actually covers less state space. This explains the reason why DIAYN have a bad performance for downstream tasks, learned skills are not directly applicable downstream tasks but they have less state coverage.
This is problem with the objetive of DIAYN itself, and more recent URL algorithms are trying to deal with the problem how to learn skills that are not just simple partitions the state space, such as VISR (Hansen et al. 2020
APS Liu & Abbeel] 2021b LSD
Park et al. 2022 etc.
The effect mentioned above is more obvious in the domain of Jaco, where the tasks of controlling robot arm to reach BL (bottom left), BR (bottom right), TL (top left), TR (top right) have very different rewarding states, so individual skill-conditioned policy with less state coverage could have more sparse rewards at the beginning of fintuning.
In summary, our proposed metrics are capable of evaluating the properties of unsupervised RL in high-dimensional environments where visualizations are not applicable. There is correlation be- tween the performance of the downstream tasks and our proposed metrics, our proposed metrics are more general and not restricted to specific tasks.
15


Reviews:
[{"review_id": "LnFyOg_PjX", "reviewer": "Reviewer_sbNE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at the problem of unsupervised RL where skills must be learned for exploration in an environment without access to rewards. The authors take the approach of considering multiple metrics for guiding this unsupervised learning that have been proposed in the literature while choosing among each of them in an automatic curriculum learning fashion. To guide the curriculum learning process the authors use a discounted version of Pareto UCB in order to handle the non-stationary and multi-objective components of the problem. The authors compare their approach to previous baselines showcasing visualizations and metrics that may indicate their approach has intuitively learned something better or more useful. ", "summary_of_the_review": "I am a bit on the fence about this paper as I think the topic is important, the components of the proposed approach make sense individually, and the results seem pretty good. However, I feel like the novelty is quite low and the approach seems a bit over-engineered given the nature of the domains considered. As a result, I lean towards rejection at the moment, and feel that the paper could be a lot better if it engaged with more of a unified theory and discussed/explored the connection with preparation for downstream tasks more concretely.\n\nUpdate After Author Feedback: \n\nWhile I really appreciate the detailed response from the authors, I honestly did not find it very convincing in addressing my concerns and thus lean towards my original assessment. I will just list some of my feedback on each point listed by the authors in order to be constructive for the authors in forming subsequent revisions:\n\n1. First of all, proving that mutual information is not an optimal single metric is very different from establishing that any such single metric does not exist. Also, the intuition that we would like each skill to be balanced in how informative it is seems purely intuitive to me. \n2. This is just a restatement of the argument from the paper. My personal feeling was that the visualization was what I got the most out of. \n3. While covering all possible downstream tasks may be computationally prohibitive, I don't see why it is not possible to perform a statistical analysis of performance on an unbiased sample. I appreciate the results provided for Mujoco and would suggest this kind of analysis should be further highlighted within the main text. \n4. I disagree that the first point has been established. While past curriculum learning strategies may or may not be applied to unsupervised RL thus far in the literature, they are potentially applicable to unsupervised RL. Additionally, the latter points are as I stated really resulting from a combination of smaller contributions. \n5. I disagree that points 1, 2, and 5 have really been established beyond reasonable doubt in this work. If these things could be formally proven, then I can see the authors perspective. However, I am currently doubtful that it is possible to show this based on the current discourse. \n6. This is not what I was looking for when I asked for a theoretical justification. More concretely, I would suggest formally establishing theories related to points 1, 2, and 5 in response to the question about over-engineering that can be stated as theorems or propositions. Point 3 can be a corollary to these established theoretical statements. ", "strength_and_weaknesses": "Strengths: \n- Based on my subjective view, I do believe the proposed approach has arrived at better skills in some sense. \n- Each component of the system seems logical to me in the context of the goal of the paper.\n\nWeaknesses: \n- While in some sense the solution makes a lot of sense, no individual component of the system is particularly novel in light of the prior literature. Additionally, the motivation to combine all of these objectives in this way seems a bit like an over-engineered solution for a topic that is largely blue sky as currently presented and far from concrete applications. \n- The proposed metrics for evaluating unsupervised RL do seem a bit arbitrary/hacky in the grand scheme of things and I feel like I get the most out of the visualizations, which is counter to the goal of this paper. Why not evaluate the skills with respect to downstream tasks? While I see where the authors are coming from arguing that the skills learned are better than the baselines visualized, it all feels quite subjective to me and really depends on what we would like to use these skills for later. \n- The paper is largely based on a series of intuitions and I didn't really find a clear overarching theory that justifies the contribution. \n- I worry that the proposed solution has a number of moving parts and may be sensitive to hyperparameters and particular implementation details. \n- I wonder if the authors can explain more about the issue with individual objectives for unsupervised RL. It does seem counterintuitive to me that no single metric would be able to result in good skills. I just wonder if the community has not found it yet or if it is obviously something more like a meta-transfer objective. I would be much more excited about this paper if it had more to say from a theoretical perspective about this topic. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0eF6u0PtJn", "reviewer": "Reviewer_kRzv", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a multi-arm bandit method to choose between to intrinsic rewards for exploration in a synthetic maze, in an Unsupervised Reinforcement Learning (URL) setting.\n\nIt also defines some metrics to track the diversity of the policies as well as their coverage of the maze, in order to be able to compare them.\n\nSince there are several methods of interest, the proposed method used a multi-objective bandit method.", "summary_of_the_review": "Overall, the paper falls short of demonstrating the usefulness of the method, both in terms of downstream evaluation, as well as in non-toy settings. On top of that, the experiments lacks too many details to be replicated. I therefore recommend the rejection of this paper in its current shape.", "strength_and_weaknesses": "The paper doesn't provide downstream evaluation of fine-tuned policies on actual task (such as, in their setting, the completion of the maze), as is commonly done in the URL literature. As such, some key underlying assumption of the paper as not empirically verified. For example, the paper proposes to maximize the combination of the proposed metrics (in a pareto optimal way). However, there is no evidence that doing so actually results in better downstream performance on tasks of interest.\n\nOne other major weakness of the work is to limit the experiments to a very toy setting, without trying to apply the findings to more challenging environments (such as Habitat, Minecraft3D, Doom, Atari) as is customary in the URL literature.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0DeeAPImBE", "reviewer": "Reviewer_eCNr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For unsupervised reinforcement learning, this paper proposes (1) a set of metrics to evaluate the exploration and skill learning of the agent without specific downstream tasks; (2) an automatic curriculum to train the agent with different intrinsic rewards in different stages, formulated by multi-armed bandit.", "summary_of_the_review": "The paper is well-written and the proposed approach is novel, but the experiments are only based on a single 2D environment makes the results not fully convincing. Besides, as an unsupervised approach, it's not tested in any downstream applications.", "strength_and_weaknesses": "Strength:\n* The paper is well-written and easy to understand.\n* The approach is novel and well motivated.\n* The experimental results are clearly showcased.\n\nWeaknesses:\n* The experiments are only based on a single 2D maze environment, which makes the results not convincing enough. There are other more realistic environments to further test on, e.g., https://arxiv.org/pdf/2110.15191.pdf.\n* There is no results to show if the proposed unsupervised RL method can really help in downstream applications, although the proposed approach doesn't require assumptions on specific downstream task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject