CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery
Authors: Michael Laskin
Hao Liu
Xue Bin Peng
Denis Yarats
Aravind Rajeswaran
Pieter Abbeel
Venue: ICLR 2022 Submitted (2022)
Keywords: unsupervised learning
reinforcement learning
exploration

Abstract: We introduce Contrastive Intrinsic Control (CIC) - an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions. In contrast to most prior approaches, CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by maximizing state entropy. We derive a novel lower bound estimate for the mutual information which combines a particle estimator for state entropy to generate diverse behaviors and contrastive learning to distill these behaviors into distinct skills.  We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. We find that CIC improves on prior unsupervised skill discovery methods by $91\%$ and the next-leading overall exploration algorithm by $26\%$ in terms of downstream task performance.


Under review as a conference paper at ICLR 2022
CIC: CONTRASTIVE INTRINSIC CONTROL FOR UNSUPERVISED SKILL DISCOVERY
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce Contrastive Intrinsic Control (CIC) an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions. In contrast to most prior approaches. CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by maximizing state entropy. We derive novel lower bound estimate for the mutual information which combines particle estimator for state entropy to generate diverse behaviors and contrastive learning to distill these behaviors into distinct skills. We evaluate our algorithm the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-fres pre-training phase followed by a short adaptation phase downstream tasks with extrinsic rewards. We find that CIC improves prior unsupervised skill discovery methods by 91% and the next-leading overall exploration algorithm by 26% in terms of downstream task performance.
INTRODUCTION
Deep Reinforcement Learning (RL) is a powerful ap- proach toward solving complex control tasks in the pres- ence of extrinsic rewards. Successful applications include playing video games from pixels (Mnih et al., 2015), mastering the game of Go (Silver et al., 2017; 2018), robotic locomotion (Schulman et al., 2016; 2017; Peng et al., 2018) and dexterous manipulation (Rajeswaran al., 2018; OpenAI, 2018: 2019) policies. While ef- fective, the above advances produced agents that are un- able to generalize to new downstream tasks beyond the one they were trained to solve. Humans and animals on the other hand are able to acquire skills without supervi- sion and apply them efficiently to a variety of downstream tasks. In this work, we seek to train agents that acquire skills without supervision with generalization capabilities by efficiently adapting these skills to downstream tasks.
[(T;2) = H(() - H((T(2)
Z3
22
This work deals with unsuper- vised skill discovery through mutual infor- mation maximization. We introduce Con- trastive Intrinsic Control (CIC) a new un- supervised RL. algorithm that explores and adapts more efficiently than prior methods.
Figure 1:
Over the last few years, unsupervised RL has emerged a promising framework for developing RL agents that can generalize to new tasks. In the unsupervised RL setting, agents are first pre-trained with self-supervised intrinsic rewards and then finetuned downstream tasks with extrinsic rewards. Unsupervised RL algorithms broadly fall into three categories - knowledge-basee Knowledge-basee methods maximize the error or uncertainty of a predictive model (Pathak et al., 2017; 2019; Burda et al., 2019b). Data-bar based methods maximize the entropy of the agent's 's vis- itation (Liu & Abbeel, 2021a; Yarats al., 2021b). Competence- based methods learn skills that generate diverse behaviors (Eysenbach et al., 2019; Gregor et al., 2017). This work falls into the latter category of competence-based methods.
data-based, and competence-basee m
'These categories for exploration algorithms were introduced by Srinivas & Abbeel (2021) and inspired by Oudeyer al. (2007).

---

Under review as a conference paper at ICLR 2022
Leep forward
Flip upright
Nudgo brick
Y
M
Walk left
Figure 2: Qualitative visualizations of unsupervised skills discovered in Walker, Quadruped, and Jaco arm environments. The Walker learns balance and move, the Quadruped learns flip upright and walk, and the 6 DOF robotic arm learns how to move without locking. Unlike prior competence- based methods for continuous control which evaluate on OpenAl Gym (e.g. Eysenbach et al. (2019)). which reset the environment when the agent loses balance, CIC is able to learn skills in fixed episode length environments which are much harder to explore (see Appendix K).
Unlike knowledge-base and data-based algorithms, competence-based algorithms simultaneously address both the exploration challenge well as distilling the generated experience in the form of reusable skills. This makes them particularly appealing, since the resulting skill-based policies (or skills themselves) can be finetuned to efficiently solve downstream tasks. While there are many self-supervised objectives that can be utilized, our work falls into a family of methods that learns skills by maximizing the mutual information between visited states and latent skill vectors. Many earlier works have investigated optimizing such objectives (Eysenbach et al., 2019; Gregor et al., 2017; Kwon, 2021; Sharma al., 2020). However, competence-basee methods have been empiri- cally challenging to train and have under-performed when compared to knowledge and data-based methods (Laskin et al., 2021).
In this work, we take closer look at the challenges of pre-truce pre-training agents with competence-basee algorithms. We introduce Contrastive Intrinsic Control (CIC) an exploration algorithm that uses a new estimator for the mutual information objective. CIC combines particle estimation for state entropy (Singh et al., 2003; Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann & Hyvärinen, 2010) for the conditional entropy which enables it to both generate diverse behaviors (exploration) and discriminate high-dimensional continuous skills (exploitation) To the best of our knowledge, CIC is the first exploration algorithm to utilize noise contrastive estimation discrim- inate between latent skill vectors. Empirically, we show that CIC adapts to downstream tasks more efficiently than prior exploration approaches the Unsupervised Reinforcement Learning Bench- mark (URL.).. CIC achieves 91 % higher returns on downstream tasks than prior competence-basee algorithms and 26% higher returns than the next-best exploration algorithm overall.
BACKGROUND AND NOTATION
Markov Decision Process: We operate under the assumption that our system is described by a Markov Decision Process (MDP) (Sutton & Barto, 2018). An MDP consiss of the tuple (S, A. 7) which has states E S, actions a E A, transition dynamics p(s'|s, reward function r, and a discount factor In an MDP, at each timestep t, an agent observes the current state s, selects an action from a policy a ~ x--[s), and then observes the reward and next state once it acts in the environment: s' ~ env.step(a). Note that usually r refers to an extrinsic reward. However, in this work we will first be pre-training an agent with intrinsic rewards اit and finetuning on extrinsic rewards oM
P.
For convenience we also introduce the variable (s)) which refers to any function of the states 8. For instance м can be single state, pair of states, sequence depending the algorithm. Our method uses 7 = (s, s') to encourage diverse state transitions while other methods have different specifications for Importantly, does not denote state-action trajectory, but is rather shorthand for any function of the states encountered by the agent. In addition to the standard MDP notation, will also be learning skills Z and our policy will be skill-conditioned a (-)ss,)).
Unsupervise vised Skill Discovery through Mutual Information Maximization: Most competence- based approaches exploration maximize the mutual information between states and skills. Our
2

---

Under review as a conference paper at ICLR 2022
work and a large body of prior research (Eysenbach al., 2019; Sharma al., 2020; Gregor al., 2017: Achiam et al., 2018; Lee et al., 2019; Liu & Abbeel, 2021b) aims to maximize mutual information objective with the following general form:
T(T;2) H(=) H(=+>) = H(T)) H(T|2)
Competence-basee algorithms use different choices for and can condition on additional informa- tion such as actions or starting states. For a full summary of competence- algorithms and their objectives see Table 1 in Appendix D.
Lower Bound Estimates of Mutual Information: The mutual information [(s; 2) is intractable to compute directly. Since we wish maximize [(5; z). can approximate this objective by instead maximizing a lower bound estimate. Most known mutual information maximization algorithms use the variational lower bound introduced in Barber & Agakov (2003):
(T::2) H()) H(( 2 H222 Ellog [<2/r)]
Note that the variational lower bound can be applied to both decompositions of the mutual infor- mation. The design decisions of competence-based algorithm therefore come down to (i) which decomposition of [(T; 2) to use, (ii) whether to use discrete or continuous skills, (iii) how to estimate (z) or HT)), and finally (iv) how to estimate (=lr) or H()
3 MOTIVATION
The results from the recent Unsupervised Reinforcement Learning Benchmark (URLB) introduced in Laskin al. (2021), suggest that pre-training with competence-basee approaches underperforms relative to knowledge-based and data-based baselines on DeepMind Control (DMC). We argue that the underlying issue with current competence-
algorithms when deployed on harder explo-
ration environments like DMC has to do with the currently used estimators for T(T;2) rather than
the objective itself. To produce structured skills that lead to diverse behaviors, [(T;2) estimators must (i) explicitly encourage diverse behaviors and (ii) have the capacity to discriminate between high- -igeeeiienion continuous skills. Current approaches do not satisfy both criteria.
Competence-bee base algorithms do not ensure diverse behaviors: Most of the best known competence- based approaches (Eysenbach al.. 2019: Gregor et al., 2017: Achiam al., 2018; Lee al., 2019), optimize the first decomposition of the mutual information H(=) H(=lr). The issue with this decomposition that while it ensures diversity of skill vectors it does not ensure diverse behavior from the policy, meaning max H(2) does not imply max H(T)). Of course, if H(z) H(2(T) ) is maximized and the skill dimension sufficiently large, then H((T) will also be maximized implicitly. Yet in practice, to learn an accurate discriminator g(=[T), the above methods assume skill spaces that are much smaller than the state space (see Table 1), and thus behavioral diversity may not be guaranteed. In contrast, the decomposition I(T;2) H(T) H(T[22 ensures diverse behaviors through the entropy term H(T). Methods that utilize this decomposition include Liu & Abbeel (2021b); Sharma et al. (2020).
Why it is important to utilize high-dimensional skills: Once a policy is capable of generating di- verse behaviors, it is important that the discriminator can distill these behaviors into distinct skills. If the set of behaviors outnumbers the set of skills, this will result in degenerate skills when one skill maps to multiple different behaviors. It therefore important that the discriminator can ac- commodate continuous skills of sufficiently high dimension. Empirically, the discriminators used in prior work utilize only low-dimensional continuous skill vectors. DIAYN (Eysenbach et al., 2019) utilized 16 dimensional skills, DADS (Sharma et al., 2020) utilizes continuous skills of dimension 5, while APS (Liu Abbeel, 2021b), an algorithm that utilizes successor features (Barreto et al., 2016; Hansen et al., 2020) for the discriminator, is only capable of learning continuous skills with dimension 10. We show how small skill spaces can lead to ineffective exploration in a simple gridworld setting in Appendix I and evidence that skill dimension affects performance in Fig. 6.
On the importance of benchmarks for evaluation: While prior competence-bard based approaches such as DIAYN (Eysenbach et al., 2019) were evaluated OpenAI Gym (Brockman et al., 2016), Gym environment episodes terminate when the agent loses balance thereby leaking some aspects of ex- trinsic signal to the exploration agent. On the other hand, DMC episodes have fixed length. We show
3

---

Under review as a conference paper at ICLR 2022
Replay Buffer
Entropy H(T))
Conditional Entropy
-H(T
h
query
T
h
Neviz Conteseive: Looe
{8. 8 B
2 ~ p(z)
[8,81
log
All
kcy
Figure 3: Architecture illustrating the practical implementation of CIC During a gradient update step, random r (s, s') tuples are sampled from the replay buffer, then a particle estimator is used to compute the entropy and a noise contrastive loss to compute the conditional entropy. The contrastive loss is backpropagated through the entire architecture. The entropy and contrastive terms are then scaled and added to form the intrinsic reward. The RI. agent is optimized with DDPG Lillicrap (2016).
in Appendix K that this small difference in environments results in large performance differences. In Fig. 11 we show that DIAYN is able to learn diverse skills in Gym but not in DMC, which is con- sistent with both observations from DIAYN and URLB papers. Due to fixed episode lengths, DMC tasks are harder for reward-free exploration since agents must learn to balance without supervision.
4 METHOD
4.1
THE CIC ESTIMATOR
From Section we are motivated to find an estimator for I(T;2) that explicitly maximizes the entropy H(s) through the second decomposition I(;;=) H(T) H(r[z). We also desire that our method's discriminator is capable supporting high-dimensional continuous skills to ensure maximal behavioral diversity. Note that is not trajectory but some function of states.
In this work, we propose a new estimator for T(T; 2) which combines the use of particle estimator for the entropy (Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann & Hyvärinen, 2010) for the conditional entropy. Our proposed sample-based estimator is:
Fai (T,, 2.) Hpariclo (T.) E f(T,22) log
Σ exp(I(r), N
1
where N is the number of samples, (8, and Hpurtcce (T) is a particle estimator (Singh al., 2003; Beirlant, 1997; Liu & Abbeel, 2021a) which estimates entropy by computing the distance between each particle h; and its k-th nearest neighbor h such that H. (T ε Eim log [/+ hi ||. The CIC estimator should achieve the best of both worlds encouraging exploration through max (T) and distilling behaviors into skills through contrastive representation learning. We first show that Eq. 3 is a valid lower bound for 2
Theorem 1. Let Feic (T, be defined as in Eq. 3, we have that Feic (T, 2) is a lower bound of the mutual information: (TT,2) Feie (T, 2), where f(T,z) )) any real function of T and
Proof. First we find variational lower bound for [TT;z) where the inequality is due to Barber & Agakov (2003).
[(T:2) H(()) H((T(2) 2 H(T) + [[og ((T)
(4)
From Contrastive Predictive Coding (CPC) (Oord al., 2018) we can also have a sample based lower bound for =(r
II high-dimensional state-action spaces the number of distinct behaviors quite large.
4

---

Under review as a conference paper at ICLR 2022
N E expl (Tg((
I(T;2) 2 PCPc (Ti, mi) = I S(Ti, - log N
(5)
As shown in Oord et al. (2018), this bound is upper bound by log which means the bound will be loose when 2(r;2) 2 log N. To overcome this limitation, we note that can also parameterize the variational density in Eq. 4 with a noise contrastive estimator q(T;|2;) = exp (f(Ti / (+ exp(((x()) We therefore have [(T;; zi) 2 H(T,) + PTPC(TT, 2i)
which completes the proof.
A favorable property of the CIC estimator is that it provides a tighter lower bound than CPC for mutual information I(T;2) 2 Feie (T, 2 FCrc(T, 2). In contrast to the CPC estimator, CIC is more suitable for exploration due to the explicit presence of H(T), which helps learning meaningful representations and behaviors as evident in recent work (Campos al., 2020; Mutti et al., 2021; Liu & Abbeel, 2021a; a; Campos et al., 2021a; Yarats et al., 2021b) whereas Eq. 5 does not explicitly encourage exploration. On the other hand, contrastive learning has been demonstrated as powerful approach for representation learning in vision and reinforcement learning (Chen al., 2020; Oord et al., 2018; Laskin et al., 2020b). It is therefore interesting to combine these two objectives into a single intrinsic reward.
4.2 INTRINSIC REWARD AND INTERPRETATION OF CIC ESTIMATION
Intrinsic Reward: We parameterize [ Trn (T) Iet (z) where T = (s, s') is transition tuple and Ivs are neural encoders. This inner product is similar the one used in the SimCLR (Chen et al., 2020) representation learning loss. We then use a particle estimator (Singh et al., 2003; Beirlant, 1997) as Liu & Abbeel (2021a) for the entropy term. Similar to Liu Abbeel (2021a) Yarats et al. (2021b) rather than using the exact form of the particle estimator we estimate the entropy up to entropy up to a proportionality constant, and therefore introduce a hyperparameter a weigh the entropy and CPC terms. With this parametrization the intrinsic reward for the unsupervised RL agent takes on the following form:
No
in 21) ar log
1
Σ BEEN2
114- - hill +11 a) ((T,)1)) - log 1 N
exp([(TT) +1))
NR
where h; an embedding of t; shown in Fig. 3, hū is kNN emhedding, N₂ is the number of kNNs, and N 1 is the number of negatives. The total number of elements in the summation is N because it includes one positive.
Explore and Exploit: We can interpret the two terms of Eq. 6 as contributing two different behaviors to the exploration algorithm. The entropy term H(T) encourages exploration by maximizing state diversity. The variational density u(r|=) encourages exploitation by ensuring that skills lead to predictable states т. Together the two terms form an intrinsic reward that incentivizes diverse yet predictable behavior from the RL agent.
Asymptotic Behavior of the Intrinsic Reward: When maximum entropy is reached the H(() term in Eq. 6 will vanish since there will stop. However, the variational density q(T 22) parameterized by CPC will continue distilling all states in the environment into skills until they are maximally distinct such that H(T[z) <
new states discover. Therefore asymptotically exploration
PRACTICAL IMPLEMENTATION
Our practical implementation consists of two main components: the RL optimization algorithm and the architecture for specifying the intrinsic reward. For fairness and clarity of comparison, we use the same RI optimization algorithm for method and all baselines in this work. Since the baselines
5

---

Under review as a conference paper at ICLR 2022
Fraction of runs with score A
1.00
Competence-baes based
SMM
DIAYN
APS
CIC
0.5
Knowledge- based
0.50
ICM
Disagreement
RND
055
Data-based
000
APT
ProtoRL
0.0
0.2
0.4
0
0.8
1.0
Expert Normalized Score
Median
IQM
Mean
Optimality Gap
CIC
ProtoRL
APT
Disagreement
ICM
D AYN
SMM
022
0.4
0.6
0.8
0.4
0.6
0.8
0.45
0.60
75
0:30
0.45
0.60
Expert Normalized Score
Figure 4: We report the aggregate statistics using stratified bootstrap intervals (Agarwal et al., 2021) for 12 downstream tasks URLB with 10 seeds, each statistic for each algorithm has 120 seeds in total. We find overall, CIC achieves leading performance on URLB terms of the IQM, mean, and OG statistics. As recommended by Agarwal et al. (2021), we use the IQM as our primary performance measure. In terms of IQM, CIC improves upon the next best skill discovery algorithm (APS) by 91% % and the next best algorithm overall (ProtoRL) by 26%.
implemented in URLB (Laskin al., 2021) use DDPG (Lillicrap et al., 2016) their backbone, we opt for the same DDPG architecture to optimize our method as well (see Appendix B). For the full algorithm
Architecture for intrinsic rewards: We use a particle estimator as in Liu & Abbeel (2021a) to es- timate H(s). To compute the variational density q(T 22), first sample skills from uniform noise ~ p(z) where p(z) is the uniform distribution over the [0, 1] interval. We then use two MLP encoders to embed Ivh (7) and I0 (2). and optimize the parameters 4/1, 4/2 with the CPC loss sim- ilar to SimCLR (Chen et al., 2020) since f(r,2) = Iv, (7) 9vz (z). We fix the hyperparameters across all domains and downstream tasks. We refer the reader to the Appendices E. and F for the full algorithm and a full list of hyperparameters.
Adapting to downstream tasks: To adapt to downstream tasks we follow the same procedure for competence- based method adaptation as in URLB (Laskin al., 2021). During the first 4k environ- ment interactions populate the DDPG replay buffer with samples and use the extrinsic rewards collected during this period to finetune the skill vector z. While it's common to finetune skills with Cross Entropy Adaptation (CMA), given limited budget of 4k samples (only 4 episodes) find that a simple grid sweep skills over the interval [0, 10 produces the best results (see Fig. 6). After this, we fix the skill and finetune the DDPG actor-critic parameters against the extrinsic reward for the remaining 96k steps. Note that competence-basee methods in URLB also finetune their skills during the first 4k finetuning steps ensuring a fair comparison between the methods. The full adaptation procedure is detailed in Appendix E.
EXPERIMENTAL SETUP
Environments We evaluate our approach on tasks from URLB, which consists of twelve down- stream tasks across three challenging continuous control domains for exploration algorithms walker, quadruped, and Jaco arm. Walker requires biped constrained 2D vertical plane to per-
It was recently shown that DDPG achieves state-of-the-art performance (Yarats et al., 2021a) on DeepMind Control (Tassa al., 2018) and more stable than SAC (Haamoja al., 2018) this benchmark.

---

Under review as a conference paper at ICLR 2022
Walker (a 0.9)
Quadruped (a 0.9)
Jaco (a 0)
Quadruped
Particl estimets ((()))
Intrinsic Rewards
53
18
216)
Environment Steps
Figure 5: We visualize the contributions the CIC intrinsic reward from the entropy and disciminator terms across three URLB domains. Since H(T) and H(T[z) terms are on different scales we set hyperpa- rameter a = 0.9 to weight the two terms equally for the Walker and Quadruped tasks. For Jaco, we find that discriminator-only CIC (a 0.0) is sufficient because random exploration results in meaningful behaviors since the training, the entropy reward decreases the entropy of CIC approaches zero. Compared to APT, CIC achieves smaller entropy as expected since the discriminator counteracts the entropy term. Compared to DIAYN, CIC achieves substantially higher entropy.
fixed to the table and therefore can't 't fall. While the CPC intrinsic reward increases throughout settles a non-zero value. Without explicit entropy maximization,
form locomotion tasks while balancing. Quadruped is more challenging due a higher-dimensional state-action space and requires quadruped to in a 3D environment to learn locomotion skills. Jaco arm 6-DOF robotic arm with finger gripper move and manipulate objects without locking. All three environments are challenging in the absence of an extrinsic reward.
Baselines: We compare CIC baselines across all three exploration categories. Knowledge- basedlines include ICM (Pathak et al.. 2017), Disagreement (Pathak et al., 2019), and RND (Burda al., 2019b).
baselines incude APT (Liu & Abbeel, 2021a) and ProtoRL (Yarats al., 2021b). Competence-based baselines include DIAYN Eysenbach et al. (2019). SMM Lee et al. (2019), and APS (Liu Abbeel, 2021b). The closest baselines to CIC are APT, which similar to CIC with information as CIC and also uses a particle entropy estimate for H()). The main difference between APS and CIC is that APS uses successor features while CIC uses contrastive estimator for the discriminator. For further details regarding baselines we refer the reader to Appendix C.
1.0 (no discriminator), and APS which uses the same decomposition of mutual
Evaluation: We follow an identical evaluation to the 2M preitnnining setup in URLB. First, pre- train each RL agent with the intrinsic rewards for 2M steps. Then, we finetune tune each agent to the downstream task with extrinsic rewards in the data-efficient regime of 100k steps. We 10 seeds across each downstream task for our method and all the baseline algorithms. For baselines, we benchmark against leading knowledge-basee have been implemented in URLB. All baselines use the same DDPG optimization algorithm to eliminate confounding factors when comparing algorithms.
data-based, and competence- approaches that
To ensure that our evaluation statistics are unbiased we use stratified bootstrap confidence intervals report aggregate statistics across M runs with N seeds as described in Rliable (Agarwal al., 2021) to report statistics for our main results in Fig. 4. Our primary success metric is the interquartile mean (IQM) and the Optimality Gap (OG). IQM discards the top and bottom 25% of runs and then computes the mean. It is less susceptible to outliers than the mean and was shown to be the most reliable statistic for reporting results for RL experiments how far policy is from optimal (expert) performance. To define expert performance we use the convention in URLB, which is the score achieved by a randomly initialized DDPG after 2M steps of finetuning (20x more steps than finetuning budget).
Agarwal et al. (2021). OG measures
RESULTS
In this section we investigate empirical answers to the following research questions: (Q1) How does CIC adaptation efficiency compare prior competence-based algorithms and exploration algorithms
7

---

Under review as a conference paper at ICLR 2022
(a) Skill projection
(b) Skill dimension
(c) Skill adaptation
(d) Skill grid sweep
0.4
Noralliza
0.2
Nomelliee score
0.2
0.3
0.8
Quad. Stand
Quad. Run
0.3
0.3
0.2
0
0.4
0.1
0.0
Projected
Not projected
0.0
0.0
2810
Rund
CEN
Baca CR51
Gaues Geves.Gnd God
Swoep
0.0
00"6""00 00000"0'0"000
dimansion
Mean shill value
Figure 6: Design choices for pre-training and adapting with skills have significant impact on performance. In (a) and (b) the agent's ' zero-shot performance is evaluated while sampling skills randomly while in (c) and (d) the agent's performance is evaluated after finetuning the skills vector. (a) we show empirically that the projecting skill vectors after sampling them from noise significantly improves the agent's performance. (b) The skill dimension is crucial hyperparameter and, unlike prior methods, CIC scales large skill vectors achieving optimal performance at dimensional skills. (c) We test several adapation strategies and find that a simple grid search performs hest given the small 4k step adaptation budget, (d) Choosing the right skill vector has substantial impact on performance and grid sweeping allows the agent to select the appropriate skill.
more broadly? (Q2) Qualitatively, does CIC discover structured skills and particularly is it able to do so in enviromments with high-dimensional state-action spaces? (Q3) Quantitatively, how does CIC behavior compare to prior methods? (Q4) Is skill selection important for efficient adaptation to downstream tasks? (Q5) How does the skill dimension affect the quality of the pre- pre-trained policy?
Adaptation efficiency of CIC and exploration baslines: Expert normalized scores of CIC and exploration algorithms from URLB are shown in Fig. 3. We find that CIC substantially outperforms prior competence- baseed algorithms (DIAYN, SMM, APS) achieving a 91% higher IQM than the next best competence-basee method (APS) and, more broadly, achieving 26% higher IQM than the next best overall baseline (ProtoRL). In further ablations, we find that the contributing factors to CIC's performance are its ability accommodate substantially larger continuous skill spaces than prior competence-based methods.
Quantitative analysis of CIC behaviors: Quantitatively, intrinsic reward profiles during pre- training and behavior entropies are shown in Fig 5. Since the CPC and etnropy terms are on different scale, we pick default hyparameter of = 0.9 that puts them on equal footing. We find that the CPC intrinsic reward increases through training while the entropy term decreases to a non-zero value. This is what we would expect to see as the discriminator distills behaviors into a set of skills with lower than pure entropy maximization without skill learning. Using a particle estimator for entropy, we find that CIC behavioral entropy is less than APT and greater than DIAYN or CIC with- out the entropy term. This suggests that the CIC agent has learned non-static skills while DIAYN skills are mostly static. Finally, we find that CIC with a = 0.0 (no entropy) on Jaco is optimal for downstream task performance. This is most likely because, unlike Walker and Quadruped, which require locomotion, Jaco tasks require reaching certain position, so low-entropy skills that take the end effector to a certain end position are favorable to skills that result in periodic motion.
For a qualitative analysis, we refer the reader to Fig. 2 and Appendix J.
Skill architecture and adaptation ablations: We find that projecting the skill to latent space be- fore inputting it as the key for the contrastive loss is an important design decision (see Fig. 6a), most likely because this reduces the diversity of the skill vector making the discriminator task simpler.
We also find empirically that the skill dimension is an important hyperparameter and that larger skills results in better zero-shot performance (see Fig. 6b), which empirically supports the hypoth- esis posed in Section 3 and Appendix I that larger skill spaces are important for internalizing di- verse behaviors. Interestingly, CIC zero-shot performance is poor in lower skill dimensions (e.g. dim(z) < 10), suggesting that when dim(z) is small CIC likely performs no better than prior competence- hased methods such as DIAYN, and that scaling to larger skill dimensions enables CIC to pre-train effecttivlyy
To measure the effect of skill finetuning described in Section 5, we sweep mean skill values along the interval of the uniform prior [0, 1] with budget of 4k total environment interactions and read out the performance on the downstream task. By sweeping, we mean simply iterating over the interval [0, 1] with fixed step size (e.g. optimal skill sampling strategy but works well due to the extremely limited number of samples for skill selection.
O, O. 0.9, 1) and setting 2, for all i. This is not an
8

---

Under review as a conference paper at ICLR 2022
We evaluate this ablation on the Quadruped Stand and Run downstream tasks. The results shown in Fig. 6 indicate that skill selection affects downstream task performance. The most optimal skill results in 8x better performance than the least optimal skill on Quadruped stand. Ablating the skill dimension, we evaluate the zero-shot performance of the agent in Walker Walk with a fixed skill of O 4 and find that the zero-shot performance monotonically increases from skill dimension 4 until reaching dimension 64 and starts decreasing for even higher skill dimensions.
8 RELATED WORK
Supervised Reinforcement Learning: To date, most of RL research has focused on supervised
RL where training supervised with an extrinsic reward function. Supervised RL has seen many breakthroughs over the last five years (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019; Silver et al., 2018; Berner et al., 2019; Andrychowicz et al., 2020; Schulman al., 2016; 2017). The field has also produced several stable RL optimization algorithms that have helped accelerated research Haarnoja et al. (2018); Hessel et al. (2018); Lillicrap et al. (2016); Schulman et al. (2017).
Unsupervised Reinforcement Learning: The sub-field of unsupervised RL consists of two primary research areas wnsupervised behavioral learning and unsupervised representation learning (Srini- vas & Abbeel, 2021). Unsupervised behavioral learning consists of learning behaviors and exploring the environment without extrinsic rewards. Unsupervised representation learning consists of learn- ing representations without supervision from high- sigh-dimensional data such as pixel observations. We evaluate our method on the recently introduced Unsupervised RL Benchmark (URLB) (Laskin et al., 2021), and focus solely the behavioral aspect of unsupervised RL in order to isolate the core issue preventing prior unsupervised skill discovery methods from exploring effectively on URLB.
The aim of unsupervised behavioral learning is to produce diverse behaviors that explore the environment without interacting with an extrinsic reward. Often referred to intrinsic motivation (Oudeyer al., 2007), this is typically achieved by defining an intrinsic reward through a self-supervised task. Most behavioral learning algorithms fall into three categories knowledge- based (Pathak et al., 2017: 2019; Burda al., 2019b;a) where the agent maximizes the error or uncertainty of some predictive model, data bata-based (Campos et al., 2021b; Liu & Abbeel, a;;;; Mutti et al., 2021; Seo et al., 2021; Yarats et al., 2021b) where the agent maximizes data diversity, and competence-bel based Eysenbach et al. (2019); Hansen et al. (2020); Liu & Abbeel (2021b); Sharma et al. (2020) where the agent maximizes the mutual information between observable variables and latent skill vector. We discuss the differences between CIC and the most closely related competence-basse exploration algorithms in Appendix D.
Unsupervised Behavioral Learning:
Unsupervised Representation Learning: Much progress in unsupervised representation learning for RL has been spurred by unsupervised learning in computer vision (Chen et al., 2020; He et al., 2020: Hénaff et al., 2020; Kingma& & Welling, 2013) and language (Brown et al., 2020; Devlin et al., 2019; Radford et al., 2019). In RI,, the most common approach for representation learning has been by adding it recently, number of works have investiagted representaton learning with autoencoders (Yarats Laskin al., 2020b; Stooke et al., 2021; Yarats et al., 2021b), and data augmentation (Laskin et al., 2020a; Yarats al., 2021a;c).
auxiliary loss in the supervised RL setting (Jaderberg al., 2017). More
al., 2019; Hafner et al., 2019; 2020), siamese networks (Schwarzer al.,
CONCLUSION
We have introduced a new competence-based algorithm - Contrastive Intrinsic Control (CIC)- which enables more effective exploration than prior unsupervised skill discovery algorithms by ex- plicitly encouraging diverse behavior while distilling predictable behaviors into skills with a con- trastive discriminator. We showed that CIC is the first competence- approach to achieve leading performance on URLB. We hope that this encourages further research in unsupervised skill discov- cry toward building more powerful exploration agents.
"The performance will, of course, be much better if we finetune the skill, but cannot do for zero-shot evaluation.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 000.//00000990 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep reinforcement learning the edge of the statistical precipice, 2021.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowiez, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
David Barber and Felix V. Agakov. The im algorithm:
variational approach to information maxi- mization. In Advances in neural information processing systems, 2003.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv:1606.05312. 2016.
J Beirlant. Nonparametric entropy estimation: An overview. International Journal of the Mathe- matical Statistics Sciences, 6:17-39, 1997.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1etp::912206600 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2019a. URL https:tt//// /ttp://///eeeeeeeeeeeeeeeee net forum? wttp:mmmmddddwwwwooaawar
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019b.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Pro- ceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learming Research, pp. 1317-1327. PMLR, 2020.
Víctor Campos, Pablo Sprechmann, Steven Stenberg Hansen, Andre Barreto, Charles Blundell, Alex Vitvitskyi, Steven Kapturowski, and Adria Puigdomenech Badia. Coverage as a principle for discovering transferable behavior in reinforcement learning, 2021a.
Víctor Campos, Pablo Sprechmann, Steven Stenberg Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell. Beyond fine-tuning: Transfer- ring behavior in reinforcement learning. In ICML 202/ Workshop on Unsupervised Reinforcement Learning, 2021b.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin, Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, 2020.
10

---

Under review as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- lions, 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In International Conference on Learning Representations, 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In Inter-
national Conference on Learning Representations, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation princi- ple for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Pro- ceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, vol- ume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sar- dinia, Italy, 13-15 May 2010. PMLR. URL gutmann1 0a html.
wer.smmrrr.prree. milr.press/v/
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Steven Hansen, Will Dabney, André Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- tional Conference on Learning Representations, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, 2020.
Matteo Hessel, Joseph Modayil, Hado Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Conference on Artificial Intelligence, 2018.
Max Jaderberg, Volodymyr Mnih. Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 2013.
Taehwan Kwon. Variational intrinsic control revisited. In International Conference on Learning Representations, 2021. URL https: http///pppeeeeeeeeeeeeeee net
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020b.

---

Under review as a conference paper at ICLR 2022
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark, 2021. URL https: http////eeeeeeeeeeeeeeeeeee.eeee
wwt.eeiwwwwweeeeeeeeeeee..e.eeee nnt Seeet/orrummdddwwPkkoo
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. CoRR, abs/1906.05274, 2019.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Hao Liu and Picter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint 2021a.
arXiv:2
Hao Liu and Pieter Abbeel. APS: active pretraining with successor features. In International Con- ference on Machine Learning, 2021b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task-agnostic exploration. In Conference on Artificial Intelligence, 2021.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0374 2018
OpenAl. Learning dexterous in-hand manipulation. CoRR, abs//808..... 088.00177, 2018.
OpenAl. Solving rubik's cube with robot hand. ArXiv, abs/1910.07113, 2019.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):265-886. 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning. 2019.
Xue Bin Peng, Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided
deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37:143:1 143:14, 2018
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings the the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp- 5171-5180. PMLR, 2019. URL http:/ http://ppcccce...eeee// html.
mmrtingss.mrr.....mmmmmmmmmm mlr sxress/v91 ppolle19..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Proceedings of Roboties: Science and Systems (RSS), 2018.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbcel. High- dimensional continuous control using generalized advantage estimation. In International Con- ference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
12

---

Under review as a conference paper at ICLR 2022
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self preficteeee representations. In International Conference on Learning Representations, 2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, De- von Hjelm, Philip Bachman, and Aaron Courville. Pretraining representations for data-efficient reinforcement learning. arXiv preprint TrXiv:2106.04799 2021b.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning. 2021.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. International Conference on Learning Representations, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Artfhur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 23(3-4):301-321, 2003.
Aravind Srinivas and Pieter Abbeel. Unsupervised learning for reinforcement learning, 2021. URL https://ient wwt.ccc//ecm/cceeoooc.eeeeeeeeee 20111122222111111 110433 QHaHBNU, pdf.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.0069) 2018.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL https: opw.reviewee net forummiddrrxooo2477777
Oriol Vinyals, Igor Babuschkin, Wojciech M Czameeki, Michael Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350--354 2019.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkami,, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through panamaraettric discriminative rewards, 2018.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv: 190.00111,, 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning, 2021a.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.
Reinforcement learning with pro- totypical representations. In International Conference Machine Learning, 2021b.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representa- fions, 2021c.
13

---

Under review as a conference paper at ICLR 2022
Tom Zahavy, Andre Barreto, Daniel J Mankowitz, Shaobo Hou, Brendan O O'Donoghue, Iurii Ke- maev, and Satinder Baveja Singh. Discovering a set policies for the worst case reward, 2021.
14

---

Under review as a conference paper at ICLR 2022
A COMPETENCE-BASER EXPLORATION ALGORITHMS
The competence- basen algorithms considered in this work aim to maximize (TT;8). The algorithms differ by ho they decompose mutual information, whether they explicitly maximize behavioral en- tropy, their skill space (discrete or continuous) and their intrinsic reward structure. We provide a list of common competence-basee algorithms in Table 1.
Table 1: Competence-bast based Unsupervised Skill Discovery Algorithms
Algoritim
Intrinsic Reward
Decomposition
Explicit max M(r)
Skill Dim.
Skill Space
SSN4HRI. (Florensa (Florenn et al., 2018)
log ( (1lae)
I[a) [(=rr)
No
5
discrete
VIC (Gregor et al., 2017)
log qo (ow))
H(2) H(=7)
No
60
discrate
VALOR (Achiam et al,, 2018)
log (=(
H[2) M(=rr)
No
discrate
DIAYN (Eysenbach
2019)
log (o)))))))))
H(2) H(alr)
No
128
discrete
DADS (Sharma et al., 2020)
(e R,
2, ((.
[[()) [(()))
Yes
5
continuous
VISR (Hansen
2020)
log (e
H(2) H(alr)
No
continuous
APS (Liu& & Abbeel, 2021b)
Pesssoor (a(2) Hagmese (8)
H(r) R(((2)
Yes
10
continuous
CIC (Ours)
(a al(a) ()umicess
H(T) P((T()]
Yes
continuous
Table 2: A list of competence-base algorithms. We describe the intrinsic reward optimized by each method decomposition the mutual information utilized by the method. explicitly maximizes state transition entropy. Finally, we note the maximal dimension used in each work and whether the skills are discrete continuous. All methods prior to CIC only support small skill spaces, either because they discrete continuous but low- wow-dimensional.
also note whether the method
B DEEP DETERMINISTIC POLICY GRADIENT (DDPG)
DDPG an actor-critic RL algorithm that performs off-policy gradient updates and learns a Q function Qø(s, a) and an actor Te(a)s). The critic is trained by satisfying the Bellman equation.
Co(d, ((s (Qolsn, a.) (5 ( no(
Here, Φ is the Polyak average of the parameters Φ. As the critic minimizes the Bellman error, the actor maximizes the action-value function.
L (0, D) Eg (o,s)s
C BASELINES
For baselines, we choose the existing set of benchmarked unsupervised RL algorithms on URLB. We provide a quick summary of each method. For more detailed descriptions of each baseline refer the reader to URLB (Laskin et al., 2021)
Competence-basell Baselines: CIC is competence- exploration algorithm. For baselines, compare it to DIAYN (Eysenbach et al., 2019), SMM (Lee et al., 2019), and APS (Liu & Abbeel, 2021b). Each of these algorithms is described in Table 1, Notably, APS is a recent state-of-the-ar competence-basee method that is the most closely related algorithm to the CIC algorithm. CIC and APS differ in their discriminator.
Knowledge-basee Baselines: For knowledge-base based baselines, we compare to ICM Pathak et al. (2017), Disagreement Pathak al. (2019), and RND Burda et (2019b). ICM and RND train dynamics model and random network prediction model and define the intrinsic reward to be propor- tional to the prediction error. Disagreement trains an ensemble dynamics models and defines the intrinsic reward to be proportional to the uncertainty of an ensemble.
Data-based Baselines: For data-based baselines we compare to APT (Liu & Abbeel, 2021a) and ProtoRL (Yarats et al., 2021b). Both methods use particle estimator to estimate the state visitation entropy. ProtoRL also performs discrete contrastive clustering as in Caron et al. (2020) as an aux- iliary task and uses the resulting clusters to compute the particle entropy. While ProtoRL is more effective than APT when learning from pixels, on state-based URLB APT is competitive with Pro- toRL. Our method CIC is effectively skill-conditioned APT agent with contrastive discriminator.
15

---

Under review as a conference paper at ICLR 2022
D RELATION TO PRIOR SKILL DISCOVERY METHODS
The most closely relatd prior algorithm to CIC is APS Liu & Abbeel (2021b). Both CIC and APS use the H(T) H(((z) decomposition of the mutual information and both used particle estima- tor (Singh et al., 2003) to compute the state entropy as in Liu & Abbeel (2021a). The main difference between CIC and APS is the diseriminator, APS uses successor features in Hansen et al. (2020) for its discriminator while CIC uses a noise contrastive estimator. Unlike successor features, which empirically only accommodate low-dimeniional continuous skill spaces (see Table 1), the noise contrastive discriminator is able to leverage higher continuous dimensional skill vectors.
The CIC discriminator is similar to the one used in DISCERN (Warde-Farles et al., 2018), a goal-
condition unsupervised RL algorithm. Both methods use contrastive discriminator by sampling negatives and computing an inner product between queries and keys. The main differences are (i) that DISCERN maximizes ((Tig) g) where g image goal embeddings while CIC maximizes (T;;2)
where 2 are abstract skill vectors; (ii) DISCERN uses the DtyyN- decomposition [(T:g) = (g)) H (g|r) while CIC decomposes through (T) (T(2),, and (iii) DISCERN discards the (g)) term by sampling goals uniformly while CIC explicitly maximizes H(T)). While DISCERN
and CIC share similarities, DISCERN operates over image goals while CIC operates abstrac
skill vectors so the two methods are not directly comparable.
Finally, another similar algorithm CIC is DADS (Sharma al., 2020) which also decomposes through (T))) While CIC uses contrastive density estimate for the discriminator, DADS uses maximum likelihood estimator similar DIAYN. DADS maximizes [s'|ss 2) and estimates entropy H((s'ls) [s) by marginalizing over such that H(s''s () = log E, g(s 2,) while CIC uses a particle estimator. Interestingly, the DADS intrinsic reward r; x log (q(s [s, 2)/ Σ, (('(s,, 2:)) looks similar to the CIC objective with zero entropy, since marginalizing over to compute entropy similar to sampling negatives for a contrastive discriminator.
16

---

Under review as a conference paper at ICLR 2022
E FULL CIC ALGORITHM
The full CIC algorithm with both pre-training and fine-tuning phases shown in Algorithm 1. We pre-train CIC for 2M steps, and finetune it on each task for 100k steps.
Algorithm Contrastive Intrinsic Control
Require: Initialize all networks: encoders 901 and 9:a. actor Ts, critic Qs. replay butfer D.
Require: Enviromment (env), M downstream tasks Tr, E M.].
Require: pre-train Ner 2M and fine-tune Npr 100K steps.
for 1..Nrr
Part I: Unsupervised Pre-training
2: Sample and encode skill & ~ p(z) and + Ivz (2)
3: Encode state S: + Gen (sx) and sample action + no(se, 2) + where (
4: Observe next state 8t-1 P(-(st, a+)
5: Add transition replay buffer <- D (st, 0,, st+1)
6: Sample minibatch from D, compute contrastive
Eq.3 update encoders gor Búzz compute CIC intrinsic reward with Eq. and update actor me and critic Q6.
7:
end for
8:
for Ti T do
Part 2: Supervised Fine- Tine-tuning
9:
Initialize all networks with weights from pre-training phase and an empty replay buffer D.
for 1... 4, 000 do
11:
Take random action ~ (0,
12:
Select skill with grid sweep over unit interval [o, 1] every 100 steps
13:
Sample minibatch from D and update actor we and critic Q
end for
15:
Fix skill that achieved highest extrinsic reward during grid sweep. for t= 4, 0.000.... Npr do
16:
17:
Encode state
94 (s:) and sample action de IT( 2) where (O) Observe state and reward 8++1, ext P(-(st,ae)
18:
Add transition to replay buffer +- DD U (st: Qe, ri 88+1) Sample minibatch from D and update actor wo and critic Qø. end for
20:
21:
22:
Evaluate performance of RL agent task Ti
end for
17

---

Under review as a conference paper at ICLR 2022
F HYPER-PARAMETERY
Baseline hyperparameters are taken from URLB Laskin et al. (2021), which were selected by per- forming grid sweep over tasks and picking the best performing set of hyperparameters. Similarly, we also performed grid sweep for CIC to pick the best performing set of hyperparameters, All hyperparameters are the same across all domains except for which is set to = 0.9 for Walker and Quadruped domains. Note that tropy terms since their absolute values are on different scales. For Jaco, we found a 0.0 work best, which means that only the discriminator contributes to the intrinsic reward. We hypothesize that particle entropy maximization is not important for Jaco arm because is fixed and has no way of falling over like Walker and Quadruped, such that meaningful behaviors can be learned with the discriminator alone.
0.9 results in equal weighing of the CPC and particle en-
Table 3: Hyper-parameter used for CIC.
DDPG hyper-parameter
Value
Replay buffer capacity
10"
Action repeat
1 states-based and for pixels-based
Seed frames
4000
n-Step returns
3
Mini-batch size
1024 states-based and 256 for pixels-based
Seed frames
4000
Discount (r)
0.99
Optimizer
Adam
Learning rate
10-4
Agent update frequency
2
Critic target EMA rate (TQ)
0.01
Features dim.
1024 states-based and 50 for pixels-based
Hidden dim.
1024
Exploration stddev clip Exploration stddev value
0.3
0.2
Number pre-training frames
up 2 x 106
Number fine-turning frames
x 10
CIC TiChhe-paaraetee
Value
Skill dim
64 continuous Uniform [0.1]
Prior
0.9 Walker, Quadruped, 0.0 Jaco 50
Skill sampling frequency (steps)
State net arch. Iv1 (s)
dim (0) + 1024 1024 - 64 ReLU MLP
Skill net arch. (z)(2)
64 + 1024 + 1024 + 64 ReLU MLP
Prediction arch.
64 1024 - 1024 64 ReLU MLP
18

---

Under review as a conference paper at ICLR 2022
G RAW NUMERICAL RESULTS
We provide a list of raw numerical results for finetuning CIC and baselines in Table 4.
Pre-trainining
environment
Donain
DDFG
CIC
[CM
Disegreemmnt
APT
ProoRL
SMM
DIYNN
APS
Fäip
538:227
671 34
417
3462:1
474.1399
544
4562:12
450124
319
465:220
Walker
Run
125
421 1 39
247-21
208115
106130
392126
306.11
426126
158.8
134116
Walk
748
895
627444
5491337
704430
7734:00
792141
770144
198
5274199
Jump
236=48
684 23
178-33
389-62
637412
648+18
617:444
9647
660-43
465+511
Quadruped
Run
157=31
110:18
357=30
459.2
492::4
5734-33
96:6
433=229
281417
Walk
229
673
126.127
293137
$36.399
7701447
412155
8011
576181
436.179
Reach bottom left
72::22
127 1
I1El1
12417
110:55
10318
129:8
4517
19:6
76:88
Jaco
Reach bottom
top
117418
172 9
9719
115110
11747
10016
125+9
13218
461:1
38:55
88111
Reach top right
94
191 5:5
103=11
13947
10025
90:10
159
4746
28:66
76
Table 4: Performance of CIC and baselines on state- state-based URLB after first pre-training for 2 x 10 steps and then finetuning with extrinsic rewards 1 10a.
H LEARNING CURVES FOR DOWNSTREAM ADAPTATION PHASE
Finetuning Learning Curves
Walker Tasks
Quadruped Tasks
Jaco Arm Tacks
Nomakzzdd Store
0.8
0.6
0.8
Normalzed
0.4
0.4
0.2
0.2
105
Disagreement
APT
DIAYN
SHM
Figure 7: Learning curves for finetuning pre- pre-trained agents for 100k steps. Task performance is aggregated for each domain, such that each curve represents the mean normalized scores over x 10 40 seeds. The shaded regions represent the standard error. CIC surpasses the performance of the prior state-of-thearr Jaco tasks while tying on Quadruped. CIC is the only algorithm that performs consistently well across all three domains.
Walker and
19

---

Under review as a conference paper at ICLR 2022
1
Toy EXAMPLE TO ILLUSTRATE THE NEED FOR LARGER SKILL SPACES
Supposs gridworid exiite
Sappoos we griwword wits skille
10e10 Oridword, red agent
V0eto Oriwwats, rod agent
K2 REG Ho-tr
Fimma) Nir) Ri(111
vilted
R2 p(2))
H((T)
Ensunes medimal behoeitoral diveritty
How cond antrps?
How nétirall anal enriayy
[(=[r)= odlice dedtet coiM
H((T=)-- 0 Map pallo wodisieet statey
Figure 8: A gridworld example motivating the need for large skill spaces. In this environment, we place an agent in 10 x 10 gridworld and provide the agent access to four discrete skills. We show that the mutual information objective can be maximized by mapping these four skills the nearest neighboring states resulting behavioral diversity and exploring only four the hundred available states.
We illustrate the need for larger skill spaces with a gridworld example. Suppose we have an agent
in 10 10 sized gridworld and that have four discrete skills our disposal. Now let T
and consider how we may achieve maximal =(r;=) in this setting. If we decompose T(T; = H(z) H(=lT) then we can achieve maximal H2)2) by sampling the four skills uniformly
p(z). We can achieve H((27)) = 0 by mapping each skill to a distinct neighboring state of the agent. Thus,
our mutual information is maximized but as a result the agent only explores four of the hundrend available states in the gridworld.
Now suppose we consider the second decomposition I(T;2) = H((T) H(T[2). Since the agent is maximizing H((T) it is likely to visit a diverse set of states at first. However, as soon as it learns an accurate discriminator we will have H(()2) and again the skills can be mapped to neighboring states to achieve minimal conditional entropy. As result, the skill conditioned policy will only be able to reach four out of the hundrend possible states in this gridworld. This argument is shown visually in Fig. 8.
Skill spaces that are too large can also be an issue. Consider if we had 100 skills at our disposal in the same gridworld. Then the agent could minimize the conditional entropy by mapping each skill to a unique state which would result the agent memorizing the environment by finding a one-to- one mapping between states and skills. While this is potential issue it has not been encountered in practice yet since current competence-basee based methods support small skill spaces relative to the observation space of the environment.
20

---

Under review as a conference paper at ICLR 2022
J QUALITATIVE ANALYSIS OF SKILLS
We provide two additional qualitative analyses of behaviors learned with the CIC algorithm. First, take simple pointmass setting and set the skill dimension in order ablate the skills learned by the CIC agent in a simple setting. We sweep over different values of and plot the behavioral flow vector field (direction in which point mass moves) Fig.9. We find that the pointmass learns function of
skills that produce continuous motion and that the direction of the motion changes the skill value. Near the origin the pointmass learns skills that span all directions, while near the edges the point mass learns to avoid wall collisions. Qualitatively, many behaviors are periodic.
Behavior flow for different skill values
2:00:00
2 = 0.20
0.40
Pointmass with skill
0.60
0.80
1:00
Figure 9: Learning curves for finetuning pre-trained agents for 100k steps. Task performance is aggregated for each domain, such that each represents the normalized scores x 10 40 seeds. The shaded regions represent the standard error. CIC surpasses the performance of the prior state-of-the-art on Walker and Jaco tasks while tying on Quadruped. CIC is the only algorithm that performs consistently well across all three domains.
Qualitatively, we find that methods like DIAYN that only support low dimensional skill vectors and do not explicitly incentivize diverse behaviors in their objective produce policies that map skills to a small set of static behaviors. These behaviors shown in Fig. 10 are non-trivial but also have low behavioral diversity and are not particularly useful for solving the downstream task. This observation consistent with Zahavy et al. (2021) where the authors found that DIAYN maps to static "yoga" poses in DeepMind Control. In contrast, behaviors produce by CIC dynamic resulting flipping, jumping, and locomotive behaviors that can then be adapted to efficiently solve downstream tasks.
21

---

Under review as a conference paper at ICLR 2022
DIAYN skills produce static "yoga" poses
CIC skills produce dynamic behaviors
Figure 10: Qualitative visualization of DIAYN and CIC pre-training on the Walker and Quadruped domains from URLB. Confirming findings in prior work Zahavy et al. (2021), we also find that DIAYN policies produce static but non-trivial behaviors mapping to "yoga" poses while CIC produces diverse and dynamic behaviors such as walking, flipping. and standing. Though it's hard to see from these images, all the DIAYN skills get stuck in frozen poses while the CIC skills are producing dynamic behavior with constant motion.
K OPENAI GYM VS. DEEPMIND CONTROL: How EARLY TERMINATION LEAKS EXTRINSIC SIGNAL
Prior work on unsupervised skill discovery for continuous control (Eysenbach al., 2019; Sharma et al., 2020) was evaluated OpenAl Gym (Brockman et al., 2016) and showed diverse exploration on Gym environments. However, Gym environment episodes terminate early when the agent loses balance, thereby leaking information about the extrinsic task (e.g. balancing or moving). However, DeepMind Control (DMC) episodes have fixed length of steps. In DMC, exploration is therefore harder since the agent needs to learn to balance without any extrinsic signal.
To evaluate whether the difference the two environments has impact on competence-basee explo- ration, run DIAYN the hopper environments from both Gym and DMC. We compare ICM, a popular exploration baseline, and Fixed baseline where the agent receives an intrinsic reward of 1 for each timestep and algorithms receive extrinsic rewards. We then measure the extrinsic reward, which loosely corresponds to the diversity of behaviors learned. Our results in Fig. 11 show that indeed DIAYN able to learn diverse behaviors in Gym but not in DMC while ICM is able to learn diverse behaviors in both environments. Interestingly, the Fixed baseline achieves the highest reward the Gym environment by learning stand and balance. These results further motivate us to evaluate on URLB which is built on top of DMC.
OpenAl Gym Hopper
DeepMind Control Hopper
0.30
05
0.25
94
3.20
5:15
Resets when agent loses balance
8.0
Rasets are fixed at 1000 steps
3.00
166
Endirnment
186
To empirically demonstrate issues inherent to competence-based exploration methods, we DIAYN (Eysenbach al., 2019) and compare it to ICM (Pathak al., 2017) and Fixed baseline where the agent receives an intrinsic reward 1.0 for each timestep and extrinsic reward on both OpenAl Gym (episode resets when agent loses balance) and DeepMind Control (DMC) (episode is fixed for Ik steps) Hopper environments. Since Gym and DMC rewards are on different scales, we normalize rewards based on the maximum reward achieved by any algorithm ( 1k for Gym, 3 for DMC). While DIAYN is able achieve higher extrinsic rewards than ICM Gym, the Fixed intrinsic reward baseline performs hest. However, DMC the Fixed DIAYN agents achieve near-zero reward while ICM does not. This consistent with findings of prior work that DIAYN is able to learn diverse behaviors in Gym (Eysenbach et al., 2019) as well as the observation that DIAYN performs poorly on DMC environments (Laskin et al., 2021)
Figure 11:
22

---

Under review as a conference paper at ICLR 2022
L PSEUDOCODE FOR THE CONTRASTIVE DISCRIMINATOR IN CIC
CIC consists of two terms [(T; 2) (T)) H(T()) 2 (()) + E q(T=)) the entropy H()) is estimated with particle estimator Singh et al. (2003); Liu & Abbeel (2021a) while the discrimina- tor ((T(2) is estimated with a contrastive loss introduced in this work. Note that contrastive learning in CIC is different than prior vision- based contrastive learning such CURL Laskin al. (2020b), since we are not performing contrastive learning over augmented images but rather over state tran- sitions and skills. The contrastive objective in CIC is used for unsupervised learning of behaviors while in CURL it is used for unsupervised learning of visual features.
We provide pseudocode for discriminator below:
def discriminator.loss (states.. next_states. skills, temp):
states and skills skills sampled from uniform dist [0,1] during agent rollout
sampled from replay buffer
states / next states: dim (B, state)
skills: dim (B, D skill)
transitions concat next_states, dim=l)
10
11
query = skill net (B, D-hidden) -> (B, D_hidden)
key = transition ietaiioonnnetaaastttioonnn # (B, 2s D_satee) -> (B, D D.hidden)
14
query normalize (query. dim=1)
15
key = normalize (key. dim=l)
16
logits (query key.T) temp (B, B)
18
19
positives are on diagonal, negatives are off diagonal # for each skill, negatives are sampled from transitions
20
# while skills are fixed
loss cross_entrops (logits)
25
24
return loss
Listing 1: CIC discriminator loss
We note that this is substantially different from prior contrastive learning works in RL such as CURL (Laskin et al., 2020b), which perform contrastive learning over images.
def curl loss (obs, W, temp):
observation images are sampled from replay buffer obs: dim (B, C. Н. W)
W: projection matrix (D.hidden, D_hidden)
query aug(obs)
key = aug(obs)
query net (query) # (B, D D.hiidee)
12
key net (B, D_hidden)
13
14
logits matmul maamul (query, W), key T) temp # (B, B)
16
positives are on diagonal negatives are off diagonal loss = cross_entropy (logits)
17
18
return loss
Listing 2: CURL contrastive loss
23

---

Under review as a conference paper at ICLR 2022
M ON TIGHTER ESTIMATES OF MUTUAL INFORMATION
In this work we have presented CIC new competence-based algorithm that achieves leading performance on URLB compared prior unsupervised RL methods. We shown that CIC results in tighter lower bound on mutual information than CPC by including the entropy term.
One might wonder whether estimating the exact mutual information (MI) or maximizing the tightest lower bound thereof is really the goal for unsupervised RL. In unsupervised representation learn- ing, -ftte-oft methods like CPC and SimCLR maximize the lower bound of MI based on Noise Contrastive Estimation (NCE). However, proven in CPC (Oord et al., 2018) and illustrated Poole et al. (2019) NCE is upper bounded by log N, meaning that the bound is loose when the MI is larger than log N. Nevertheless, these methods have been repeatedly shown to excel in practice. In Tschannen et al. (2020) the authors show that the effectiveness NCE results from the inductive bias in both the choice of feature extractor architectures and the parameterization of the employed MI estimators.
We have similar belief for unsupervised RL that with the right parameterization and inductive bias, the MI objective will facilitate behavior learning in unsupervised RL. This is why CIC lower bounds MI with (i) the particle based entropy estimator ensure explicit exploration and (ii) trastive conditional entropy estimator to leverage the power of contrastive learning to discriminate skills. As demonstrated in our experiments, CIC outperforms prior methods, showing the effective- ness of optimizing an intrinsic reward with the CIC MI estimator.
N LIMITATIONS
While CIC achieves leading results on URLB, we would also like to address its limitations. First, in this paper only consider MDPs (and not partially observed MDPs) where the full state is observable. We focus on MDPs because generating diverse behaviors in environments with large exploration. Combining CIC
state spaces has been the primary bottleneck for competence- with visual representation learning to scale this method to pixel-based inputs is a promising future direction for research not considered in this work. Another limitation that our adaptation strategy to downstream tasks requires finetuning, Since we learn skills, it would be interesting investigate alternate ways of adapting that would enable zero-shot generalization such as learning generalized reward functions during pre- pre-training.
24


Reviews:
[{"review_id": "eHLnxmOOVKG", "reviewer": "Reviewer_cP9K", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe empirical evaluation carried out in the paper is extensive with many baseline algorithms for skill-discovery from each class (knowledge-based, data-based, competence-based) on the recently proposed URLB benchmark. Further, evaluation metrics (IQM, Optimality gap etc.) used to measure performance are adopted from the recommendations in [1].  \n\nThe discussion and analysis on the reasons for failure of current competence-based skill discovery algorithms. Ablation experiments (in Figure 6) help justify the design choices made by the CIC algorithm. \n\nWeaknesses:\n\nThe proposed algorithm seems like a variation on an existing algorithm [2]. It refines some of the practical design choices used in the general framework of competence-based algorithms for unsupervised skill discovery. However, the authors have shown the differences between the various algorithms (in Table 1) and discussed their pros & cons which is useful to contextualize their contributions.\n\nA few questions I have for the authors on the high-level motivations behind their algorithm design and would like some clarification on:\n\n1] If we intuitively think of the notion of a skill as a form of abstraction of long-term behavior. For example motion primitives like walking, flipping etc. as shown in the paper which occurs over maybe a few tens or hundreds of steps. So, why do several skill discovery algorithms use only highly localized information in the state space (various tau instantiations such as single states or in this case single state-transitions (s, s’)) to infer its corresponding skill latents? Isn’t it more intuitive to infer these latents from more “global” quantities like entire episodes of policy rollouts. Could the authors comment on this? \n\n2] To maximize the mutual information between state transitions and skills as defined In the CIC, we need to maximize the first term ($H[\\tau]$) and minimize the second term ($H[\\tau | z]$). This conditional entropy would be minimal when the corresponding distribution $p(\\tau | z)$ is sharp/narrow (ideally like a delta-like density function) over the state-transition space. This seems rather counterintuitive to me. If we think of a single skill latent, say walking, shouldn’t the density $p( \\tau | Z=z_{walk})$ have a high value for all the possible state-transitions of the walking primitive, which would be a rather wide distribution. Wouldn’t the pressure to keep this distribution as narrow as possible over the state-transition space given a single skill latent lead to several latents codes which essentially cover the same underlying behavior (redundant copies of different walking styles). Wouldn’t this be undesirable for generalization on downstream tasks which would require composing these unsupervised skills?    \n\n3] The authors argue for the need for increasing the dimensionality of the skill latents to ensure skill can be decoded back to a diverse set of behaviors. Couldn’t this be achieved by largely retaining the small latent spaces used for skills in prior work and using a more expressive policy decoder. This could allow for greater representation flexibility when the skill latents are decoded back to the action-space and ensure that skill latents give rise to a diverse set of behaviors.  \n\nWriting/Presentation:\n\nThe paper on the whole is well-written and easy to follow.  \n\nI found the phrasing in this sentence rather confusing and ambiguous. “If the set of behaviors outnumbers the set of skills, this will result in degenerate skills -- when one skill maps to multiple different behaviors”. What does “behaviors” refer to in this context -- action trajectories? If so, isn’t it expected that the set of skills would be much smaller (essentially a compressed representation) than the total number of “unique” action sequences. Unless, the authors  \n\nA few minor typos I found are below:\n\n“Why most competence-base algorithms …. ” -> “competence-based algorithms … ”\n“Both CIC and CIC use ...” -> do you mean CIC and APS?\nIn Algorithm 1 -> “Contrastive Intrinisc Control” -> “Intrinsic”\n\n[1] Agrawal et.al, “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, NeurIPS 2021.\n\n[2] “APS: Active Pretraining with Successor Features”, ICML 2021.\n", "summary_of_the_paper": "The paper proposes an algorithm (Contrastive Intrinsic Control) for unsupervised skill discovery by maximizing mutual information between skill latents and state transitions. The proposed algorithm is a refinement over existing methods [2]. It uses a contrastive method to estimate conditional entropy and measures entropy on state-transitions as opposed to simply states as done in previous methods [2]. The proposed algorithm shows good performance gains compared to existing competence-based skill discovery algorithms. Further, the paper also contains a rather extensive empirical evaluation of various skill discovery algorithms on the recently proposed URLB benchmark. \n", "summary_of_the_review": "Although the proposed algorithm CIC is a variation on an existing algorithm [2], it shows impressive performance gains over several existing algorithms on a large suite of continuous control tasks from the URLB benchmark. These large-scale empirical evaluations and analysis of several algorithms for unsupervised skill discovery methods on a standard benchmark such as URLB would benefit the community. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "d96-ifjTkIw", "reviewer": "Reviewer_VTZ1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I report below some detailed comments and concerns that the authors might address in their author response.\n\nMUTUAL INFORMATION OBJECTIVE\n\n1) There is now a bunch of works targeting the mutual information between a code and the visited states to the purpose of unsupervised RL (some of them are summarized in Table 1 of the paper). Every work is proposing a lower bound to the mutual information. I was wondering if there is a formal way to compare the lower bound of CIC with previous works. Which one is the tightest? Do the authors believe that getting the tightest bound of the MI is really the goal in this setting, or even maximizing the exact MI would not be necessarily better than other methods (i.e., a good inductive bias matters more than the MI approximation)?\n\n2) Since the work motivates the approach as an approximation of the MI, as it is common in previous works as well, I was wondering if the authors also considered a direct non-parametric estimation of the MI (e.g., https://journals.aps.org/pre/pdf/10.1103/PhysRevE.69.066138) instead of independent estimates of the state entropy and the conditional entropy. Especially, such a direct estimation would not require the additional hyper-parameter $\\alpha$.\n\nMETHODOLOGY\n\n3) Can the authors clarify what is the entropy term $H(\\tau)$ denoting? Is it the entropy of state-to-state transitions, or the entropy of the joint probability of two states within a trajectory, or something else? Especially, what is the intuition behind using $\\tau$ instead of $s$?\n\n4) The method is built upon a non-specific base algorithm (DDPG) that was originally developed for standard RL, i.e. RL problems where the reward does not change over time but it comes from a consistent reward function. Do the authors experienced any instability working with the non-Markovian intrinsic rewards? Do they believe that the methodology could benefit from an objective-specific algorithm, i.e., an algorithm carefully designed to work with this kind of intrinsic reward?\n\nEXPERIMENTS\n\n5) How can we rule out the possibility that CIC is just a better way to pre-train a DDPG agent in these settings w.r.t. other baselines? This would be significant anyway, but do the authors believe that the same results would generalize to different base algorithms (say TRPO, SAC, A2C...)? \n\n6) Moreover, DDPG is known to be quite strong on continuous control tasks. Do the authors believe that the combination of DDPG and CIC would be successful in different settings (e.g., visual discrete domains such as Atari games) as well? Or perhaps the base algorithm should be selected to accomodate the specific domain?\n\n7) CIC is quite similar to APS (Liu and Abbeel, 2021), as they both employ non-parametric entropy estimation and a discriminator loss, based on contrastive predictive coding and successor representations respectively. However, in the reported results CIC is way better than APS. Do the authors think it is the different discriminator loss the main cause for this performance gap, or there is some other factor at play? Can they confront the discriminator rewards of APS and CIC (as in Figure 5)?\n\n8) From my understanding, the empirical results are not directly comparable with the URLB (Laskin et al., 2021) despite a very similar setting. I see that the benchmark is very recent, and thus should be considered concurrent to this work, but I believe that reporting a direct comparison with their results would further strengthen the empirical analysis.\n\n9) The results section seems to imply that a key factor under the improvement over previous competence-based methods is the ability of CIC to cope with a larger skill space. Can the authors clarify why previous methods are prevented to work with a comparable skill space? Can they also provide a comparison with previous methods when working with the same (potentially lower-dimensional) skill space?\n\nMINOR\n- Adaptation efficiency paragraph: Fig. 3 is reported instead of Fig. 4;\n- The normalized score of Fig. 6 does not seem to match the one of Fig. 4;\n- It is not easy to track the different baselines in Fig. 4 (top). What is this plot representing?", "summary_of_the_paper": "This paper tackles the problem of unsupervised pre-training of a (code-conditioned) policy to improve the performance of downstream RL tasks. In line with previous works in unsupervised skills discovery, it proposes a method, called CIC, to maximize a variational lower bound to the mutual information between the code and the visited states. The lower bound is obtained through the combination of non-parametric state entropy estimation and a contrastive predictive coding loss for the conditional entropy. The paper provides an empirical analysis of CIC over a set of continuous control domains.", "summary_of_the_review": "To my understanding, the main selling points of this paper are:\n- It tackles the very relevant problem of unsupervised pre-training for reinforcement learning;\n- The methodology is clear, and a quite natural extension of previous works in unsupervised skills discovery literature;\n- Strong empirical results, CIC seems to advance significantly the state-of-the-art performance of unsupervised pre-training in continuous control domains.\n\nInstead, potential shortcomings are:\n- The novelty seems limited, as CIC is essentially similar to APS (Liu and Abbeel, 2021) with a different discriminator loss (which has been employed for unsupervised skills discovery before);\n- It is not completely clear from the paper what are the specific factors that lead to such a performance improvement over previous works.\n\nWhereas the reported empirical progress might be a sufficient reason for acceptance, my current evaluation is just slightly positive in consideration of the mentioned concerns. I do not think the limited novelty is a crucial problem here, if the authors could better clarify in their response how the CIC methodology is so successful, I will consider raising my score to a clear accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Grspvahjoz9", "reviewer": "Reviewer_h3mC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The empirical evaluation is rigorous in comparison to common practice (120 runs and IQM stabilization) with strong SOA and baselines. \n\nLiked the structured review of prior art that organizes work in an interesting thematic way. \n\nI liked the empirical decomposition of reward into entropy and discriminator terms. It is interesting that latent state entropy is important early in learning and task discrimination is more important later on. \n\nUseful to know that higher dimensional skills are important (64D performed best). \n\nTechnically I(T;Z)  =  H(T) - H(T|Z)  =  H(Z) - H(Z|T) so theoretically it should not matter which way it is decomposed. The way the terms are calculated, however, could have a significant effect on the practical performance. \n\nIt isn’t clear what a “simple grid sweep of skills over the interval [0,1]” means.  \n\nEarly stopping does not “leak information” so much as stop wasted exploration in irrelevant parts of the state space? \n\nTypo: “we use particule” -> particle \n\nThe text refers to optimality gap, but the figures see to use expert normalized score which I assume is algorithm performance / DDPG baseline performance. \n\nDid not fully understand the implications of the noise and projection argument, but it seems like an important and worth while design decision to investigate. ", "summary_of_the_paper": "The paper builds upon the DIAYN idea (Eysenback 2018) that an agent could develop skills in an unsupervised environment by finding a set of skills that collectively visits the whole state space but encourages each skill to cover a different subspace and later use one of these skills to simplify the learning of a downstream task. In this paper skills are learned in an unsupervised way using a mutual information based objective. The mutual information between latent states T and skill vector Z, I(T;Z), is decomposed as I(T;Z)=H(T)-H(T|Z) which the paper argues leads to explicit maximization of diversity of latent states T as well as distinct skills with focused effects by penalizing with H(T|Z). More diversity equals better exploration of more distant states leading to more interesting behaviors. The paper explains this decomposition also allows them to user higher-dimensional skill vectors that improve representational capacity and downstream performance. The paper develops a method to calculate the terms of this decomposition and formally shows it is a lower bound for the true mutual information. The latent state entropy H(T) is estimated using an unnormalized k-nearest neighbor method requiring an ad hoc scaling factor \\alpha and the conditional entropy H(T|Z) is calculated using an NCE supervised neural net.  The paper illustrates the method by using their loss function to pretrain a DDPG architecture on unsupervised scenarios and then showing a benefit on downstream tasks. The proposed method, CIC, shows a larger IQM stabilized expert normalized score compared to state of the art methods.  Experiments also show that high-dimensio", "summary_of_the_review": "The paper proposes a new algorithm for unsupervised behavior learning that is rigorously shown to be more effective and clearly argues for its design choices through supplementary experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8qwu1NKYxRf", "reviewer": "Reviewer_tKs2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The experiments are done on URLB, which provides a good evaluation scheme for unsupervised RL. Well-established and common evaluation schemes are important for assessing methods empirically.\n* The empirical performance, which is provided with relevant statistics, is good compared to multiple baseline methods.\n* The analysis on the effect of different choices of multiple hyperparameters (Fig.6).\n\n**Weaknesses**\n\n* This work's originality is somewhat limited. Particle-based entropy maximization with state representations trained using a contrastive learning scheme has been explored in APT (Liu & Abbeel, 2021a). Using the same entropy maximization form for skill discovery was done by APS (Liu & Abbeel, 2021b).\n* The motivation for using noise-contrastive estimation is not entirely clear. While the authors make a comparison with CPC as a lower bound of mutual information, CPC is not very commonly used in skill discovery and the usual variational lower bound can already be tight if the variational approximation $q$ approximates the true distribution $p$ perfectly.\n* Theorem 1 is not technically correct because of the particle-based entropy term in $F_{CIC}$, which is also supported by the authors in that they introduced the weighting hyperparameter $\\alpha$ due to that it doesn't consider the proportionality constant (Sec.4.2).\n* The authors employ $I(\\tau; z) = \\mathcal{H}(\\tau) - \\mathcal{H}(\\tau|z)$ as the decomposition of the mutual information (the 2nd line in Sec.4.1), but they use $q(z|\\tau)$ instead of $q(\\tau|z)$ for the rest of Sec.4.1.\n* I think the derivation of the noise-contrastive estimator in Sec.4.1 needs more details. For instance, if there are noise samples, where do the correct samples come from?\n* Using the notation $\\tau$ for something other than trajectories can be misleading.\n* Lack of empirical analysis with different values of $\\alpha$.", "summary_of_the_paper": "To tackle the unsupervised skill discovery problem, the authors attempt to maximize the mutual information between (latent) skills and states $I(\\tau; z)$ by using the nonparametric particle-based entropy estimation for $\\mathcal{H}(\\tau)$ and noise-contrastive estimation for $\\mathcal{H}(\\tau|z)$. They evaluate their method and the baselines on the Unsupervised Reinforcement Learning Benchmark (URLB), where the agents are pre-trained without extrinsic rewards and fine-tuned for downstream tasks with extrinsic rewards and fewer environment steps.", "summary_of_the_review": "While the empirical results basically show that CIC can outperform multiple baselines on URLB with an appropriately tuned value of $\\alpha$, I am mainly concerned about the correctness of the claims and the novelty of the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the question of skill discovery in reinforcement learning: can we (without supervision) discover behaviors so that later (when supervision is available via a reward signal) we can learn faster? The paper proposes a new contrastive loss that an agent can optimize for this purpose, based on a decomposition of mutual information between skills and transitions. The reviewers praised the extensive experimental evaluation and good empirical results, as well as the analysis of failure modes of related algorithms.

Unfortunately, there appeared to be errors in the derivation and implementation. (These include typos in derivations that made them difficult to follow, as well as uploaded code that didn't match the experimental results.) While the authors claim to have fixed all of them, the reviewers were not all completely convinced by the end of the discussion period. In any case, these errors caused confusion during review; so, whether the errors are fixed or not, it seems clear that there hasn't been time for a full evaluation of the corrected derivations and code. For this reason, it seems wise to ask that this paper be reviewed again from scratch before being published.
Decision: Reject