Unsupervised Reinforcement Learning with Contrastive Intrinsic Control
Authors: Michael Laskin
Hao Liu
Xue Bin Peng
Denis Yarats
Aravind Rajeswaran
Pieter Abbeel
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Unsupervised Learning

Abstract: We introduce Contrastive Intrinsic Control (CIC), an unsupervised reinforcement learning (RL) algorithm that maximizes the mutual information between state-transitions and latent skill vectors. CIC utilizes contrastive learning between state-transitions and skills vectors to learn behaviour embeddings and maximizes the entropy of these embeddings as an intrinsic reward to encourage behavioural diversity. We evaluate our algorithm on the Unsupervised RL Benchmark (URLB) in the asymptotic state-based setting, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. We find that CIC improves over prior exploration algorithms in terms of adaptation efficiency to downstream tasks on state-based URLB.

Contrastive Intrinsic Control for Unsupervised Reinforcement Learning
Michael Laskin
Hao Liu UC Berkeley
Xue Bin Peng UC Berkeley
UC Berkeley mllaskin@brrkeey..o edu
Denis Yarats NYU, Meta AI
Aravind Rajeswaran UC Berkeley, Meta AI
Pieter Abbeel UC Berkeley, Covariant
Abstract
We introduce Contrastive Intrinsic Control (CIC), unsupervised reinforcement learning (RL) algorithm that maximizes the mutual information between state- transitions and latent skill vectors. CIC utilizes contrastive learning between state-transitions and skills vectors to learn behaviour embeddings and maximizes the entropy of these embeddings diversity. We evaluate our algorithm on the Unsupervised RL Benchmark (URLB) in the asymptotic state-based setting, which consists of long reward-free pre- training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. We find that CIC improves over prior exploration algorithms terms of adaptation efficiency to downstream tasks on statebbased URLB.
intrinsic reward to encourage behavioural
Deep RL is a powerful approach toward solving complex control tasks the presence of extrinsic rewards. Successful applications include playing video games from pixels [T] mastering the game of Go B robotic locomotion 5 6 and dexterous manipulation 7 g policies. While effective, the above advances produced agents that are unable to generalize to new downstream tasks beyond the one they were trained to solve. Humans and animals on the other hand are able to acquire skills with minimal supervision and apply them to solve variety of downstream tasks. In this work, we seek to train agents that acquire skills without supervision with generalization capabilities by efficiently adapting these skills to downstream tasks.
Over the last few years, unsupervised RL has emerged a promising framework for developing RL agents that can generalize to new tasks. In the unsupervised RL setting, agents are first pre-trained with self-supervised intrinsic rewards and then finetuned downstream tasks with extrinsic rewards. Unsupervised RL algorithms broadly fall into three categories ahowledge-basee data-based, and competence- based methods Knowledge-baad based methods maximize the error or uncertainty of predictive model 212
13 14 Data- based methods maximize the entropy of the agent's 's visitation [I5] This work falls
16]. Competence- based methods learn skills that generate diverse behaviors 17 18
into the latter category of competence-based exploration methods.
Unlike knowledge-based and data-based algorithms, competence- based algorithms simultaneously address both the exploration challenge as well as distilling the generated experience in the form of reusable skills. This makes them particularly appealing, since the resulting skill-based policies (or skills themselves) can be finetuned to efficiently solve downstream tasks. While there are many self-supervised objectives that can be utilized, our work falls into a family of methods that learns skills by maximizing the mutual information between visited states and latent skill vectors. Many earlier
Project website and code: httpp::/sstte woatseess.ogggee google. www.oo///viiii wttpe///////eeeeeeeeeeeeeeeeeeee http://rieeeccccncuree222222222
"Thee categories for exploration algorithms were introduced by 110 and inspired by
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Leap
Flip upright
Nudge brick
Aaka Jog
Walk left
Figure 1: Qualitative visualizations of unsupervised skills discovered in Walker, Quadruped, and Jaco arm environments. The Walker learns balance and move, the Quadruped learns to flip upright and walk, and the 6 DOF robotic arm learns how move without locking. Unlike prior competence-basee methods for continuous control which evaluate on OpenAI Gym (e.g. [2], which reset the environment when agent loses balance, CIC is able to learn skills in fixed episode length environments which are much harder to explore (see Appendix Tp.
works have investigated optimizing such objectives [17 methods have been empirically challenging to train and have under-performed when compared to knowledge and data-based methods (21).
18 19 20 However, competence- based
In this work, we take closer look at the challenges of pre-training agents with competence- algorithms. We introduce Contrastive Intrinsic Control (CIC) an exploration algorithm that uses a new estimator for the mutual information objective. CIC combines particle estimation for state entropy (22 [5] to both generate diverse behaviors (explore) and discriminate high-dimensional continuous skills (exploit). To the best of our knowledge, CIC is the first exploration algorithm utilize noise contrastive estimation to discriminate between state transitions and latent skill vectors. Empirically, show that CIC adapts downstream tasks more efficiently than prior exploration approaches on the state-based Unsupervised Reinforcement Learning Benchmark (URLB). CIC achieves 79% higher returns downstream tasks than prior competence- based algorithms and 18% higher returns than the next-best exploration algorithm overall.
and noise contrastive estimation (23) for the conditional entropy which enables it
Background and Notation
We operate under the assumption that system is described by Markov Decision Process (MDP) (24). An MDP consists of the tuple AS, P, r, which has states E S, actions E A, transition dynamics p(s'|s, a) P, reward function r, and a discount factor y. In an MDP, at each timestep I, an agent observes the current state 8, selects an action from a policy T-Iss), and then observes the reward r and next state s' once it acts the environment. Note that usually refers to an extrinsic reward. However, in this work we will first be pre-training an agent with intrinsic rewards jii and finetuning on extrinsic rewards ex
Markov Decision Process:
For convenience we also introduce the variable r = (s, s') which is a tuple denoting a transition between two consecutive states. Importantly, does not denote state-action trajectory. In addition to the standard MDP notation, we will also learning skills 2 E Z where Z is the skill set, which can be discrete continuous real-valued vector space, and our policy will be skill-conditioned a T (-18,2).
Unsupervised Skill Discovery through Mutual Information Maximization: Most competence- based approaches to exploration maximize the mutual information between states and skills. Our work and large body of prior research [17 20 18 25 26 27 aims to maximize mutual information objective with the following general form:
[(T; 2) H(z) 1(2)r) = -
- H(([2)
(1)
Competence-based algorithms use different choices for and can condition on additional information such as actions or starting states. For a full summary of competence-ben algorithms and their objectives see Table

---

Replay Buffer
Intrinsic Reward
Representation Learning
query
h
I-NN
Noise Concastive
{s, a, s
B
2 p(z)
1 Myuniste()
los IA,
key
im1
Figure 2: Architecture illustrating the practical implementation of CIC During a gradient update step, random т = (s, s') tuples are sampled from the replay buffer, then a particle estimator is used to compute the entropy and a noise contrastive loss to compute the conditional entropy. The contrastive loss is backpropagated through the entire architecture. The entropy and contrastive terms are then scaled and added to form the intrinsic reward. The RL agent is optimized with DDPG (29).
Lower Bound Estimates of Mutual Information: The mutual information [s;; z) is intractable to compute directly. Since we wish to maximize [(s; 2), we can approximate this objective by instead maximizing lower bound estimate. Most known mutual information maximization algorithms use the variational lower bound introduced [28]:
I(r;))= = -
H(T[2) 2 H(T) + E,,, (l,g [([[2)]
(2)
The variational lower bound can be applied to both decompositions of the mutual information. The design decisions of competence-based a algorithm therefore come down to (i) which decomposition of [TT; 2) to use, (ii) whether use discrete or continuous skills, (iii) how to estimate (() or (()), and finally (iv) how to estimate H (2lr) (T
Motivation
Results from the recent Unsupervised Reinforcement Learning Benchmark (URLB) 21 show that competence-based approaches underperform relative to knowledge-basse based and data-based baselines on DeepMind Control (DMC). We argue that the underlying issue with current competence- based algorithms when deployed on harder exploration environments like DMC has to do with the currently used estimators for [(T; z) rather than the objective itself. To produce structured skills that lead to diverse behaviors. 7 /(T; =) estimators must (i) explicitly encourage diverse behaviors and (ii) have the capacity to discriminate between high-dimensional continuous skills. Current approaches do not satisfy both criteria.
Competence- basee algorithums do not ensure diverse behaviors: Most of the best known competence- based approaches [17] [8] 25] 26). optimize the first decomposition of the mutual information H(2) H= The issue with this decomposition that while it ensures diversity of skill vectors it does not ensure diverse behavior from the policy, meaning max H(2) does not imply max H(T)). Of course, if (2) ([=/T) is maximized and the skill dimension is sufficiently large, then H(T) will also be maximized implicitly. Yet in practice, to learn an accurate discriminator g(2/7), the above methods assume skill spaces that are much smaller than the state space (see Table 2) and thus behavioral diversity may not be guaranteed. In contrast, the decomposition I(T;=) H(T) H((T==) ensures diverse behaviors through the entropy term H(T). Methods that utilize this decomposition include 27 20).
Why it is important to utilize high--dieensooal skills: Once policy is capable of generating diverse behaviors, it is important that the discriminator can distill these behaviors into distinct skills. If the set of behaviors outnumbers the set of skills, this will result in degenerate skills when one skill maps to multiple different behaviors. It is therefore important that the discriminator can accommodate continuous skills of sufficiently high dimension. Empirically, the discriminators used prior work utilize only low-dimensional continuous skill vectors. DIAYN [17] utilized 16 dimensional skills, DADS 20 utilizes continuous skills of dimension 2 - 5, while APS (27, an algorithm that utilizes successor features [30] 31 for the discriminator, is only capable of learning continuous skills with
3

---

OpenAl Gym Hopper
DeepMind Control Hopper
0.30
9.25
0.
9
when agent lases balance
30
Erwiermmntt
166
at 1000 steps
Enwernment Staps
106
DIAYN
Fixed
ICM
Figure 3: To empirically demonstrate issucs inherent to competence-basee exploration methods, we run DIAYN (17 and compare to ICM 122 and a Fixed baseline where the agent receives intrinsic reward of 1.0 for each timestep and extrinsic reward both OpenAl Gym (episode resets when agent loses balance) and DeepMind Control (DMC) (episode is fixed for steps) Hopper environments. Since Gym and DMC rewards are on different scales, we normalize rewards based on the maximum reward achieved by any algorithm 1k for Gym, 3 for DMC). While DIAYN is able to achieve higher extrinsic rewards than ICM on Gym, the Fixed intrinsic reward baseline performs best. However, DMC the Fixed and DIAYN agents achieve near-zero reward while ICM does not. This is consistent with findings prior work that DIAYN is able learn diverse behaviors Gym [17 as well as the observation that DIAYN performs poorly on DMC environments [1]
dimension 10. We show how small skill spaces lead to ineffective exploration in simple gridworld setting in Appendix G and evidence that skill dimension affects performance in Fig. 5
On the importance of benchmarks for evaluation: While prior competence-based approaches such as DIAYN [17] were evaluated on OpenAI Gym [32), Gym environment episodes terminate when the agent loses balance thereby leaking some aspects of extrinsic signal to the exploration agent. On the other hand, DMC episodes have fixed length. We show in Fig 3 that this small difference in environments results in large performance differences. Specifically, learn diverse skills in Gym but not in DMC, which consistent with both observations from DIAYN and URLB papers. Due to fixed episode lengths, DMC tasks harder for reward-free exploration since agents must learn to balance without supervision.
find that DIAYN is able to
3 Method
3.1
Contrastive Intrinsic Control
From Section 2 we are motivated to find lower bound for I(T: with discriminator that is capable of supporting high-dimensional continuous skillg" Additionally, we wish to increase the diversity of behaviors so that the discrininator can continue learning new skills throughout training. We choose the forward decomposition of MI =[T;))= H(T) - H(T[2) similar to (27) and estimate the lower bound with Eq. 2 The entropy is estimated H(() with particle-based estimator similar [15). As
such, the primary technical contribution of this work is novel estimator for the discriminator.
To improve the discriminator, propose utilize noise contrastive estimation (NCE) 23 between state-transitions and latent skills as a lower bound for [(T; has been shown previously that such estimators provide valid lower bound for mutual information [33]. However, to the best of our knowledge, this is the first work to investigate contrastive representation learning for intrinsic control.
Representation Learning: Specifically, we propose to learn embeddings by parameterizing the discriminator with a contrastive density estimator. This is novel choice that differs from prior works which utilize classifier [7 or non-contrastive density estimation 20
N N
1
log ((([T))== ([(T)))==(TT2))) log
(3)
where f(T, 2) is any real valued function.
In digh-dimensional state-action spaces the number of distinct behaviors be quite large.

---

Median
Optimality
CIC
ProtoRL
APT
RND
Disagreement
ICM
APS
DIAYN
SMM
0.2
0.8
0.30
0.60
0.75
0.30
0.45
0.60
Expert Normallized Score
SMM
APS
Disagreement
ProtoRL
Figure 4: We report the aggregate statistics using stratified bootstrap intervals 35 for 12 downstream tasks
URLB with 10 seeds, each statistic for each algorithm has 120 seeds in total. We find that overall, CIC achieves leading performance on URLB terms
recommended by we use the IQM as our primary performance measure. In terms IQM, CIC improves upon the next best skill discovery algorithm (APS) by 79% and the next best algorithm overall (ProtoRL) by 18%.
IQM, mean, and OG statistics.
parameterize this function Goh. (T) Dv (z)/ (T) (=)][T where (s, is transition tuple, Iek are neural encoders, and is a temperature parameter. This inner product is similar to the one used in SimCLR (34).
For
practical algorithm,
f(r,2)
The representation learning loss backpropagates gradients from the NCE loss which maximizes similarity between state-transitions and corresponding skills.
Ion (2s) 1l,,, (T: Irbe [T
Doh (T:)
Jit (Tx) Io2 10220 (zi)
FaI((T)
log
N
exp
[1gen (T₃ Dot (2:)l
where - 1 is the number of negatives. The total number of elements the summation is N because it includes one positive, so the index includes the positive index similar to the objective in 33 We provide pseudocode for the CIC representation learning loss:
PyTorch like pseudocode for the CIC loss
def cic_loss (s, s_next, 2, temp):
states: 5, s_next (B, D), skills: (B, D)
10
concat (s, s_next, dim=1)
11
12
query query (z)
key = key_net(tau)
15
query normalize (query,,, (query, dim=1)
16
key normalize (key, dim=1)
17
logits matmul (query, key T) / temp #logits (B, B) labels arange (logits.. shape [0]) positives on diagonal
19
20
21
coftmax_cross_enttee API same as in PyTorch docs loss = softmax cross httpt/aaxccoosseetrrttoe (logits, labels)
24
return loss
Listing 1: Pseudocode for the CIC loss.
Although we have representation learning objective, we still need to specify intrinsic reward for the algorithm for which there can be multiple choices. Prior works consider specifying an intrinsic reward that is proportional to state-transition entropy (15), the discrimina- tor [17], similarity score between states and skills (36,, and the uncertainty of the discriminator [37]
Intrinsic reward:

---

(a) Skill projection
(b)
Skill dimension
(c) Skill adaptation
(d) Skill grid sweep
0.4
0.2
Nonmalted scone
0.2
0.3
6.
Quall: Sland
Quad, Run
Normalzzd
Mermaltee
0.2
0.1
0.0
Projected
Not projected
0.0
128
0.0
zelo
Rand
CEM
Bels
CBM
Gabuss Snd
Sweep
0.0
"0"67""""
Mean skill valus
Figure 5: Design choices for pre-training and adapting with skills have significant impact on performance. (a) and (b) the agent's zero-shot performance evaluated while sampling skills randomly while (c) and (d) the agent's performance is evaluated after finetuning the skills vector. (a) we show empirically that the projecting skill vectors after sampling them from noise significantly improves the agent's performance.
(b) The skill dimension crucial hyperparameter and, unlike prior methods, CIC scales to large skill vectors achieving optimal performance at 64 dimensional skills. (c) We test several adapation strategies and find that a simple grid search performs best given the small 4k step adaptation budget, (d) Choosing the right skill vector has substantial impact on performance and grid sweeping allows the agent to select the appropriate skill.
We investigate each of these choices in Section and find that intrinsic reward that maximizes state-transition entropy coupled with representation learning via the CPC loss defined in Sec. 3. is the simplest variant that also performs well (see Table []. which use for all other experiments.
For the intrinsic reward, we use a particle estimate 122 38 as in [15] of the state-transition entropy. Similar to 15 [6 we estimate the entropy up to proportionality constant, because want the agent to maximize entropy rather than estimate its exact value.
The APT particle entropy estimate is proportional to the distance between the current visited state transition and previously seen neighboring points.
N2 Σ AgNN
1
H Hparticle (T)
log hh
h
(5)
Nk
where h; an embedding r; shown in Fig. 2
hū a kNN embedding, N₂ is the number of kNNs.
Explore and Exploit: With these design choices the two components of the CIC algorithm can be interpreted as exploration with intrinsic rewards and exploitation using representation learning to distill behaviors into skills. The marginal entropy maximizes the diversity of state-transition embeddings while the contrastive discriminator log ((T[2) encourages exploitation by ensuring that skills lead to predictable states Together the two terms incentivize the discovery of diverse yet predictable behaviors from the RL agent. While CIC shares similar intrinsic reward structure to APT [15], we show that the new representation learning loss from the CIC estimator results in substantial performance gains in Sec
Practical Implementation
Our practical implementation consists of two main components: the RL optimization algorithm and CIC architecture. For fairness and clarity comparison, we use the same RL optimization algorithm for our method and all baselines in this work. Since the baselines implemented in URLB (2 use DDPC (29) as their backbone, we opt for the same DDPG architecture to optimize method well (see Appendix B).
CIC Architectre:: We use particle estimator as in 15] to estimate H(T). To compute the variational density g(r|2), we first sample skills from uniform noise p(z) where p(2) is the uniform distribution over the [0, 1| interval. We then use two MLP encoders to embed Joh (7) and gos(z). (z).
'I was recently was shown that a DDPG achieves state of of- the-art performance 39 on DeepMind Control 40 and is more stable than SAC 41 on this benchmark.

---

and optimize the parameters 111 8h2 with the CPC loss similar to SimCLR [4] since f(T,2)
9o. (r)" Gei (z). We fix the hyperparameters across all domains and downstream tasks. We refer the reader to the Appendices D and for the full algorithm and a full list of hyperparameters.
Adapting downstream tasks: To adapt downstream tasks we follow the same procedure for competence-basee method adaptation as in URLB [21]. During the first 4k environment interactions we populate the DDPG replay buffer with samples and use the extrinsic rewards collected during this period to finetune the skill vector z. While it's common to finetune skills with Cross Entropy Adaptation (CMA), given our limited budget of 4k samples (only 4 episodes) we find that a simple grid sweep of skills over the interval [0. 1] produces the best results (see Fig. [ After this, we fix the skill 2 and finetune the DDPG actor-critic parameters against the extrinsic reward for the remaining 96k steps. Note that competence-base based methods in URLB also finetune their skills during the first 4k finetuning steps ensuring fair comparison between the methods. The full adaptation procedure detailed in Appendix D
5 Experimental Setup
Environments We evaluate our approach on tasks from URLB, which consists of twelve downstream tasks across three challenging continuous control domains for exploration algorithms walker, quadruped, and Jaco arm. Walker requires biped constrained to a 2D vertical plane to perform locomotion tasks while balancing. Quadruped is more challenging due to a nigher-dimensione state-action space and requires quadruped to in 3D environment to learn locomotion skills. Jaco arm is 6-DOF robotic arm with a three- three-finger gripper to move and manipulate objects without locking. All three environments are challenging in the absence of an extrinsic reward.
Baselines: We implemented CIC using the URLB 21 codebase and compare CIC to baselines included in URLB across all three exploration categories. Knowledge-basee basedlines include Disagreement [13]. and RND [14. Data-based baselines incude APT [15] and Pro-
ICM 12
toRL 16 Competence-basee baselines include DIAYN 177. SMM [26). and APS 27 The closest baselines to CIC are APT, which is similar to CIC but without state-skill CPC representation learning (no discriminator), and APS which uses the same decomposition of mutual information CIC and also uses a particle entropy estimate for H((T). The main difference between APS and CIC is that APS uses successor features while CIC uses contrastive estimator for the discriminator. For further details regarding baselines we refer the reader to Appendix C
Evaluation: We follow an identical evaluation to the 2M pre-training setup in URLB. First, we pre-train each RL agent with the intrinsic rewards for 2M steps. Then, we finetune each agent to the downstream task with extrinsic rewards for 100k steps. All baselines were run for 10 seeds per downstream task for each algorithm using the code and hyperparameters provided by URLB [21]. Built on top of URLB, CIC is also run for 10 seeds per task. A total of 1080 9 algorithms x 12 tasks x 10 seeds experiments were run for the main results. Importantly, all baselines and CIC use DDPG agent their backbone.
To ensure that our evaluation statistics are unbiased we use stratified bootstrap confidence intervals to report aggregate statistics across M runs with N seeds as described in Rliable [35 to report statisties for our main results in Fig. 4 Our primary success metric is the interquartile mean (IQM) and the Optimality Gap (OG). IQM discards the top and bottom 25% of runs and then computes the mean. less susceptible to outliers than the mean and was shown to be the most reliable statistic for reporting results for RL experiments in (35). OG measures how far policy is from optimal (expert) performance. To define expert performance we use the convention URLB, which is the score achieved by a randomly initialized DDPG after 2M steps of finetuning (20x more steps than our finetuning budget).
6 Results
We investigate empirical answers to the following research questions: (Q1) How does CIC adaptation efficiency compare to prior competence algorithms and exploration algorithms more broadly?
*URLB is open- under an MIT license http::////gtttb..ooo///llleeeeee bea.rrrreerrrcrcrrrrbaaaaaaaaa/ benchmark/ blob/main/LICENSE
7

---

(Q2) Which intrinsic reward instantiation of CIC performs best? (Q3) How do the two terms in the CIC objective affect algorithm performance? (Q4) How does skill selection affect the quality of the pre-trained policy? (Q5) Which architecture details matter most?
Adaptation efficiency of CIC and exploration baslines: Expert normalized scores of CIC and exploration algorithms from URLB are shown in Fig. 4 We find that CIC substantially outperforms prior competence- algorithms (DIAYN, SMM, APS) achieving 79% higher IQM than the next best competence-based method (APS) and, more broadly, achieving a 18% higher IQM than the next best overall baseline (ProtoRL). In further ablations, we find that the contributing factors to CIC's performance are its ability to accommodate substantially larger continuous skill spaces than prior competence- basee methods.
Intrinsic reward specification: The intrinsic reward for competence-based algorithms can be instantiated in many different ways. Here, we analyze intrinsic reward for CIC with the form Pint = (T) + (TT, 2), where D is some function of (T, Prior works, select D to be (i) the discriminator (27), (ii) cosine similarity between embeddings [36), (iii) uncertainty of the discriminator [37]. and (iv) just the entropy (TT, = 0 [15]. We run CIC with each of these variants on the walker and quadruped tasks and measure the final mean performance across the downstream tasks (see Tab. D]. The results show that the entropy-only intrinsic reward performs best followed by an uncertainty-
intrinsic reward. We hypothesize that the reason why a simple entropy-only intrinsic reward works well is that state-skill CPC representation learning clusters similar behaviors together. Since similar behaviors are clustered, maximizing the entropy of state-transitee transstnnition embeddings produces increasingly diverse behaviors.
disc.
similarity
uncertainty
entropy
APS
walker
0.79 + 0.04 0.45 + 0.07
0.79 + 0.03
0.76 + 0.04
0.82 + 0.02
0.50 + .04
quad.
0.60 = 0.05
0.70 = 0.03
0.76 + 0.03
.48 = 0.02
mean
0.62
0.70
0.73
0.80
0.49
Table 1: Comparing different potential intrinsic rewards for CIC, we find that entropy- based intrinsic reward performs best, suggesting that the CIC discriminator is primarily useful for representation learning. These are normalized scores averaged over 5 seeds across 8 downstream tasks. Note that all intrinsic reward specifications outperform the baseline methods. Since the particle entropy estimates a quantity proportional to the entropy, two-term intrinsic rewards need to be carefully balanced with a hyperparameter. We believe this is the reason the various intrinsic rewards perform worse than entropy-only one.
The importance of representation learning: To what extent does representation learning with the state-skill CIC loss affect the agent's is exploration capability? To answer this question train the CIC agent with the entropy intrinsic reward with and without the representation learning auxiliary loss for 2M steps. The zero-shot reward plotted in Fig. iindiiatsss that without representation learning the policy collapses. With representation learning, the agent is able to discover diverse skills evidenced by the non-zero reward. This result suggests that state-skill CPC representation learning is a critical part of CIC.
Qualitative analysis of CIC behaviors: Qualitatively, we find that CIC is able learn locomotion behaviors in DMC without extrinsic information such as early termination as in OpenAl Gym. While most skills are higher entropy and thus more chaotic, we show in Fig that structured behaviors can be isolated by fixing a particular skill vector. For example, in the walker and quadruped domains balancing, walking, and flipping skills can be isolated. For more qualitative investigations we refer the reader to Appendix H
Skill architecture and adaptation ablations: We find that projecting the skill to a latent space before inputting it as the key for the contrastive loss is an important design decision (see Fig. 5p)), most likely because this reduces the diversity of the skill vector making the discriminator task simpler.
We also find empirically that the skill dimension is an important hyperparameter and that larger skills results in better zero-shot performance (see Fig. 5D), which empirically supports the hypothesis posed in Section 2 and Appendix that larger skill spaces are important for internalizing diverse hehaviors. Interestingly, CIC zero-shot performance is poor in lower skill dimensions (c.g. dim(z) < 10).
8

---

Zero-shel estrinsic
OC Tartide antepy and oc learning
Altatton farticie entrayy tue ac ro bouring Altattion 2. ac mp sen but sarvie entmpy
100
0.00
0.25
0.75 1.50 1.75 2.00
Figure 6: Mean zero-shot extrinsic rewards for Quadruped stand over seeds with and without state-skill representation learning. Without representation learning, the algorithm collapses. Similarly, with CIC representation learning but no entropy term (in which case we use the discriminator as the intrinsic reward) the policy also collapses. Note that there is no finetuning happening here. We'r re showing the task-specific extrinsic reward during reward- free pre-training as a way to sense-check exploration policy.
suggesting that when dim(z) is small CIC performs no better than prior competence prased methods such DIAYN, and that scaling to larger skills enables CIC to pre-train effectively.
To measure the effect of skill finetuning described in Section 4 we sweep mean skill values along the interval of the uniform prior [0, with a budget of 4k total environment interactions and read out the performance on the downstream task. By sweeping, we mean simply iterating over the interval [0, 1] with fixed step size (e.g. v 0, 0.1......... 0.9, 1) and setting 2; v for all i. This is not an optimal skill sampling strategy but works well due to the extremely limited number of samples for skill selection. We evaluate this ablation on the Quadruped Stand and Run downstream tasks. The results shown in Fig. 5 indicate that skill selection can substantially affect zero-shot downstream task performance.
7 Related Work
The most closely related prior algorithms to CIC are APT [T5] and APS 27 Both CIC and APS use the H(T) H(([=) decomposition of the mutual information and both used particle estimator 22 to compute the state entropy as in [15. The main difference between CIC and APS is the discriminator. APS uses successor features in [31] for its discriminator while CIC uses a noise contrastive estimator. Unlike successor features, which empirically only accommodate low-dimensional continuous skill spaces (see Table 2 the noise contrastive discriminator is able to leverage higher continuous dimensional skill vectors. Like APT, CIC has an intrinsic reward that maximizes H(T). However, CIC also does contrastive skill learning to shape the embedding space and outputs a ciil--ondiiioned policy.
The CIC discriminator is similar to the one used in DISCERN (36]. goal-conditioned unsupervised RL algorithm. Both methods use a contrastive discriminator by sampling negatives and computing an inner product between queries and keys. The main differences are (i) that DISCERN maximizes
((r:g) where are image goal embeddings while CIC maximizes [(T:2) 2) where 2 are abstract skill vectors; (ii) DISCERN uses the DIAYN-style decomposition 1 [(T;g)
H (g) H (glT) while
CIC decomposes through (T) - H (T]2), and (iii) DISCERN discards the H (g) term by sampling goals uniformly while CIC explicitly maximizes H(T). While DISCERN and CIC share similarities,
DISCERN operates over image goals while CIC operates over abstract skill vectors so the two methods are not directly comparable.
Finally, another similar algorithm to CIC is DADS 20 which also decomposes through H(T) (T]zz). While CIC uses contrastive density estimate for the discriminator, DADS uses a maximum likelihood estimator similar DIAYN. DADS maximizes [ss' 8, 2) and estimates entropy H(''|s) by marginalizing such that H(s Hs''ss)
log E. g(s (/|ss, [s, 24) while CIC uses particle estimator.

---

Table 2: Competence- saasetence-based Unsupervised Skill Discovery Algorithms
Algorithm
Intrinsic Reward
Decomposition
Explicit max H(7)
Skill Dim.
Skill Space
SSN4HRL
log ( sx)
[[()) [(a
No
6
discrete one-hot
VIC TB
log (u sgr))
[[(2) []
No
60
discrete one-hot
VALOR
log ( #1:H4
H(2) ((arr)
discrete one-hot
DIAYN
logge(( ex)
H(2) H(2
128
discrete one-hot continuous continuous
DADS
20
Σ, log 8).
H(T) H(7==)
Yes
VISR
31
log (u et)
H(2) ((3
No
10
APS
Plocessoor ( 2) + (pu)ice(8)
H(()) 71(T 2)
continuous
CIC
Fa Mporncte (8,
M(7) 7((T
continuous
Table 3: A list of competence based algorithms. We describe the intrinsic reward optimized by each method and decomposition
mutual information utilized by the method. also note whether the method explicitly maximizes state transition entropy. Finally, we note maximal dimension used each work and whether the skills are discrete or continuous. All methods prior to CIC only support small skill spaces, either because they discrete continuous but low-dimensional.
Limitations and Impact
While CIC achieves leading results on state- baste URLB, we would also like to address its limitations. First, in this paper we only consider MDPs (and not partially observed MDPs) where the full state is observable. We focus on MDPs because generating diverse behaviors in environments with large state spaces has been the primary bottleneck for competence based exploration. Combining CIC with visual representation learning to scale this method to pixel-based inputs is a promising future direction for research not considered in this work.
One issue with unsupervised RL algorithms (and hence CIC) in terms of potentially negative societal impact is that self-supervised exploration be dangerous. Since self-supervised agents maximize intrinsic rewards, this can lead to destructive behavior. For example, when deploying CIC on a Walker or Quadruped robot it learns chaotic exploration behaviors that would most likely break the robot in real-words settings. Alignment of exploration agents to prevent them from learning dangerous policies a promising direction for future work.
Conclusion
We have introduced new competence-basee algorithm Contrastive Intrinsic Control (CIC) which enables more effective exploration than prior unsupervised skill discovery algorithms by explicitly encouraging diverse behavior while distilling predictable behaviors into skills with contrastive discriminator. We showed that CIC is the first competence based approach to achieve leading performance URLB. We hope that this encourages further research in developing RL agents capable generalization.
10
Acknowledgements
We would like thank Ademi Adeniji, Xinyang Geng, Fangchen Liu for helpful discussions. We would also like thank Phil Bachman for useful feedback. This work was partially supported by Berkeley DeepDrive, NSF A14OPT AI Institute for Advances in Optimization under NSF 21 12533, and the Office of Naval Research grant N00014-21-1-2769.
References
[1] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Seee httpss:/ssitess http::/s/stttegooooo...////////s
10

---

[2] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
[3] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Artfhur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, (62(6419):1140-111 2018.
[4] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
[5] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[6] Xue Bin Peng, P. Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37:143:1-143:14, 2018.
[7] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Roboties: Science and Systems (RSS), 2018.
[8] OpenAl. Learning dexterous in-hand manipulation. CoRR, abs// 00177770000077700000000000 2018.
[9] OpenAl. Solving rubik's cube with robot hand. ArXiv, abs/1910.0711111111 2019.
[10] Aravind Srinivas and Pieter Abbeel. Unsupervised learning for reinforcement learning, 2021.
[11] Pierre-Yve Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265- 286, 2007.
[12] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
[13] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International Conference on Machine Learning. 2019.
[14] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019.
[15] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04555 2021.
[16] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference Machine Learning. 2021.
[17] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
[18] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In International Conference on Learning Representations, 2017.
[19] Taehwan Kwon. Variational intrinsic control revisited. In International Conference on Learning Representations, 2021.
[20] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- tions, 2020.
[21] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbecl. Urlb: Unsupervised reinforcement learning benchmark, 2021.

---

[22] Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 23(3-4):301-322 -321, 2003.
[23] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirleenth International Conference Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[24] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
[25] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10299, 2018.
[26] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. CoRR,
2019.
[27] Hao Liu and Pieter Abbeel. APS: active pretraining with successor features. In International Conference Muchine Learning, 2021.
[28] David Barber and Felix V. Agakov. The im algorithm: A variational approach to information maximization. In Advances in neural information processing systems, 2003.
(29) Timothy P. Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[30] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv: 06ti:/666.0000 153 2016.
[31] Steven Hansen, Will Dabney, André Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- tional Conference on Learning Representations, 2020.
[32] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606001500 2016.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.0333 2018
[34] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, 2020.
[35] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Belle- mare. Deep reinforcement learning at the edge of the statistical precipice, 2021.
[36] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- -onpparaettric discriminative rewards, 2018
[37] DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. CoRR, abs/2 107. 14226. 2021.
[38] J Beirlant. Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, 6: 7-19, 1997.
[39] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. 2021.
[40] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 2018.
12

---

[41] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning. 2018.
[42] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In International Conference on Learning Representations, 2018.
[43] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
[44] Tom Zahavy, Andre Barreto, Daniel J Mankowitz, Shaobo Hou, Brendan O' Kemaev, and Satinder Baveja Singh. Discovering a of policies for the worst case reward, 2021.
O'Donoghue, Iurii
[45] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised rep- resentations for reinforcement learning. In International Conference on Machine Learning. 2020.
[46] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5171-5180. PMLR, 2019.
[47] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Abstract, summary of contributions introduction, and results in Section 6
(b) Did you describe the limitations of your work? [Yes] See Section g
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section g
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either the supplemental material or as a URL)? [Yes] 4open. 4open. sci www.sscciiiiccccceeeeeeeeeeeeece ott.:///emee/ccccoooooor////////
md (b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
https://anonymonn
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix [
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5
(b) Did you mention the license of the assets? [Yes] See Section 5
(c) Did you include any assets either in the supplemental material URL? [No]
We use previous assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "l7YnBLD0e6x", "reviewer": "Reviewer_yva3", "summary": "This paper presents a contrastive-learning based method that encourages behavioural diversity for unsupervised RL (where unsupervised means without reward). Their method operates by maximizing the mutual information between state-transitions and skill vectors. The authors provide experimental evidence with the Unsupervised Reinforcement Learning Benchmark (URLB) for the efficiency of their proposed method by examining whether pre-training agents with their method can better adapt to downstream tasks.", "questions": "1. In equation (2), what is the expectation over?\n1. Is the second to last line of the pseudocode on page 5 correct? It's not clear that the `cross_entropy` call gives you $F_{CIC}$. In particular, it seems like it's missing the first term?\n1.  In line 277 it says \"most likely because this reduces the diversity of the skill vector\" to justify the projection, but it's not clear why this projection is necessarily reducing the diversity of the skill vector?\n1. In the \"skill architecture and adaptation ablations\", did you try running DIAYN with larger skills?\n", "limitations": "Both limitations and potential negative societal impacts were discussed adequately (in my opinion) in section 7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\nThis work is mostly an extension of the work of Liu & Abbeel 2021 ([15] and [27] in paper) where the main contribution (as acknowledged by the authors) is in a novel _discriminator_ (where the discrimination is between latent skills). Their proposed improvement is also based on existing work (e.g. the noise-contrastive estimator of Gutmann & Hyvärinen 2010 ([23] in paper), the parameterization of the function for density estimation of Chen et al. 2020 ([34] in paper)).\n\nHowever, in my opinion this is an original and novel combination of existing methods towards addressing a problem for which (most of) those methods were not originally designed.\n\n# Quality\nOverall I feel the quality of the paper is reasonably good. There are a few issues with clarity that I mention below. I appreciated the ablations and sensitivity analyses done in Figure 5, as it really helped motivate the design choices made in the paper. Finally, there are a few questions related to experiment quality in the Questions section belo.\n\n# Clarity\nThis is perhaps the weakest point of the paper (although Figure 2 is really nice!). The writing oscillates between proper mathematical notation and a github README file (e.g. \"$r,s'\\sim$ `env.step(a)`\" in line 57). Further, there are some improperly specified mathematical objects in the writing, which I list below:\n*  In line 53, you say \"$r$\" is \"sampled\" from `env.step(a)`, but a few lines above you stated that $r$ was the reward function.\n*  In line 55 it says \"$\\tau(s)$ ... refers to any function of the states $s$\". This is ill-defined and should be more specific. $\\tau$ is a function of what to what? In line 57 you're using $\\tau$ as a tuple (e.g. $\\tau=(s, s')$), so it's not at all clear what it's meant to represent.\n*  In line 60,  $\\mathcal{Z}$ has not been introduced.\n* The paragraph in lines 86-95 is not at all clear because it's not clear what $\\tau$ is.\n* In footnote 4 in page 4 it says \"Note that $\\tau$ is not a trajectory but some function of states\" which, again, is not at all clear what it's meant to represent.\n* It's not clear what is meant in lines 173-174. In particular what are \"negatives\" and \"positives\"? Is $N$ the same thing as $N_k$ in the equation?\n*  One method of finetuning is presented in lines 195-204, but another one is presented in lines 223-224. It's not clear which one was actually used for the experiments, and if both were used, when each was used.\n* The histogram in Figure 5(d) (and accompanying discussion) is not clear. You're evaluating over a pre-selected grid of latents, and the histogram shows the performance of each of these grids? It's not clear how this is measuring the effect of skill fine-tuning?\n\n# Significance\nThis work is addressing an important problem for a large portion of RL research (lifelong learning, zero-shot transfer, generalizability, etc.), and as such can have an important impact on the community. The empirical evaluations presented in Figure 5 are quite useful, as they help shed light on the most necessary components of the proposed algorithm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ikCqnTOx0jP", "reviewer": "Reviewer_4Ffu", "summary": "The paper considers the problem of exploration and skill discovery in unsupervised reinforcement learning. To this end, the paper proposes CIC, an algorithm to explore novel states as well as distill the experience into reusable skills, which can be efficiently adapted to downstream tasks. This work builds upon the ideas of several previous works that maximize the mutual information between (some function of) the visited states and the latent skill vectors. These works differ in the different design choices for estimating the mutual information (generally it is a lower bound), as succinctly summarized in the paper in lines 74-76. \n\nCIC estimates the entropy term, $\\mathcal{H}(\\tau)$, using a particle estimate similar to some previous work. The contribution of the paper is the use of a contrastive density estimator, specifically the NCE loss proposed in CPC (Oord et al, 2018), for the variational term in the MI lower bound. The unsupervised pre-training phase involves optimizing the CPC loss for contrastive representation learning between state transitions and skills as well as maximizing the intrinsic reward. The fine-tuning phase involves training the agent (with a few steps) using only the external task reward. \n\nThis approach is shown to outperform similar skill discovery methods on the URLB benchmark using DDPG as the backbone, and extensive experiments investigating some design and hyperparameter choices have been performed.", "questions": "**The use of CPC loss as variational distribution:** The noise contrastive density estimator used for $q(\\tau|z)$ and the representation learning loss is exactly the same, which is the CPC loss from (Oord et al., 2018). The CPC loss is a lower bound on the MI, so it is unclear why this should be a good choice for the variational distribution. Moreover, replacing Eq. (3) in Eq. (2) contradicts the result of (Oord et al., 2018). Since CIC frames itself within the MI maximization family of skill discovery algorithms, it is crucial that the connection between the proposed method and MI maximization is presented as clearly as possible.\n\n**$F_{CIC}$ definition vs. implementation:** The notation in the definition of $F_{CIC}$ is not clear, and there seem to be inconsistencies between this definition and the pseudocode provided for calculating this loss. The index $i$ is not specified, but assuming it is the batch index, it would seem that $F_{CIC}$ is calculated for some $z_i$ with $\\tau_i$ as the positive sample and the remaining $\\tau_j$ in the batch as negative samples. The pseudocode, however, samples a batch of $\\tau_i$ and a batch of $z_i$, similar to how most contrastive learning methods (simCLR, CURL etc.) implement this loss.\n\n**Intrinsic reward specified not specified clearly:** The intrinsic reward is specified in the middle of the experimental results section as a written description, rather than explicit mathematical definitions.  Also, it is not clear which intrinsic reward is used when the authors compare CIC with the baselines or perform ablations (such as in Fig 4, 5, 6). It is appreciable that the paper considers a variety of intrinsic rewards to understand which one works best, however, I believe that specifying them clearly would help improve the readability of the paper.\n\n**CIC used in experiments does not use intrinsic reward to maximize MI:** The paper presents CIC as part of a family of methods that maximize the MI using intrinsic reward (Table 2), however their best performing model (and presumably most of their empirical results) considers only the entropy term as the intrinsic reward. This disconnect between the theory (which describes a novel method of estimating the variational lower bound on MI), and the experiments (which seem to not use this variational lower bound) significantly hurts the quality of the paper. If my understanding is correct, the intrinsic reward closest to the MI maximization objective would be (i) discriminator (from line 253-254), which is also the worst-performing variant of CIC. In addition, the contribution of using a contrastive density estimator in the variational lower bound on MI is made redundant.\n\n**Minor comments:**\n\n- Typo in Line 67, should direct to Table 2 in Appendix A.\n- CPC loss, referred in line 165 and numerous other places, is not defined. I assume it refers to $F_{CIC}$, which is same as the CPC loss.\n- $N$ is not defined for Eq. (3), assumed to be the batch size.\n- The experiment plots, Fig 4, 5, 6, are present on different pages and going back and forth between Section 6 and the plots hurts the readability of the paper. I suggest moving the plots closer to the relevant text.", "limitations": "The authors acknowledge some limitations of their work, particularly that the method is described for MDPs (not POMDPs), and that they do not consider visual inputs in their experiments. The authors have also given though to the potential negative societal impacts of their work (really unsupervised RL in general), which is appreciable. \n\nThe main limitations of the work as per my review are the somewhat limited novelty in the proposed method, some questions on the theoretical soundness, and the discrepancy between the method described in the theory section and the one used in the experiments. There are also some concerns regarding the notation and writing, which are relatively minor and easily fixable.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of exploring novel states and generalizing to multiple downstream tasks are central challenges in RL, and skill discovery algorithms provide an appealing solution to both these problems. There are several works which aim to maximize (some lower bound on) the MI between states and skill vectors. In this regard, the paper has somewhat limited novelty. This work proposes maximizing MI explicitly using CPC loss for representation learning (in addition to the intrinsic reward) and the contrastive density estimator used to estimate the conditional entropy term is also well known. Nevertheless, a combination of well-known techniques can still be valuable, provided it is supported with substantial theoretical and empirical analysis. The paper performs many experiments to showcase and analyze the proposed method, however, the theoretical justification is lacking and unclear in a few places.\n\nThe paper contextualizes their work well by providing a succinct and fairly complete overview of similar work (Table 1 is quite helpful). The authors provide intuition-based but clear motivation for their design choices in Section 2, which sets up the proposed method well. However, parts of Section 3 are not clear, particularly the use of CPC loss as the variational distribution, mismatch between the specification of $F_{CIC}$ and its computation in the pseudocode, and the connection to MI maximization. The intrinsic reward mentioned from Section 3 onwards is not defined until Section 6, which might be confusing for the reader. It is also not clear what intrinsic reward is used in the ‘default’ CIC algorithm (used in Fig 4, 5, 6) - I assume it's just the entropy term since it is shown to be the best performing variant in Section 6. If so, the discussion about the variational lower bound on MI and the contribution of using a contrastive density estimator are made redundant. The experimental results, notwithstanding the issues mentioned here, are extensive and the ablation studies for various design choices are insightful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GX4kv6QJPdB", "reviewer": "Reviewer_tFmb", "summary": "This work proposes CIC, a competence-based unsupervised RL method that maximizes the mutual information between latent skills and state transitions. CIC outperforms related baselines on the Unsupervised RL benchmark (URLB) where the agent only has access to the extrinsic reward for a short adaptation phase in the downstream tasks.", "questions": " 1. I'm curious how the estimation of decomposed mutual information in Table 3 would distribute across different methods. And how CIC may outperform baselines.\n\n2. The decomposition of mutual information used by CIC encourages diverse behaviors through the entropy term $H(\\tau)$. I'm curious whether there is a trade-off for losing some diversity of the skills $H(z)$?\n\n3. I see that previous works employ too small skill dimensions and it's great that CIC employs a larger skill dimension. But the skill dimension should not be too large as well, so there should be some sweet spot that may require hyperparameter tuning. Since the tasks targeted are unsupervised adaptation, does it make sense to tune the dimension without the knowledge of the diversity of the downstream tasks?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***\n\nThis study has significant strengths in the well-established experimental section.\n\n1. Decent experimental setup\n\nThe URLB is a well-established benchmark to evaluate the unsupervised adaptation performance of RL methods. I appreciate the comparison between Gym and DMControl.\n\n2. High statistical significance\n\nI appreciate the authors running experiments across as many as 10 seeds, and reporting IQM scores. This provides high statistical significance for the results compared to other methods that report only mean results with 3~5 seeds. Also they compared many relevant prior works.\n\n3. Ablations\n\nBecause there are many overlapping contributions with existing studies (APS, APT, CURL,..etc.), it is easy to give a mixed feeling of contribution. But the authors managed to prevent this through ablations in Figure 5.\n\n\n***Weaknesses***\n\nThe weaknesses of this work lie in the originality and motivation. As this is a completely experimental paper, I understand that theoretically, it is difficult to prove the strength of this study. I felt the connection between the motivation and contribution seemed a bit vague.\n\n1. Originality\n\nAs the authors are aware, there were previous works that use particle-based entropy or contrastive learning for unsupervised learning (APT, APS, CURL). The novelty of CIC is that it is the first that used contrastive learning between state transition and latent skill. The absence of a related work section may make it feel that novelty is lacking. From this point of view, I appreciate the comparison made in Appendix D. Maybe it would be better if the authors add such content to the main manuscript with a little more detail.\n\n2. Motivation\n\nThe two points made in the motivation section (\"Competence-based algorithms do not ensure diverse behaviors\", \"Why it is important to utilize high-dimensional skills\") are indeed very important and useful points. But there seems to be a lack of a link between these motivations and the contrastive learning and particle-based entropy. Some questions remain about whether contrastive learning and particle-based entropy are the \"only\" or the \"best\" ways to estimate $H(\\tau|z)$ and $H(\\tau)$, respectively. Some of these concerns seem to be addressed in Appendix L, which I think would be better to be in the main manuscript.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept