CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

1 INTRODUCTION . max I ( s ; z ) = max ( ℋ ( s ) − ℋ ( s |z ) ) z1 z2 z3 Figure 1 : This work deals with unsupervised skill discovery through mutual information maximization . We introduce Contrastive Intrinsic Control ( CIC ) – a new unsupervised RL algorithm that explores and adapts more efficiently than prior methods . Deep Reinforcement Learning ( RL ) is a powerful approach toward solving complex control tasks in the presence of extrinsic rewards . Successful applications include playing video games from pixels ( Mnih et al. , 2015 ) , mastering the game of Go ( Silver et al. , 2017 ; 2018 ) , robotic locomotion ( Schulman et al. , 2016 ; 2017 ; Peng et al. , 2018 ) and dexterous manipulation ( Rajeswaran et al. , 2018 ; OpenAI , 2018 ; 2019 ) policies . While effective , the above advances produced agents that are unable to generalize to new downstream tasks beyond the one they were trained to solve . Humans and animals on the other hand are able to acquire skills without supervision and apply them efficiently to a variety of downstream tasks . In this work , we seek to train agents that acquire skills without supervision with generalization capabilities by efficiently adapting these skills to downstream tasks . Over the last few years , unsupervised RL has emerged as a promising framework for developing RL agents that can generalize to new tasks . In the unsupervised RL setting , agents are first pre-trained with self-supervised intrinsic rewards and then finetuned to downstream tasks with extrinsic rewards . Unsupervised RL algorithms broadly fall into three categories - knowledge-based , data-based , and competence-based methods1 . Knowledge-based methods maximize the error or uncertainty of a predictive model ( Pathak et al. , 2017 ; 2019 ; Burda et al. , 2019b ) . Data-based methods maximize the entropy of the agent ’ s visitation ( Liu & Abbeel , 2021a ; Yarats et al. , 2021b ) . Competence-based methods learn skills that generate diverse behaviors ( Eysenbach et al. , 2019 ; Gregor et al. , 2017 ) . This work falls into the latter category of competence-based methods . 1These categories for exploration algorithms were introduced by Srinivas & Abbeel ( 2021 ) and inspired by Oudeyer et al . ( 2007 ) . Unlike knowledge-based and data-based algorithms , competence-based algorithms simultaneously address both the exploration challenge as well as distilling the generated experience in the form of reusable skills . This makes them particularly appealing , since the resulting skill-based policies ( or skills themselves ) can be finetuned to efficiently solve downstream tasks . While there are many self-supervised objectives that can be utilized , our work falls into a family of methods that learns skills by maximizing the mutual information between visited states and latent skill vectors . Many earlier works have investigated optimizing such objectives ( Eysenbach et al. , 2019 ; Gregor et al. , 2017 ; Kwon , 2021 ; Sharma et al. , 2020 ) . However , competence-based methods have been empirically challenging to train and have under-performed when compared to knowledge and data-based methods ( Laskin et al. , 2021 ) . In this work , we take a closer look at the challenges of pre-training agents with competence-based algorithms . We introduce Contrastive Intrinsic Control ( CIC ) – an exploration algorithm that uses a new estimator for the mutual information objective . CIC combines particle estimation for state entropy ( Singh et al. , 2003 ; Liu & Abbeel , 2021a ) and noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) for the conditional entropy which enables it to both generate diverse behaviors ( exploration ) and discriminate high-dimensional continuous skills ( exploitation ) . To the best of our knowledge , CIC is the first exploration algorithm to utilize noise contrastive estimation to discriminate between latent skill vectors . Empirically , we show that CIC adapts to downstream tasks more efficiently than prior exploration approaches on the Unsupervised Reinforcement Learning Benchmark ( URLB ) . CIC achieves 91 % higher returns on downstream tasks than prior competence-based algorithms and 26 % higher returns than the next-best exploration algorithm overall . 2 BACKGROUND AND NOTATION . Markov Decision Process : We operate under the assumption that our system is described by a Markov Decision Process ( MDP ) ( Sutton & Barto , 2018 ) . An MDP consiss of the tuple ( S , A , P , r , γ ) which has states s ∈ S , actions a ∈ A , transition dynamics p ( s′|s , a ) ∼ P , a reward function r , and a discount factor γ . In an MDP , at each timestep t , an agent observes the current state s , selects an action from a policy a ∼ π ( ·|s ) , and then observes the reward and next state once it acts in the environment : r , s′ ∼ env.step ( a ) . Note that usually r refers to an extrinsic reward . However , in this work we will first be pre-training an agent with intrinsic rewards rint and finetuning on extrinsic rewards rext . For convenience we also introduce the variable τ ( s ) which refers to any function of the states s. For instance τ can be a single state , a pair of states , or a sequence depending on the algorithm . Our method uses τ = ( s , s′ ) to encourage diverse state transitions while other methods have different specifications for τ . Importantly , τ does not denote a state-action trajectory , but is rather shorthand for any function of the states encountered by the agent . In addition to the standard MDP notation , we will also be learning skills z ∈ Z and our policy will be skill-conditioned a ∼ π ( ·|s , z ) . Unsupervised Skill Discovery through Mutual Information Maximization : Most competencebased approaches to exploration maximize the mutual information between states and skills . Our work and a large body of prior research ( Eysenbach et al. , 2019 ; Sharma et al. , 2020 ; Gregor et al. , 2017 ; Achiam et al. , 2018 ; Lee et al. , 2019 ; Liu & Abbeel , 2021b ) aims to maximize a mutual information objective with the following general form : I ( τ ; z ) = H ( z ) −H ( z|τ ) = H ( τ ) −H ( τ |z ) ( 1 ) Competence-based algorithms use different choices for τ and can condition on additional information such as actions or starting states . For a full summary of competence-based algorithms and their objectives see Table 1 in Appendix D. Lower Bound Estimates of Mutual Information : The mutual information I ( s ; z ) is intractable to compute directly . Since we wish to maximize I ( s ; z ) , we can approximate this objective by instead maximizing a lower bound estimate . Most known mutual information maximization algorithms use the variational lower bound introduced in Barber & Agakov ( 2003 ) : I ( τ ; z ) = H ( z ) −H ( z|τ ) ≥ H ( z ) + E [ log q ( z|τ ) ] ( 2 ) Note that the variational lower bound can be applied to both decompositions of the mutual information . The design decisions of a competence-based algorithm therefore come down to ( i ) which decomposition of I ( τ ; z ) to use , ( ii ) whether to use discrete or continuous skills , ( iii ) how to estimate H ( z ) or H ( τ ) , and finally ( iv ) how to estimate H ( z|τ ) or H ( τ |z ) . 3 MOTIVATION . The results from the recent Unsupervised Reinforcement Learning Benchmark ( URLB ) introduced in Laskin et al . ( 2021 ) , suggest that pre-training with competence-based approaches underperforms relative to knowledge-based and data-based baselines on DeepMind Control ( DMC ) . We argue that the underlying issue with current competence-based algorithms when deployed on harder exploration environments like DMC has to do with the currently used estimators for I ( τ ; z ) rather than the objective itself . To produce structured skills that lead to diverse behaviors , I ( τ ; z ) estimators must ( i ) explicitly encourage diverse behaviors and ( ii ) have the capacity to discriminate between high-dimensional continuous skills . Current approaches do not satisfy both criteria . Competence-base algorithms do not ensure diverse behaviors : Most of the best known competencebased approaches ( Eysenbach et al. , 2019 ; Gregor et al. , 2017 ; Achiam et al. , 2018 ; Lee et al. , 2019 ) , optimize the first decomposition of the mutual information H ( z ) − H ( z|τ ) . The issue with this decomposition is that while it ensures diversity of skill vectors it does not ensure diverse behavior from the policy , meaning maxH ( z ) does not imply maxH ( τ ) . Of course , if H ( z ) − H ( z|τ ) is maximized and the skill dimension is sufficiently large , thenH ( τ ) will also be maximized implicitly . Yet in practice , to learn an accurate discriminator q ( z|τ ) , the above methods assume skill spaces that are much smaller than the state space ( see Table 1 ) , and thus behavioral diversity may not be guaranteed . In contrast , the decomposition I ( τ ; z ) = H ( τ ) − H ( τ |z ) ensures diverse behaviors through the entropy term H ( τ ) . Methods that utilize this decomposition include Liu & Abbeel ( 2021b ) ; Sharma et al . ( 2020 ) . Why it is important to utilize high-dimensional skills : Once a policy is capable of generating diverse behaviors , it is important that the discriminator can distill these behaviors into distinct skills . If the set of behaviors outnumbers the set of skills , this will result in degenerate skills – when one skill maps to multiple different behaviors . It is therefore important that the discriminator can accommodate continuous skills of sufficiently high dimension . Empirically , the discriminators used in prior work utilize only low-dimensional continuous skill vectors . DIAYN ( Eysenbach et al. , 2019 ) utilized 16 dimensional skills , DADS ( Sharma et al. , 2020 ) utilizes continuous skills of dimension 2 − 5 , while APS ( Liu & Abbeel , 2021b ) , an algorithm that utilizes successor features ( Barreto et al. , 2016 ; Hansen et al. , 2020 ) for the discriminator , is only capable of learning continuous skills with dimension 10 . We show how small skill spaces can lead to ineffective exploration in a simple gridworld setting in Appendix I and evidence that skill dimension affects performance in Fig . 6 . On the importance of benchmarks for evaluation : While prior competence-based approaches such as DIAYN ( Eysenbach et al. , 2019 ) were evaluated on OpenAI Gym ( Brockman et al. , 2016 ) , Gym environment episodes terminate when the agent loses balance thereby leaking some aspects of extrinsic signal to the exploration agent . On the other hand , DMC episodes have fixed length . We show in Appendix K that this small difference in environments results in large performance differences . In Fig . 11 we show that DIAYN is able to learn diverse skills in Gym but not in DMC , which is consistent with both observations from DIAYN and URLB papers . Due to fixed episode lengths , DMC tasks are harder for reward-free exploration since agents must learn to balance without supervision .

Summary: The paper builds upon the DIAYN idea (Eysenback 2018) that an agent could develop skills in an unsupervised environment by finding a set of skills that collectively visits the whole state space but encourages each skill to cover a different subspace and later use one of these skills to simplify the learning of a downstream task. In this paper skills are learned in an unsupervised way using a mutual information based objective. The mutual information between latent states T and skill vector Z, I(T;Z), is decomposed as I(T;Z)=H(T)-H(T|Z) which the paper argues leads to explicit maximization of diversity of latent states T as well as distinct skills with focused effects by penalizing with H(T|Z). More diversity equals better exploration of more distant states leading to more interesting behaviors. The paper explains this decomposition also allows them to user higher-dimensional skill vectors that improve representational capacity and downstream performance. The paper develops a method to calculate the terms of this decomposition and formally shows it is a lower bound for the true mutual information. The latent state entropy H(T) is estimated using an unnormalized k-nearest neighbor method requiring an ad hoc scaling factor \alpha and the conditional entropy H(T|Z) is calculated using an NCE supervised neural net.  The paper illustrates the method by using their loss function to pretrain a DDPG architecture on unsupervised scenarios and then showing a benefit on downstream tasks. The proposed method, CIC, shows a larger IQM stabilized expert normalized score compared to state of the art methods.  Experiments also show that high-dimensio