Learning more skills through optimistic exploration

1 INTRODUCTION . Reinforcement learning ( RL ) has proven itself capable of learning useful skills when clear taskspecific rewards are available ( OpenAI et al. , 2019a ; b ; Vinyals et al. , 2019 ) . However , truly intelligent agents should , like humans , be able to learn even in the absence of supervision in order to acquire repurposable task-agnostic knowledge . Such unsupervised pre-training has seen recent success in language ( Radford et al. , 2019 ; Brown et al. , 2020 ) and vision ( Chen et al. , 2020a ; b ) , but its potential has yet to be fully realized in the learning of behavior . The most promising class of algorithms for unsupervised skill discovery is based upon maximizing the discriminability of skills represented by latent variables on which a policy is conditioned . Objectives are typically derived from variational approximations to the mutual information between latent variables and states visited ( Gregor et al. , 2016 ; Eysenbach et al. , 2019 ; Warde-Farley et al. , 2019 ; Hansen et al. , 2020 ; Baumli et al. , 2021 ) , employing a learned parametric skill discriminator . Both the policy and discriminator have the objective of strong predictive performance by the discriminator , though the discriminator is trained with supervised learning rather than RL . The end result is a policy capable of producing consistently distinguishable behavioral motifs , or “ skills. ” These skills can be evaluated zero-shot , fine-tuned , or composed in a hierarchical RL setup to maximize task reward when one is introduced ( Eysenbach et al. , 2019 ; Hansen et al. , 2020 ) . Unsupervised skill learning objectives can also be maximized in conjunction with task reward , in order to promote robustness of learned behavior to environment perturbations ( Mahajan et al. , 2019 ; Kumar et al. , 2020 ) . ∗equal contribution The degree to which unsupervised skill discovery methods are useful in such downstream applications depends on how many skills they are able to learn . Indeed , Eysenbach et al . ( 2019 ) showed that as more skills are learned , the performance obtained using the learned skills in a hierarchical reinforcement learning setup improves . In follow up work , Achiam et al . ( 2018 ) showed that methods like DIAYN can struggle to learn large numbers of skills and proposed gradually increasing the number of skills according to a curriculum to make skill discovery easier . The aim of this work is to improve the ability of skill discovery methods to learn more skills . We highlight an exploration problem intrinsic to the entire class of variational skill discovery algorithms which can inhibit discovery of new skills . During skill learning , the policy will necessarily need to explore new states of the environment . The discriminator must then make latent predictions for states it has never seen before , resulting in incorrect and/or low-confidence predictions . The policy will in turn be penalized for this poor discriminator performance , and discouraged from seeking out new states . We refer to this problem as “ pessimistic exploration ” and describe it further in section 2 . To motivate our solution , we argue that it is important for the policy to distinguish between two kinds of uncertainty in the discriminator - aleatoric uncertainty that comes from the policy producing similar trajectories for different skills , and epistemic uncertainty that comes from a lack of training data . The former indicates poor policy performance , but the latter is in fact desirable , and serves as a signal for potential discriminator learning . Unfortunately , skill discovery algorithms treat both types of uncertainty the same and thus ignore this important signal . We propose to capture it . Our primary contribution , which we present in section 3 , is an exploration bonus tailored to skill discovery algorithms , designed to overcome pessimistic exploration . We identify states of high epistemic uncertainty in the discriminator by training an ensemble of discriminators and measuring their disagreement . We call this exploration bonus discriminator disagreement intrinsic reward , or DISDAIN . Intuitively , the ensemble members may disagree in novel states , but must come to agree in frequently visited ones . More formally , we derive DISDAIN from a Bayesian perspective that 1 ) represents the posterior over discriminator parameters using an ensemble , and 2 ) encourages the policy to maximize information gain ( i.e . reduce uncertainty ) about the parameters of the discriminator . In section 4 , we demonstrate empirically that DISDAIN improves skill learning over unbonused skill discovery algorithms in both an illustrative grid world ( Four Rooms , Sutton et al . ( 1999 ) ) and the Atari 2600 learning environment ( ALE , Bellemare et al . ( 2013 ) ) more so than augmenting with popular exploration bonuses not tailored to skill discovery . 2 UNSUPERVISED SKILL LEARNING THROUGH VARIATIONAL INFOMAX . 2.1 INTRODUCTION . We now formalize our setting of interest : unsupervised skill learning through variational information maximization ( Gregor et al. , 2016 ; Eysenbach et al. , 2019 ; Warde-Farley et al. , 2019 ; Hansen et al. , 2020 ; Baumli et al. , 2021 ) . We consider a Markov decision process ( MDP ) , defined by the tuple M = ( S , A , pE , ρ , r , γ ) , where S and A are state and action spaces , respectively , the environment dynamics pE ( s′ | s , a ) specifies the probability of transitioning to state s′ ∈ S when taking action a ∈ A in state s ∈ S , ρ ( s ) denotes the probability of starting an episode in state s , and γ ∈ [ 0 , 1 ) is a discount factor . Since we focus on unsupervised training , we ignore the environmental reward r. Our agents seek to learn a repertoire of skills , indexed by the latent variable Z and represented by the policy πθ ( a | s , z ) , which is parameterized by θ and maps from states and latent variables to distributions over actions . The latent variables are sampled z ∼ p ( Z ) at the beginning of each trajectory and then fixed , so each z represents a temporally extended behavior . The skill trajectory length T may differ from the episode length , thus a new skill might be resampled within an episode . For conciseness , we will denote trajectories sampled from the policy by τ ∼ π ( z ) when conditioning on a particular skill z , and τ ∼ π when collecting trajectories across skills . To simplify our discussion , and because it is the most common case in practice , we will assume that Z is categorical with cardinality NZ , although much of the discussion carries over to continuous Z . A large and growing class of objectives for unsupervised skill discovery are derived from maximizing the mutual information between the latent skill Z and some feature of the resulting trajectories O ( τ ) : F ( θ ) ≡ I ( Z , O ) = H ( Z ) −H ( Z | O ) = Ez∼p ( z ) , τ∼π ( z ) [ log p ( z | o ( τ ) ) − log p ( z ) ] . ( 1 ) For example , variational intrinsic control ( VIC , Gregor et al . ( 2016 ) ) maximizes I ( Z ; S0 , ST ) - the mutual information between the skill and initial and final states ( oT = ( s0 , sT ) ) .1 Diversity is all you need ( DIAYN , Eysenbach et al . ( 2019 ) ) , on the other hand , maximizes I ( Z , S ) - the mutual information between the skill and each state along the trajectory ( ot = st for t ∈ 1 : T ) . Intuitively , VIC produces skills that vary in the destination reached ( without regard for the path taken ) , while DIAYN produces skills that vary in the path taken ( with less emphasis on the destination reached ) . In practice , maximizing equation 1 is not straightforward , because it requires calculating the conditional distribution p ( Z | O ) . In general , it is necessary to estimate it with a learned parametric model qφ ( Z | O ) . We refer to this model as the discriminator , since it is trained to discriminate between skills . Fortunately , replacing p with q still yields a lower bound on F ( θ ) ( Barber and Agakov , 2004 ) , and we may instead maximize the proxy objective F̃ ( θ ) : F ( θ ) ≥ F̃ ( θ ) = Ez∼p ( z ) , τ∼π ( z ) [ log qφ ( z | o ( τ ) ) − log p ( z ) ] . ( 2 ) Optimizing F̃ ( θ ) with respect to the policy parameters θ corresponds to RL on the reward : rskill = log qφ ( z | o ) − log p ( z ) . ( 3 ) Since the intent is for the agent to learn a full repertoire of skills , the skill prior p ( z ) is typically fixed to be uniform ( Eysenbach et al. , 2019 ; Achiam et al. , 2018 ; Baumli et al. , 2021 ) , in which case − log p ( z ) = logNZ . If the discriminator simply ignores the trajectory and guesses skills uniformly as well , then log qφ ( z | o ) = − logNZ and the reward will be zero . If the discriminator instead guesses perfectly , then log qφ ( z | o ) = 0 , and the reward will be logNZ . More generally , the expected reward is an estimate of the logarithm of the effective number of skills . Thus , measuring the reward in bits ( i.e . using log2 in equation 3 ) , we can estimate the number of skills learnt as : nskills = 2 E [ rskill ] . ( 4 ) We adopt this quantity as our primary performance metric for our experiments . To make sure the bound in equation 2 is as tight as possible , the discriminator qφ ( Z | O ) must also be fit to its target p ( Z | O ) through supervised learning ( SL ) on the negative log likelihood loss : L ( φ ) ≡ −Ez∼p ( z ) , τ∼π ( z ) [ log qφ ( z | o ( τ ) ) ] . ( 5 ) The loss is minimized , and the bound in equation 2 tight , when qφ ( Z | O ) = p ( Z | O ) . The joint optimization of F̃ ( θ ) by RL and L ( φ ) by SL forms a cooperative communication game between policy and discriminator . The agent samples a skill z ∼ p ( Z ) and generates the “ message ” τ ∼ π ( z ) . The discriminator receives the message τ and attempts to decode the original skill z . When the policy produces trajectories for different skills that do not overlap in the features O ( τ ) , the discriminator will easily learn to label trajectories , and when the discriminator makes accurate and confident predictions , the reward in equation 3 will be high . Ideally , the end result is a policy exhibiting a maximally diverse set of skills . This joint-training system is depicted in Figure 2a .

Summary: This paper is concerned with unsupervised RL where an extrinsic reward signal is not available. The objective is for the agent to master the environment by exploring it while learning a diverse set of skills. This is done by simultaneously training a policy conditioned on a latent variable and a discriminator that tries to infer the latent variable from trajectories. The authors identify that the intrinsic rewards used in skill discovery result in the agent being pessimistic towards exploring novel parts of the environment. To alleviate this, the authors propose a new auxiliary objective, which results in a bonus based on the disagreement of an ensemble of discriminators. Empirical results on the grid world and Atari show improvements in skill discovery and solving downstream tasks compared to baselines.