Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching

1 INTRODUCTION . Deep reinforcement learning ( RL ) algorithms have been shown to effectively solve a wide variety of complex problems ( e.g. , Mnih et al. , 2015 ; Bellemare et al. , 2013 ) . However , they are often designed to solve one single task at a time and they need to restart the learning process from scratch for any new problem , even when it is defined on the very same environment ( e.g. , a robot navigating to different locations in the same apartment ) . Recently , Unsupervised RL ( URL ) has been proposed as an approach to address this limitation . In URL , the agent first interacts with the environment without any extrinsic reward signal . Afterward , the agent leverages the experience accumulated during the unsupervised learning phase to efficiently solve a variety of downstream tasks defined on the same environment . This approach is particularly effective in problems such as navigation ( see e.g. , Bagaria et al. , 2021 ) and robotics ( see e.g. , Pong et al. , 2020 ) where the agent is often required to readily solve a wide range of tasks while the dynamics of environment remains fixed . In this paper , we focus on the unsupervised objective of discovering a set of skills that can be used to efficiently solve sparse-reward downstream tasks . In particular , we build on the insight that mutual information ( MI ) between the skills ’ latent variables and the states reached by them effectively formalizes the dual objective of learning policies that both cover and navigate the environment efficiently ( e.g. , Gregor et al. , 2016 ) . Indeed , maximizing MI has been shown to be a powerful approach for encouraging exploration in RL ( Houthooft et al. , 2016 ; Mohamed & Rezende , 2015 ) and for unsupervised skill discovery ( e.g. , Gregor et al. , 2016 ; Eysenbach et al. , 2019 ; Achiam et al. , 2018 ; Sharma et al. , 2020 ; Campos et al. , 2020 ) . Nonetheless , learning policies that maximize MI is a challenging optimization problem . Several approximations have been proposed to simplify it at the cost of possibly deviating from the original objective of coverage and directedness ( see Sect . 4 for a review of related work ) . In this paper , we propose UPSIDE ( UnsuPervised Skills that dIrect then DiffusE ) to learn a set of policies that can be effectively used to cover the environment and solve goal-reaching downstream tasks . Our solution builds on the following components ( Fig . 1 ) : • Policy structure ( Sect . 3.1 , see Fig . 1 ( A ) ) . We consider policies composed of two parts : 1 ) a directed part , referred to as the skill , that is trained to reach a specific region of the environment , and 2 ) a diffusing part that induces a local coverage around the region attained by the first part . This structure favors coverage and directedness at the level of a single policy . • New constrained objective ( Sect . 3.2 , see Fig . 1 ( B ) & ( C ) ) . We then introduce a constrained optimization problem designed to maximize the number of policies under the constraint that the states reached by each of the diffusing parts are distinct enough ( i.e. , they satisfy a minimum level of discriminability ) . We prove that this problem can be cast as a lower bound to the original MI objective , thus preserving its coverage-directedness trade-off . UPSIDE solves it by adaptively adding or removing policies to a given initial set , without requiring any prior knowledge on a sensible number of policies . • Tree structure ( Sect . 3.3 , see Fig . 1 ( D ) & ( E ) ) . Leveraging the directed nature of the skills , UPSIDE effectively composes them to build longer and longer policies organized in a tree structure . This overcomes the need of defining a suitable policy length in advance . Thus in UPSIDE we can consider short policies to make the optimization easier , while composing their skills along a growing tree structure to ensure an adaptive and thorough coverage of the environment . The combination of these components allows UPSIDE to effectively adapt the number and the length of policies to the specific structure of the environment , while learning policies that ensure coverage and directedness . We study the effectiveness of UPSIDE and the impact of its components in hardto-explore continuous navigation and control environments , where UPSIDE improves over existing baselines both in terms of exploration and learning performance . 2 SETTING . We consider the URL setting where the agent interacts with a Markov decision process ( MDP ) M with state space S , action space A , dynamics p ( s′|s , a ) , and no reward . The agent starts each episode from a designated initial state s0 ∈ S.1 Upon termination of the chosen policy , the agent is then reset to s0 . This setting is particularly challenging from an exploration point of view since the agent can not rely on the initial distribution to cover the state space . We recall the MI-based unsupervised skill discovery approach ( see e.g. , Gregor et al. , 2016 ) . Denote by Z some ( latent ) variables on which the policies of length T are conditioned2 ( we assume that Z is categorical for simplicity and because it is the most common case in practice ) . There are three optimization variables : ( i ) the cardinality of Z denoted by NZ , i.e. , the number of policies ( we write Z = { 1 , . . . , NZ } = [ NZ ] ) , ( ii ) the parameters π ( z ) of the policy indexed by z ∈ Z , and ( iii ) the 1More generally , s0 could be drawn from any distribution supported over a compact region . 2In URL , there is no terminal state and the agent has to define a criterion to terminate the episode and restart . In practice , most of unsupervised skill discovery methods set a fixed number of steps . policy sampling distribution ρ ( i.e. , ρ ( z ) is the probability of sampling policy z at the beginning of the episode ) . Denote policy z ’ s action distribution in state s by π ( ·|z , s ) and the entropy function by H. Let the variable ST be the random ( final ) state induced by sampling a policy z from ρ and executing π ( z ) from s0 for an episode . Denote by pπ ( z ) ( sT ) the distribution over ( final ) states induced by executing policy z , by p ( z|sT ) the probability of z being the policy to induce state sT , and let p ( sT ) = ∑ z∈Z ρ ( z ) pπ ( z ) ( sT ) . Maximizing the MI between Z and ST can be written as max NZ , ρ , π I ( ST ; Z ) = H ( ST ) −H ( ST |Z ) = − ∑ sT p ( sT ) log p ( sT ) + ∑ z∈Z ρ ( z ) EsT |z [ log pπ ( z ) ( sT ) ] = H ( Z ) −H ( Z|ST ) = − ∑ z∈Z ρ ( z ) log ρ ( z ) + ∑ z∈Z ρ ( z ) EsT |z [ log p ( z|sT ) ] , ( 1 ) where in the expectations sT |z ∼ pπ ( z ) ( sT ) . In the first formulation , the entropy term over states captures the requirement that policies thoroughly cover the state space , while the second term measures the entropy over the states reached by each policy and thus promotes policies that have a directed behavior . Learning the optimal NZ , ρ , and π is a challenging problem and several approximations have been proposed ( see e.g. , Gregor et al. , 2016 ; Eysenbach et al. , 2019 ; Achiam et al. , 2018 ; Campos et al. , 2020 ) . Many approaches focus on the so-called reverse formulation of the MI ( second line of Equation 1 ) . In this case , the conditional distribution p ( z|sT ) is usually replaced with a parametric model qφ ( z|sT ) called the discriminator that is trained via a negative log likelihood loss simultaneously with all other variables . Then one can maximize the lower bound ( Barber & Agakov , 2004 ) : I ( ST ; Z ) ≥ Ez∼ρ ( z ) , τ∼π ( z ) [ log qφ ( z|sT ) − log ρ ( z ) ] , where we denote by τ ∼ π ( z ) trajectories sampled from the policy indexed by z . As a result , each policy π ( z ) can be trained with RL to maximize the intrinsic reward rz ( sT ) : = log qφ ( z|sT ) − log ρ ( z ) . 3 THE UPSIDE ALGORITHM In this section we detail the three main components of UPSIDE , which is summarized in Sect . 3.4 . 3.1 DECOUPLED POLICY STRUCTURE OF DIRECT-THEN-DIFFUSE . While the trade-off between coverage and directedness is determined by the MI objective , the amount of stochasticity of each policy ( e.g. , injected via a regularization on the entropy over the actions ) has also a major impact on the effectiveness of the overall algorithm ( Eysenbach et al. , 2019 ) . In fact , while randomness can promote broader coverage , a highly stochastic policy tends to induce a distribution pπ ( z ) ( sT ) over final states with high entropy , thus decreasing −H ( ST |Z ) and losing in directedness . In UPSIDE , we define policies with a decoupled structure ( see Fig . 1 ( A ) ) composed of a ) a directed part ( of length T ) that we refer to as skill , with low stochasticity and trained to reach a specific region of the environment and b ) a diffusing part ( of length H ) with high stochasticity to promote local coverage of the states around the region reached by the skill . Coherently with this structure , the state variable in the conditional entropy in Equation 1 becomes any state reached during the diffusing part ( denote by Sdiff the random variable ) and not just the skill ’ s terminal state . Following Sect . 2 we define an intrinsic reward rz ( s ) = log qφ ( z|s ) − log ρ ( z ) and the skill of policy z maximizes the cumulative reward over the states traversed by the diffusing part . Formally , we can conveniently define the objective function : max π ( z ) Eτ∼π ( z ) [ ∑ t∈J α · rz ( st ) + β · H ( π ( ·|z , st ) ) ] , ( 2 ) where J = { T , . . . , T + H } and α = 1 , β = 0 ( resp.α = 0 , β = 1 ) when optimizing for the skill ( resp . diffusing part ) . In words , the skill is incentivized to bring the diffusing part to a discriminable region of the state space , while the diffusing part is optimized by a simple random walk policy ( i.e. , a stochastic policy with uniform distribution over actions ) to promote local coverage around sT . Table 1 illustrates how UPSIDE ’ s policies compare to other methods . Unlike VIC and similar to DIAYN , the diffusing parts of the policies tend to “ push ” the skills away so as to reach diverse regions of the environment . The combination of the directedness of the skills and local coverage of the diffusing parts thus ensures that the whole environment can be properly visited with NZ |S| policies.3 Furthermore , the diffusing part can be seen as defining a cluster of states that represents the goal region of the directed skill . This is in contrast with DIAYN policies whose stochasticity may be spread over the whole trajectory . This allows us to “ ground ” the latent variable representations of the policies Z to specific regions of the environment ( i.e. , the clusters ) . As a result , maximizing the MI I ( Sdiff ; Z ) can be seen as learning a set of “ cluster-conditioned ” policies .

Summary: The paper proposes an unsupervised exploration method for reinforcement learning, called UPSIDE, that combines learning of directed skills that enable covering distant states, and a diffusive part that explores locally and helps expand the explored region further. The main contributions are both the topology of the policy (division to tree-structured skills and diffusive parts), a theory for training such policies, and a practical implementation that simplifies some of the technicalities induced by the theory. The experiments illustrate well the assumed benefits and include both toy tasks (a point mass in a maze) as well as more complex tasks such as HalfCheetah and Ant from OpenAI Gym. The experiments also compare to prior methods such as DIYAN and provide ablations of the importance of the different components.