Exploration in Reinforcement Learning with Deep Covering Options

1 INTRODUCTION . Temporal abstraction , often formalized via the options framework ( Sutton et al. , 1999 ) , has the potential to greatly improve the performance of reinforcement learning ( RL ) agents by representing actions at different time scales . However , the question of which options an agent should construct , and the related question of what objective function that option construction process should be optimizing , remain open . One recent approach is to construct options that aid exploration by providing agents with more decisive behavior than the dithering common to random exploration ( e.g. , Menache et al. , 2002 ; Stolle and Precup , 2002 ; Şimşek and Barto , 2004 ; Şimşek et al. , 2005 ; Şimşek and Barto , 2009 ; Machado et al. , 2017 ; Eysenbach et al. , 2019 ) . The Laplacian ( Chung , 1996 ) , the matrix extracted from the graph induced by the agent ’ s policy and the dynamics of the environment , is often used when discovering options for exploration ( e.g. , Machado and Bowling , 2016 ; Machado et al. , 2017 ; 2018 ; Jinnai et al. , 2019b ) . The options discovered with such an approach encourage agents to navigate to parts of the state space that are infrequently visited . However , the existing methods either lack a principled way of constraining the number of discovered options ( e.g. , Machado and Bowling , 2016 ; Machado et al. , 2017 ; 2018 ) or are limited to the tabular setting ( e.g. , Jinnai et al. , 2019b ) . In this paper we show how recent developments in eigenfunction estimation of the Laplacian ( Wu et al. , 2019 ) can be used to extend a principled approach for option discovery ( Jinnai et al. , 2019b ) to the non-linear function approximation case . This new algorithm for option discovery , deep covering options , is computationally tractable and it is applicable to environments with large ( or continuous ) state-spaces . Despite methods that learn representations generally being more flexible , more scalable , and often leading to better performance , before this paper , covering options could not be easily combined with modern representation learning techniques . Deep covering options discovers a small set of options that encourage exploration by minimizing the agent ’ s expected cover time—the expected number of steps required to visit every state in the environment ( Broder and Karlin , 1989 ) . Moreover , unlike most previous approaches to discovering options for exploration , it can be applied to both settings where a pretraining ( unsupervised ) phase is available ( e.g. , Eysenbach et al. , 2019 ) and to the traditional , fully online , setting . We evaluate our method , in both settings , in three different platforms to demonstrate its applicability in a wide range of domains . First , we apply it to the Pinball domain ( Konidaris and Barto , 2009 ) , which has a discrete action-space and a continuous state-space . Second , we apply it to three MuJoCo control tasks ( Todorov et al. , 2012 ) , which are continuous state- and action-space domains . In all of these domains , our method improves over the baseline . Finally , we perform a qualitative analysis of our method in three Atari 2600 games ( Bellemare et al. , 2013 ) to demonstrate its potential in domains with very large state-spaces . Deep covering options successfully finds under-explored regions of the state space and builds options to target those regions . 2 BACKGROUND AND RELATED WORK . We assume the standard reinforcement learning setting ( Sutton and Barto , 1998 ) , where the environment is modeled as a Markov Decision Process ( MDP ) , ( S , A , T , R , γ ) , where S is the set of states , A is the set of actions , T : S ×A×S → [ 0 , 1 ] is the state transition function , R : S ×A → R is the reward function , and 0 ≤ γ ≤ 1 is the discount factor . We use the options framework ( Sutton et al. , 1999 ) to represent temporally extended actions . It defines an option as a triple ( I , π , β ) , where I ⊆ S is the set of states in which the option can initiate , π : S → Pr ( A ) is the policy the agent follows when that option is being executed , and β : S → [ 0 , 1 ] , is the termination condition . We refer to a set of states in which β ( s ) = 1 as a termination set . 2.1 RELATED WORK . Many option discovery algorithms are based on the reward signals generated by the environment and are thus task dependent . These methods often decompose the trajectories reaching the rewarding states into options . Several papers have proposed generating options from trajectories reaching these rewarding states ( e.g. , McGovern and Barto , 2001 ; Menache et al. , 2002 ; Konidaris and Barto , 2009 ) , while other approaches use the observed rewards to generate options with gradient descent ( e.g. , Mankowitz et al. , 2016 ; Bacon et al. , 2017 ; Harb et al. , 2018 ; Tiwari and Thomas , 2019 ) . These approaches are often ineffective in sparse reward problems , where only a few state-action pairs lead to a positive reward . Fewer papers have tackled the problem of option discovery for exploration without using reward signals . Eysenbach et al . ( 2019 ) proposed to generate options maximizing an information theoretic objective so that each option generates diverse behavior . While many option discovery methods are limited to discrete state and action space tasks , their method can generate options that solve many continuous control tasks , even when ignoring the environment ’ s reward function . Machado et al . ; Machado et al . ( 2017 ; 2018 ) proposed eigenoptions , a method to generate options using the Laplacian eigenvectors ( Chung , 1996 ) . Their approach is similar to covering options but requires the set of options to be orthogonal to each other and introduces a prohibitively large number of options at each iteration . Several papers have proposed identifying subgoal states without reward information through graph concepts such as clustering ( Menache et al. , 2002 ; Şimşek et al. , 2005 ) , visitation statistics ( Şimşek and Barto , 2004 ; Stolle and Precup , 2002 ) , and betweenness centrality ( Şimşek and Barto , 2009 ) . As they use graph algorithms to discover subgoals , their scope is often limited to tabular domains . 2.2 COVERING OPTIONS . Covering options ( Jinnai et al. , 2019b ) is an approach that minimizes the expected cover time of a uniformly random policy by augmenting the agent ’ s action set with options obtained from the eigenvector associated with the second smallest eigenvalue of the Laplacian . Covering options can be seen as increasing the likelihood that a random walk is going to lead to a rewarding state since the expected cover time is the time required for a random walk to visit all the vertices in a graph ( Broder and Karlin , 1989 ) . Covering options achieves such an objective by minimizing the upper bound of the expected cover time , E [ C ( G ) ] , which is given by the second smallest eigenvalue of the normalized Laplacian , λ2 , also known as the algebraic connectivity ( Fiedler , 1973 ) : E [ C ( G ) ] ≤ n 2 lnn λ2 ( 1 + o ( 1 ) ) , ( 1 ) where n is the number of vertices of the graph . Equation 1 shows that the larger the algebraic connectivity , the smaller the upper bound of the expected cover time . Intuitively , algebraic connectivity represents how densely the graph is connected . The eigenvector f corresponding to λ2 is an embedding of a graph to a one-dimensional interval where nodes connected by an edge tend to be placed nearby ( see Figure 1 , adapted from Jinnai et al. , 2019b ) . A pair of nodes with the maximum and minimum value in f are the most distant nodes in the embedding space . Connecting these two nodes greedily maximizes the algebraic connectivity to a first order approximation ( Ghosh and Boyd , 2006 ) . Covering options works as follows : 1 . Compute the second smallest eigenvalue and the corresponding eigenvector f of the Laplacian exactly by solving the following constraint optimization problem : λ2 = inf fTA1=0 fTAf=1 G ( f ) G ( f ) = 1 2 ∑ s∈S [ ( f ( s ) − f ( s′ ) ) 2 A ( s , s′ ) ] , ( 2 ) where A is the adjacency matrix of the state-space graph where the entry at ( s , s′ ) is 1 if s and s′ are adjacent and 0 otherwise . 2 . Let vi and vj be the state with largest and smallest value in the eigenvector respectively . Generate two options ; one with I = { vi } and β = { vj } and the other one with I = { vj } and β = { vi } . Each option policy is the optimal path from the initial state to the termination state . 3 . Set G ← G ∪ { ( vi , vj ) } and repeat the process until the number of options reaches a threshold . While this method is an efficient algorithm with performance guarantees , it is limited to small discrete MDPs as it requires a state-space graph . Moreover , explicitly computing the matrix that encodes the environment ’ s adjacency matrix is unrealistic beyond small problems . Finally , the method is constrained to point options where both the initiation and termination sets consist of a single state ( Jinnai et al. , 2019a ) . Options generated by this method are therefore only executable at a single state . This is not useful for tasks with large ( or continuous ) state-spaces as the probability of visiting the state in the initiation set of the option tends to zero . Even if the agent visits the state in the option ’ s initiation set and starts following the corresponding option ’ s policy , the probability of reaching the state in the termination set is also small ( see Figure 2 ) . In the next section we introduce an approach that addresses these limitations . 3 DEEP COVERING OPTIONS . We propose deep covering options , a new algorithm that finds options that speed-up exploration in domains with large ( or continuous ) state-spaces . It directly seeks to optimize an objective for exploration . If the objective function is optimized , the options generated by the algorithm greedily maximize the algebraic connectivity of the underlying state-space graph to a first order approximation ( Ghosh and Boyd , 2006 ) , which in turn minimize the upper bound on the expected cover time . Deep covering options consists of four steps ( see Algorithm 1 ) : Algorithm 1 Deep covering options 1 : Input : Set of state-transitionsH , a percentile 0 ≤ k ≤ 100 2 : Compute f by minimizing G̃ ( f ) usingH ( Equation 5 ) 3 : β′ ← k-th percentile value of f inH 4 : βo ( s ) ← { 1 if f ( s ) < β′ 0 otherwise 5 : Io ← { s|βo ( s ) = 0 , s ∈ S } 6 : Train πo off-policy by maximizing the total accumulated pseudo-rewards ro = f ( s ) − f ( s′ ) usingH and f 7 : Return ( Io , πo , βo ) 1. compute an eigenfunction of the Laplacian of the state-space graph approximately ( line 2 in Algorithm 1 ) , 2. identify an under-explored region in the state-space using the eigenfunctions ( line 3 ) , 3. set the under-explored region as the termination set and set the compliment of it as the initiation set ( line 4 , 5 ) , 4. train a policy of the option using the pseudo-reward induced by the eigenfunctions ( line 6 ) . There are two problems in Equation 2 that prevent its applicability to non-tabular domains . First , the equation requires the adjacency matrixA as input . Second , a constrained optimization problem is hard to solve using gradient-based methods . We address these issues by approximating the computation of the Laplacian with the following objective ( Wu et al. , 2019 , Equation 6 ) : G̃ ( f1 , f2 , ... , fd ) = 1 2 E ( s , s′ ) ∼H [ d∑ k=1 ( fk ( s ) − fk ( s′ ) ) 2 ] + ηEs∼ρ , s′∼ρ [ ∑ j , k ( fj ( s ) fk ( s ) − δjk ) ( fj ( s ′ ) fk ( s ′ ) − δjk ) ] , ( 3 ) whereH is the set of sampled state-transitions , ρ is a distribution of states in the dataset ( ρ ( s ) is the number of occurrence of s inH divided by the size ofH ) , η is the Lagrange multiplier , and δjk is 1 if j 6= k and 0 otherwise . Such an expression , inspired by spectral graph drawing theory , uses the repulsive term ( the summation multiplied by η ) to ensure the functions f1 , ... , fd are orthogonal to each other . Unlike G , G̃ is a constraint-free objective to compute the eigenfunction , only requiring trajectories instead of the state-space graph . As we only require the second eigenfunction ( unlike eigenoptions ) , we can simplify the objective function to take only two arguments : G̃ ( f1 , f2 ) = G ( f1 , f2 ) + ηEs∼ρ , s′∼ρ [ ∑ j , k ( fj ( s ) fk ( s ) − δjk ) ( fj ( s ′ ) fk ( s ′ ) − δjk ) ] . ( 4 ) Assume G ( f1 ) ≤ G ( f2 ) without loss of generality . G ( f1 ) = 0 and f1 is a constant function because the first eigenvalue of the Laplacian matrix is zero . To simplify the equation , we assume f1 = 1 without loss of generality . Then : G̃ ( f ) = G̃ ( 1 , f ) = 1 2 E ( s , s′ ) ∼H [ ( f ( s ) −f ( s′ ) ) 2 ] +ηEs∼ρ , s′∼ρ [ ( f ( s ) 2−1 ) ( f ( s′ ) 2−1 ) +f ( s ) 2f ( s′ ) 2 ] . ( 5 ) Deep covering options compute the second eigenfunction f by minimizing G̃ ( f ) instead of G ( f ) ( see Algorithm 1 ) . Our objective function only needs sampled state-transitions H instead of a complete state-space graph . As it is an unconstrained optimization problem , we can optimize by simple gradient-based methods . The objective function is essentially the same as the objective function of covering options which has theoretical guarantee on the expected cover time but computed approximately so that it scales to large or infinite state-space domains . k Reward 5 2.5311 ± 1.71 10 3.1873 ± 3.33 30 3.2210 ± 4.54 50 3.0748 ± 3.28 Table 1 : The effect of the size of the termination set ( percentile k ) on the performance of Deep covering options in Pinball with 3 options . Reward is averaged over 100 episodes and 5 runs . While covering options is constrained to options with the initiation set consisting of a single state , we set the termination set as a set of states with f value smaller than its k-th percentile . As proposed by Machado et al . ( 2017 ; 2018 ) , we define the initiation set to be the complement of the termination set . We train the option policy off-policy , maximizing the total pseudo-reward ro = f ( s ) − f ( s′ ) so that it learns to reach the termination set ( i.e. , the set of states with f ( s ) < β′ ) .

Summary: The paper proposes an algorithm to extend the recently proposed method of “covering options” from a tabular setting to continuous state spaces (or large discrete state spaces). The proposed algorithm approximately computes the second eigenfunction of the normalized laplacian of the state space, uses it to identify an under-explored region and trains an option to terminate in such a region. Each new learnt option is added to the initial set of primitive actions and a policy over this growing set of actions is learnt separately. An online algorithm is also proposed that does the above option learning process intermittently in addition to training for an external task. The paper shows empirical evidence of better or equal performance to base algorithms which do not discover options, prior work such as DIAYN (Eysenbach et. al, 2019) (that discover options via mutual information maximization between visited states and options), as well as ablations of their proposed method with different number of options.