MADE: Exploration via Maximizing Deviation from Explored Regions

1 Introduction . Online RL is a useful tool for an agent to learn how to perform tasks , particularly when expert demonstrations are unavailable and reward information needs to be used instead [ 92 ] . To learn a satisfactory policy , an RL agent needs to effectively balance between exploration and exploitation , which remains a central question in RL [ 23 , 15 ] . Exploration is particularly challenging in environments with sparse rewards . One popular approach to exploration is based on intrinsic motivation , often applied by adding an intrinsic reward ( or bonus ) to the extrinsic reward provided by the environment . In provable exploration methods , bonus often captures the value estimate uncertainty and the agent takes an action that maximizes the upper confidence bound ( UCB ) [ 5 , 8 , 41 , 48 , 44 ] . In tabular setting , UCB bonuses are often constructed based on either Hoeffding ’ s inequality , which only uses visitation counts , or Bernstein ’ s inequality , which uses value function variance in addition to visitation counts . ⇤Equal Contribution . 2Our code is available at https : //github.com/tianjunz/MADE . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The latter is proved to be minimax near-optimal in environments with bounded rewards [ 44 , 65 ] as well as bounded total reward [ 112 ] and reward-free settings [ 64 , 49 , 45 , 113 ] . It remains an open question how one can efficiently compute confidence bounds to construct UCB bonus in non-linear function approximation . Furthermore , Bernstein-style bonuses are often hard to compute in practice beyond tabular setting , due to difficulties in computing value function variance . In practice , various approaches are proposed to design intrinsic rewards : visitation pseudo-count bonuses estimate count-based UCB bonuses using function approximation [ 10 , 15 ] , curiosity-based bonuses seek states where model prediction error is high , uncertainty-based bonuses [ 77 , 87 ] adopt ensembles of networks for estimating variance of the Q-function , empowerment-based approaches [ 52 , 34 , 84 , 68 ] lead the agent to states over which the agent has control , and information gain bonuses [ 51 ] reward the agent based on the information gain between state-action pairs and next states . Although the performance of practical intrinsic rewards is good in certain domains , empirically they are observed to suffer from issues such as detachment , derailment , and catastrophic forgetting [ 3 , 23 ] . Moreover , these methods usually lack a clear objective and can get stuck in local optimum [ 3 ] . Indeed , the impressive performance currently achieved by some deep RL algorithms often revolves around manually designing dense rewards [ 13 ] , complicated exploration strategies utilizing a significant amount of domain knowledge [ 23 ] , or operating in the known environment regime [ 88 , 69 ] . Motivated by current practical challenges and the gap between theory and practice , we propose a new algorithm for exploration by maximizing deviation from explored regions . This yields a practical algorithm with strong empirical performance . To be specific , we make the following contributions : 1 . Exploration via maximizing deviation Our approach is based on modifying the standard RL objective ( i.e . the cumulative reward ) by adding a regularizer that adaptively changes across iterations . The regularizer can be a general function depending on the state-action visitation density and previous state-action coverage . We show that the regularized objective naturally admits several common existing exploration methods . We then choose a particular regularizer that MAximizes the DEviation ( MADE ) of the next policy visitation d⇡ from the regions covered by prior policies ⇢kcov : Lk ( d ⇡ ) = J ( d⇡ ) + ⌧k X s , a q d ⇡ ( s , a ) ⇢kcov ( s , a ) . ( 1 ) Here , k is the iteration number , J ( d⇡ ) is the standard RL objective , and the regularizer encourages d ⇡ ( s , a ) to be large when ⇢kcov ( s , a ) is small . We give an algorithm for solving the regularized objective and prove that with access to an approximate planning oracle , it converges to the global optimum . We show that objective ( 1 ) results in an intrinsic reward that can be easily added to any RL algorithm to improve performance , as suggested by our empirical studies . Furthermore , the intrinsic reward applies a simple modification to the UCB-style bonus that considers prior visitation counts . This simple modification can also be added to existing bonuses in practice . 2 . Tabular studies In the special case of tabular parameterization , we show that MADE only applies some simple adjustments to the Hoeffding-style count-based bonus . We compare the performance of MADE to Hoeffding and Bernstein bonuses in three different RL algorithms , for the exploration task in the stochastic diabolical bidirectional lock [ 3 , 66 ] , which has sparse rewards and local optima . Our results show that MADE robustly improves over the Hoeffding bonus and is competitive to the Bernstein bonus , across all three RL algorithms . Interestingly , MADE bonus and exploration strategy appear to be very close to the Bernstein bonus , without computing or estimating variance , suggesting that MADE potentially captures some environmental structures . Additionally , we empirically show that MADE regularizer can improve the optimization rate in policy gradient methods . 3 . Experiments on MiniGrid and DeepMind Control Suite We empirically show that MADE works well when combined with model-free ( IMAPLA [ 25 ] , RAD [ 55 ] ) and model-based ( Dreamer [ 35 ] ) RL algorithms , greatly improving the sample efficiency over existing baselines . When tested in the procedurally-generated MiniGrid environments , MADE manages to converge with two to five times fewer samples compared to state-of-the-art method BeBold [ 111 ] . In DeepMind Control Suite [ 95 ] , we build upon the model-free method RAD [ 55 ] and the model-based method Dreamer [ 35 ] , improving the return up to 150 in 500K steps compared to baselines . Figure 1 shows normalized sample size to achieve maximum reward with respect to our algorithm . 2 Background . Markov decision processes . An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P , r , ⇢ , ) , where S is the state space , A is the action space , P : S ⇥ A 7 ! ( S ) is the transition kernel , r : S ⇥ A 7 ! [ 0 , 1 ] is the ( extrinsic ) reward function , ⇢ : S 7 ! ( S ) is the initial distribution , and 2 [ 0 , 1 ) is the discount factor . A stationary ( stochastic ) policy ⇡ 2 ( A | S ) specifies a distribution over actions in each state . Each policy ⇡ induces a visitation density over state-action pairs d⇡ : S ⇥ A 7 ! [ 0 , 1 ] defined as d⇡ ⇢ ( s , a ) : = ( 1 ) P1 t=0 t Pt ( st = s , at = a ; ⇡ ) , where Pt ( st = s , at = a ; ⇡ ) denotes ( s , a ) visitation probability at step t , starting at s0 ⇠ ⇢ ( · ) and following ⇡ . An important quantity is the value a policy ⇡ , which is the discounted sum of rewards V ⇡ ( s ) : = E [ P1 t=0 t rt | s0 = s , at ⇠ ⇡ ( · | st ) for all t 0 ] starting at state s 2 S . Policy mixture . For a sequence of policies Ck = ( ⇡1 , . . . , ⇡k ) with corresponding mixture distribution wk 2 k 1 , the policy mixture ⇡mix , k = ( Ck , wk ) is obtained by first sampling a policy from wk and then following that policy over subsequent steps [ 36 ] . The mixture policy induces a state-action visitation density according to d⇡mix ( s , a ) = P k i=1 w k i d ⇡i ( s , a ) . While the ⇡mix may not be stationary in general , there exists a stationary policy ⇡0 such that d⇡ 0 = d⇡mix Puterman [ 80 ] . Online reinforcement learning . Online RL is the problem of finding a policy with a maximum value from an unknown MDP , using samples collected during exploration . Oftentimes , the following objective is considered , which is a scalar summary of the performance of policy ⇡ : JM ( ⇡ ) : = Es⇠⇢ [ V ⇡ ( s ) ] = ( 1 ) 1 E ( s , a ) ⇠d⇡⇢ ( · , · ) [ r ( s , a ) ] = J ( d ⇡ ) . ( 2 ) We drop index M when it is clear from context . We denote an optimal policy by ⇡ ? 2 arg max ⇡ J ( ⇡ ) and use the shorthand V ? : = V ⇡ ? to denote the optimal value function . It is straightforward to check that J ( ⇡ ) can equivalently be represented by the expectation of the reward over the visitation measure of ⇡ . We slightly abuse the notation and sometimes write J ( d⇡ ) to denote the RL objective . 3 Adaptive regularization of the RL objective . 3.1 Regularization to guide exploration . In online RL , the agent faces a dilemma in each step : whether it should select a seemingly optimal policy ( exploit ) or it should explore different regions of the MDP . To allow flexibility in this choice and trade off between exploration and exploitation , we propose to add a regularizer to the standard RL objective that changes throughout iterations of an online RL algorithm : Lk ( d ⇡ ) = J ( d⇡ ) | { z } exploitation +⌧k R ( d ⇡ ; { d⇡i } k i=1 ) | { z } exploration . ( 3 ) Here , R ( d⇡ ; { d⇡i } k i=1 ) is a function of state-action visitation of ⇡ as well as the visitation of prior policies ⇡1 , . . . , ⇡k . The temperature parameter ⌧k determines the strength of regularization . Objective ( 3 ) is a population objective in the sense that it does not involve empirical estimations affected by the randomness in sample collection . In the following section , we give our particular choice of regularizer and discuss how this objective can describe some popular exploration bonuses . We then provide a convergence guarantee for the regularized objective in Section 3.2 . 3.2 Exploration via maximizing deviation from policy cover . We develop our exploration strategy MADE based on a simple intuition : maximizing the deviation from the explored regions , i.e . all states and actions visited by prior policies . We define policy cover at iteration k to be the density over regions explored by policies ⇡1 , . . . , ⇡k , i.e . ⇢kcov ( s , a ) : = 1 k P k i=1 d ⇡i ( s , a ) . We then design our regularizer to encourage d⇡ to be different from ⇢kcov : Rk ( d ⇡ ; { d⇡i } k i=1 ) = X s , a s d⇡ ( s , a ) ⇢kcov ( s , a ) . ( 4 ) It is easy to check that the maximizer of above function is d⇡ ( s , a ) / 1 ⇢kcov ( s , a ) . Our motivation behind this particular deviation is that it results in a simple modification of UCB bonus in tabular case . We now compute the reward yielded by the new objective . First , define a policy mixture ⇡mix , k with policy sequence ( ⇡1 , . . . , ⇡k ) and weights ( ( 1 ⌘ ) k 1 , ( 1 ⌘ ) k 2⌘ , ( 1 ⌘ ) k 3⌘ , , . . . , ⌘ ) for ⌘ > 0 . Let d⇡mix , k be the visitation density of ⇡mix , k . We compute the total reward at iteration k by taking the gradient of the new objective with respect to d⇡ at d⇡mix , k : rk ( s , a ) = ( 1 ) rdLk ( d ) d=d⇡mix , k = r ( s , a ) + ( 1 ) ⌧krdRk ( d ; { d⇡i } ki=1 ) d=d⇡mix , k , ( 5 ) which gives the following reward rk ( s , a ) = r ( s , a ) + ( 1 ) ⌧k/2p d⇡mix , k ( s , a ) ⇢kcov ( s , a ) . ( 6 ) The intrinsic reward is constructed based on two densities : ⇢kcov a uniform combination of past visitation densities and d̂⇡mix , k a weighted mixture of the past visitation densities . As we will discuss shortly , policy cover ⇢kcov ( s , a ) is related to the visitation count of ( s , a ) pair in previous iterations and resembles count-based bonuses [ 10 , 44 ] or their approximates such as RND [ 15 ] . Therefore , for an appropriate choice of ⌧k , MADE intrinsic reward decreases as the number of visitations increases . MADE intrinsic reward is also proportional to 1/ p d⇡mix , k ( s , a ) , which can be viewed as a correction applied to the count-based bonus . In effect , due to the decay of weights in ⇡mix , k , the above construction gives a higher reward to ( s , a ) pairs visited earlier . Experimental results suggest that this correction may alleviate major difficulties in sparse reward exploration , namely detachment and catastrophic forgetting , by encouraging the agent to revisit forgotten states and actions . Empirically , MADE ’ s intrinsic reward is computed based on estimates d̂⇡mix , k and ⇢̂kcov from data collected by iteration k. Furthermore , practically we consider a smoothed version of the above regularizer by adding > 0 to both numerator and denominator ; see Equation ( 7 ) . MADE intrinsic reward in tabular case . In tabular empirical setting , the empirical estimation of policy cover is simply ⇢̂kcov ( s , a ) = Nk ( s , a ) Nk , where Nk ( s , a ) is ( s , a ) pair ’ s visitation count and Nk is the total count , by iteration k. Setting ⌧k = 1/ p Nk , MADE simply modifies the Hoeffding-type bonus via the mixture density and is proportional to 1/ q d̂⇡mix , k ( s , a ) Nk ( s , a ) . Bernstein bonus is another tabular UCB bonus that modifies Hoeffding bonus via an empirical estimate of the value function variance . Bernstein bonus is shown to improve over Hoeffding count-only bonus by exploiting additional environment structure [ 106 ] and close the gap between algorithmic upper bounds and information-theoretic limits up to logarithmic factors [ 112 , 113 ] . However , a practical and efficient implementation of a bonus that exploits variance information in non-linear function approximation parameterization still remains an open question ; see Section 6 for further discussion . On the other hand , our proposed modification based on the mixture density can be easily and efficiently incorporated with non-linear parameterization . Deriving some popular bonuses from regularization . The regularization in ( 3 ) can describe some popular bonuses . Exploration bonuses that only depend on state-action visitation counts can be expressed in the form ( 3 ) by setting the regularizer a linear function of d⇡ and the exploration bonus ri ( s , a ) , i.e. , Rk ( d⇡ ; { d⇡i } ki=1 ) = P s , a d ⇡ ( s , a ) ri ( s , a ) . One can check that taking the gradient of the regularizer with respect to d⇡ recovers ri ( s , a ) . As another example , one can set the regularizer to Shannon entropy Rk ( d⇡ ; { d⇡i } ki=1 ) = P s , a d ⇡ ( s , a ) log d⇡ ( s , a ) , which gives the intrinsic reward log d⇡ ( s , a ) ( up to an additive constant ) and recovers the result in Zhang et al . [ 109 ] . Algorithm 1 Policy computation for adaptively regularized objective 1 : Inputs : Iteration count K , planning error ✏p , visitation density error ✏d . 2 : Initialize policy mixture ⇡mix,1 = with C1 = ( ⇡1 ) and w1 = ( 1 ) 3 : for k = 1 , . . . , K do 4 : Estimate the visitation density d̂⇡mix , k of ⇡mix , k via a visitation density oracle . 5 : Compute reward rk ( s , a ) = r ( s , a ) + ( 1 ) ⌧krdRk ( d ; { ⇡i } ki=1 ) d=d̂⇡mix , k . 6 : Run approximate planning on modified MDP Mk = ( S , A , P , rk , ) and return ⇡k+1 . 7 : Update policy mixture Ck+1 = ( Ck , ⇡k+1 ) and wk+1 = ( ( 1 ⌘ ) wk , ⌘ ) . 8 : Return : ⇡mix , K = ( Ck , wk ) .

Summary: The authors propose adding a regularisation term, that encourages a policy to visit states, less visited by previous policies, to the RL problem to address the exploration-exploitation tradeoff. The result is a convex regularised optimisation problem in the state occupancy of the policy. In addition, the authors propose an algorithm to solve this problem that is based on the Frank-Wolfe algorithm, similar to Hazan et al. i.e., they iteratively find a policy that has a state occupancy with maximal correlation with the negative gradient of the objective. Experiments are performed in several domains and include a comparison with various baselines from the reward free literature.