MADE: Exploration via Maximizing Deviation from Explored Regions

1 Introduction . Online RL is a useful tool for an agent to learn how to perform tasks , particularly when expert demonstrations are unavailable and reward information needs to be used instead [ 92 ] . To learn a satisfactory policy , an RL agent needs to effectively balance between exploration and exploitation , which remains a central question in RL [ 23 , 15 ] . Exploration is particularly challenging in environments with sparse rewards . One popular approach to exploration is based on intrinsic motivation , often applied by adding an intrinsic reward ( or bonus ) to the extrinsic reward provided by the environment . In provable exploration methods , bonus often captures the value estimate uncertainty and the agent takes an action that maximizes the upper confidence bound ( UCB ) [ 5 , 8 , 41 , 48 , 44 ] . In tabular setting , UCB bonuses are often constructed based on either Hoeffding ’ s inequality , which only uses visitation counts , or Bernstein ’ s inequality , which uses value function variance in addition to visitation counts . ⇤Equal Contribution . 2Our code is available at https : //github.com/tianjunz/MADE . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The latter is proved to be minimax near-optimal in environments with bounded rewards [ 44 , 65 ] as well as bounded total reward [ 112 ] and reward-free settings [ 64 , 49 , 45 , 113 ] . It remains an open question how one can efficiently compute confidence bounds to construct UCB bonus in non-linear function approximation . Furthermore , Bernstein-style bonuses are often hard to compute in practice beyond tabular setting , due to difficulties in computing value function variance . In practice , various approaches are proposed to design intrinsic rewards : visitation pseudo-count bonuses estimate count-based UCB bonuses using function approximation [ 10 , 15 ] , curiosity-based bonuses seek states where model prediction error is high , uncertainty-based bonuses [ 77 , 87 ] adopt ensembles of networks for estimating variance of the Q-function , empowerment-based approaches [ 52 , 34 , 84 , 68 ] lead the agent to states over which the agent has control , and information gain bonuses [ 51 ] reward the agent based on the information gain between state-action pairs and next states . Although the performance of practical intrinsic rewards is good in certain domains , empirically they are observed to suffer from issues such as detachment , derailment , and catastrophic forgetting [ 3 , 23 ] . Moreover , these methods usually lack a clear objective and can get stuck in local optimum [ 3 ] . Indeed , the impressive performance currently achieved by some deep RL algorithms often revolves around manually designing dense rewards [ 13 ] , complicated exploration strategies utilizing a significant amount of domain knowledge [ 23 ] , or operating in the known environment regime [ 88 , 69 ] . Motivated by current practical challenges and the gap between theory and practice , we propose a new algorithm for exploration by maximizing deviation from explored regions . This yields a practical algorithm with strong empirical performance . To be specific , we make the following contributions : 1 . Exploration via maximizing deviation Our approach is based on modifying the standard RL objective ( i.e . the cumulative reward ) by adding a regularizer that adaptively changes across iterations . The regularizer can be a general function depending on the state-action visitation density and previous state-action coverage . We show that the regularized objective naturally admits several common existing exploration methods . We then choose a particular regularizer that MAximizes the DEviation ( MADE ) of the next policy visitation d⇡ from the regions covered by prior policies ⇢kcov : Lk ( d ⇡ ) = J ( d⇡ ) + ⌧k X s , a q d ⇡ ( s , a ) ⇢kcov ( s , a ) . ( 1 ) Here , k is the iteration number , J ( d⇡ ) is the standard RL objective , and the regularizer encourages d ⇡ ( s , a ) to be large when ⇢kcov ( s , a ) is small . We give an algorithm for solving the regularized objective and prove that with access to an approximate planning oracle , it converges to the global optimum . We show that objective ( 1 ) results in an intrinsic reward that can be easily added to any RL algorithm to improve performance , as suggested by our empirical studies . Furthermore , the intrinsic reward applies a simple modification to the UCB-style bonus that considers prior visitation counts . This simple modification can also be added to existing bonuses in practice . 2 . Tabular studies In the special case of tabular parameterization , we show that MADE only applies some simple adjustments to the Hoeffding-style count-based bonus . We compare the performance of MADE to Hoeffding and Bernstein bonuses in three different RL algorithms , for the exploration task in the stochastic diabolical bidirectional lock [ 3 , 66 ] , which has sparse rewards and local optima . Our results show that MADE robustly improves over the Hoeffding bonus and is competitive to the Bernstein bonus , across all three RL algorithms . Interestingly , MADE bonus and exploration strategy appear to be very close to the Bernstein bonus , without computing or estimating variance , suggesting that MADE potentially captures some environmental structures . Additionally , we empirically show that MADE regularizer can improve the optimization rate in policy gradient methods . 3 . Experiments on MiniGrid and DeepMind Control Suite We empirically show that MADE works well when combined with model-free ( IMAPLA [ 25 ] , RAD [ 55 ] ) and model-based ( Dreamer [ 35 ] ) RL algorithms , greatly improving the sample efficiency over existing baselines . When tested in the procedurally-generated MiniGrid environments , MADE manages to converge with two to five times fewer samples compared to state-of-the-art method BeBold [ 111 ] . In DeepMind Control Suite [ 95 ] , we build upon the model-free method RAD [ 55 ] and the model-based method Dreamer [ 35 ] , improving the return up to 150 in 500K steps compared to baselines . Figure 1 shows normalized sample size to achieve maximum reward with respect to our algorithm . 2 Background . Markov decision processes . An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P , r , ⇢ , ) , where S is the state space , A is the action space , P : S ⇥ A 7 ! ( S ) is the transition kernel , r : S ⇥ A 7 ! [ 0 , 1 ] is the ( extrinsic ) reward function , ⇢ : S 7 ! ( S ) is the initial distribution , and 2 [ 0 , 1 ) is the discount factor . A stationary ( stochastic ) policy ⇡ 2 ( A | S ) specifies a distribution over actions in each state . Each policy ⇡ induces a visitation density over state-action pairs d⇡ : S ⇥ A 7 ! [ 0 , 1 ] defined as d⇡ ⇢ ( s , a ) : = ( 1 ) P1 t=0 t Pt ( st = s , at = a ; ⇡ ) , where Pt ( st = s , at = a ; ⇡ ) denotes ( s , a ) visitation probability at step t , starting at s0 ⇠ ⇢ ( · ) and following ⇡ . An important quantity is the value a policy ⇡ , which is the discounted sum of rewards V ⇡ ( s ) : = E [ P1 t=0 t rt | s0 = s , at ⇠ ⇡ ( · | st ) for all t 0 ] starting at state s 2 S . Policy mixture . For a sequence of policies Ck = ( ⇡1 , . . . , ⇡k ) with corresponding mixture distribution wk 2 k 1 , the policy mixture ⇡mix , k = ( Ck , wk ) is obtained by first sampling a policy from wk and then following that policy over subsequent steps [ 36 ] . The mixture policy induces a state-action visitation density according to d⇡mix ( s , a ) = P k i=1 w k i d ⇡i ( s , a ) . While the ⇡mix may not be stationary in general , there exists a stationary policy ⇡0 such that d⇡ 0 = d⇡mix Puterman [ 80 ] . Online reinforcement learning . Online RL is the problem of finding a policy with a maximum value from an unknown MDP , using samples collected during exploration . Oftentimes , the following objective is considered , which is a scalar summary of the performance of policy ⇡ : JM ( ⇡ ) : = Es⇠⇢ [ V ⇡ ( s ) ] = ( 1 ) 1 E ( s , a ) ⇠d⇡⇢ ( · , · ) [ r ( s , a ) ] = J ( d ⇡ ) . ( 2 ) We drop index M when it is clear from context . We denote an optimal policy by ⇡ ? 2 arg max ⇡ J ( ⇡ ) and use the shorthand V ? : = V ⇡ ? to denote the optimal value function . It is straightforward to check that J ( ⇡ ) can equivalently be represented by the expectation of the reward over the visitation measure of ⇡ . We slightly abuse the notation and sometimes write J ( d⇡ ) to denote the RL objective . 3 Adaptive regularization of the RL objective . 3.1 Regularization to guide exploration . In online RL , the agent faces a dilemma in each step : whether it should select a seemingly optimal policy ( exploit ) or it should explore different regions of the MDP . To allow flexibility in this choice and trade off between exploration and exploitation , we propose to add a regularizer to the standard RL objective that changes throughout iterations of an online RL algorithm : Lk ( d ⇡ ) = J ( d⇡ ) | { z } exploitation +⌧k R ( d ⇡ ; { d⇡i } k i=1 ) | { z } exploration . ( 3 ) Here , R ( d⇡ ; { d⇡i } k i=1 ) is a function of state-action visitation of ⇡ as well as the visitation of prior policies ⇡1 , . . . , ⇡k . The temperature parameter ⌧k determines the strength of regularization . Objective ( 3 ) is a population objective in the sense that it does not involve empirical estimations affected by the randomness in sample collection . In the following section , we give our particular choice of regularizer and discuss how this objective can describe some popular exploration bonuses . We then provide a convergence guarantee for the regularized objective in Section 3.2 . 3.2 Exploration via maximizing deviation from policy cover . We develop our exploration strategy MADE based on a simple intuition : maximizing the deviation from the explored regions , i.e . all states and actions visited by prior policies . We define policy cover at iteration k to be the density over regions explored by policies ⇡1 , . . . , ⇡k , i.e . ⇢kcov ( s , a ) : = 1 k P k i=1 d ⇡i ( s , a ) . We then design our regularizer to encourage d⇡ to be different from ⇢kcov : Rk ( d ⇡ ; { d⇡i } k i=1 ) = X s , a s d⇡ ( s , a ) ⇢kcov ( s , a ) . ( 4 ) It is easy to check that the maximizer of above function is d⇡ ( s , a ) / 1 ⇢kcov ( s , a ) . Our motivation behind this particular deviation is that it results in a simple modification of UCB bonus in tabular case . We now compute the reward yielded by the new objective . First , define a policy mixture ⇡mix , k with policy sequence ( ⇡1 , . . . , ⇡k ) and weights ( ( 1 ⌘ ) k 1 , ( 1 ⌘ ) k 2⌘ , ( 1 ⌘ ) k 3⌘ , , . . . , ⌘ ) for ⌘ > 0 . Let d⇡mix , k be the visitation density of ⇡mix , k . We compute the total reward at iteration k by taking the gradient of the new objective with respect to d⇡ at d⇡mix , k : rk ( s , a ) = ( 1 ) rdLk ( d ) d=d⇡mix , k = r ( s , a ) + ( 1 ) ⌧krdRk ( d ; { d⇡i } ki=1 ) d=d⇡mix , k , ( 5 ) which gives the following reward rk ( s , a ) = r ( s , a ) + ( 1 ) ⌧k/2p d⇡mix , k ( s , a ) ⇢kcov ( s , a ) . ( 6 ) The intrinsic reward is constructed based on two densities : ⇢kcov a uniform combination of past visitation densities and d̂⇡mix , k a weighted mixture of the past visitation densities . As we will discuss shortly , policy cover ⇢kcov ( s , a ) is related to the visitation count of ( s , a ) pair in previous iterations and resembles count-based bonuses [ 10 , 44 ] or their approximates such as RND [ 15 ] . Therefore , for an appropriate choice of ⌧k , MADE intrinsic reward decreases as the number of visitations increases . MADE intrinsic reward is also proportional to 1/ p d⇡mix , k ( s , a ) , which can be viewed as a correction applied to the count-based bonus . In effect , due to the decay of weights in ⇡mix , k , the above construction gives a higher reward to ( s , a ) pairs visited earlier . Experimental results suggest that this correction may alleviate major difficulties in sparse reward exploration , namely detachment and catastrophic forgetting , by encouraging the agent to revisit forgotten states and actions . Empirically , MADE ’ s intrinsic reward is computed based on estimates d̂⇡mix , k and ⇢̂kcov from data collected by iteration k. Furthermore , practically we consider a smoothed version of the above regularizer by adding > 0 to both numerator and denominator ; see Equation ( 7 ) . MADE intrinsic reward in tabular case . In tabular empirical setting , the empirical estimation of policy cover is simply ⇢̂kcov ( s , a ) = Nk ( s , a ) Nk , where Nk ( s , a ) is ( s , a ) pair ’ s visitation count and Nk is the total count , by iteration k. Setting ⌧k = 1/ p Nk , MADE simply modifies the Hoeffding-type bonus via the mixture density and is proportional to 1/ q d̂⇡mix , k ( s , a ) Nk ( s , a ) . Bernstein bonus is another tabular UCB bonus that modifies Hoeffding bonus via an empirical estimate of the value function variance . Bernstein bonus is shown to improve over Hoeffding count-only bonus by exploiting additional environment structure [ 106 ] and close the gap between algorithmic upper bounds and information-theoretic limits up to logarithmic factors [ 112 , 113 ] . However , a practical and efficient implementation of a bonus that exploits variance information in non-linear function approximation parameterization still remains an open question ; see Section 6 for further discussion . On the other hand , our proposed modification based on the mixture density can be easily and efficiently incorporated with non-linear parameterization . Deriving some popular bonuses from regularization . The regularization in ( 3 ) can describe some popular bonuses . Exploration bonuses that only depend on state-action visitation counts can be expressed in the form ( 3 ) by setting the regularizer a linear function of d⇡ and the exploration bonus ri ( s , a ) , i.e. , Rk ( d⇡ ; { d⇡i } ki=1 ) = P s , a d ⇡ ( s , a ) ri ( s , a ) . One can check that taking the gradient of the regularizer with respect to d⇡ recovers ri ( s , a ) . As another example , one can set the regularizer to Shannon entropy Rk ( d⇡ ; { d⇡i } ki=1 ) = P s , a d ⇡ ( s , a ) log d⇡ ( s , a ) , which gives the intrinsic reward log d⇡ ( s , a ) ( up to an additive constant ) and recovers the result in Zhang et al . [ 109 ] . Algorithm 1 Policy computation for adaptively regularized objective 1 : Inputs : Iteration count K , planning error ✏p , visitation density error ✏d . 2 : Initialize policy mixture ⇡mix,1 = with C1 = ( ⇡1 ) and w1 = ( 1 ) 3 : for k = 1 , . . . , K do 4 : Estimate the visitation density d̂⇡mix , k of ⇡mix , k via a visitation density oracle . 5 : Compute reward rk ( s , a ) = r ( s , a ) + ( 1 ) ⌧krdRk ( d ; { ⇡i } ki=1 ) d=d̂⇡mix , k . 6 : Run approximate planning on modified MDP Mk = ( S , A , P , rk , ) and return ⇡k+1 . 7 : Update policy mixture Ck+1 = ( Ck , ⇡k+1 ) and wk+1 = ( ( 1 ⌘ ) wk , ⌘ ) . 8 : Return : ⇡mix , K = ( Ck , wk ) .

Summary: The paper proposes an alternative exploration strategy by MAximizing the DEviation (MADE) of the next policy from explored regions of prior policies by adding an adaptive regularizer (function of state-action visitation counts) to the primary RL objective at each iteration to balance between exploration the new state space and exploit the visited states.  In addition, MADE has been theoretically proven to converge to the global optimum given access to an approximate planning oracle.  The authors show that MADE applies some simple adjustments to the Hoeffding-style count-based bonus. Moreover, experiments in three different RL algorithms (Value iteration, PPO, Q-Learning) show the superiority over the state-of-the-art UCB-bonus methods (Hoeffding and Bernstein bonus). In addition, the authors demonstrate that MADE can practically work both in Model-based (Dreamer) and Model-Free (IMPALA, RAD) RL algorithms, significantly improving sample-efficiency over baselines.