Revisiting Prioritized Experience Replay: A Value Perspective

1 INTRODUCTION . Learning from important experiences prevails in nature . In rodent hippocampus , memories with higher importance , such as those associated with rewarding locations or large reward-prediction errors , are replayed more frequently ( Michon et al. , 2019 ; Roscow et al. , 2019 ; Salvetti et al. , 2014 ) . People who have more frequent replay of high-reward associated memories show better performance in memory tasks ( Gruber et al. , 2016 ; Schapiro et al. , 2018 ) . A normative theory suggests that prioritized memory access according to the utility of memory explains hippocampal replay across different memory tasks ( Mattar & Daw , 2018 ) . As accumulating new experiences is costly , utilizing valuable past experiences is a key for efficient learning ( Ólafsdóttir et al. , 2018 ) . Differentiating important experiences from unimportant ones also benefits reinforcement learning ( RL ) algorithms ( Katharopoulos & Fleuret , 2018 ) . Prioritized experience replay ( PER ) ( Schaul et al. , 2016 ) is an experience replay technique built on deep Q-network ( DQN ) ( Mnih et al. , 2015 ) , which weighs the importance of samples by their surprise , the magnitude of the temporal-difference ( TD ) error . As a result , experiences with larger surprise are sampled more frequently . PER significantly improves the learning efficiency of DQN , and has been adopted ( Hessel et al. , 2018 ; Horgan et al. , 2018 ; Kapturowski et al. , 2019 ) and extended ( Daley & Amato , 2019 ; Pan et al. , 2018 ; Schlegel et al. , 2019 ) by various deep RL algorithms . Surprise quantifies the unexpectedness of an experience to a learning agent , and biologically corresponds to the signal of reward prediction error in dopamine system ( Schultz et al. , 1997 ; Glimcher , 2011 ) , which directly shapes the memory of animal and human ( Lisman & Grace , 2005 ; McNamara et al. , 2014 ) . However , how surprise is related to the importance of experience in the context of RL is not well understood . We address this problem from an economic perspective , by linking surprise to value of experience in RL . The goal of RL agent is to maximize the expected cumulative reward , which is achieved through learning from experiences . For Q-learning , an update on an experience will lead to a more accurate prediction of the action-value or a better policy , which increases the expected cumulative reward the agent may get . We define the value of experience as the increase in the expected cumulative reward resulted from updating on the experience ( Mattar & Daw , 2018 ) . The value of experience quantifies the importance of experience from first principles : assuming that the agent is economically rational and has full information about the value of experience , it will choose the most valuable experience to update , which will yield the highest utility . As supplements , we derive two more value metrics , which corresponds to the evaluation improvement value and policy improvement value due to update on an experience . In this work , we mathematically show that these value metrics are upper-bounded by surprise for Q-learning . Therefore , surprise implicitly tracks the value of experience , and accounts for the importance of experience . We further extend our framework to maximum-entropy RL , which augments the reward with an entropy term to encourage exploration ( Haarnoja et al. , 2017 ) . We derive the lower and upper bounds of these value metrics for soft Q-learning , which are related to surprise and ” on-policyness ” of the experience . Experiments in Maze and CartPole support our theoretical results for both tabular and function approximation RL methods , showing that the derived bounds hold in practice . Moreover , we also show that experience replay using the upper bound as priority improves maximum-entropy RL ( i.e. , soft DQN ) in Atari games . 2 MOTIVATION . 2.1 Q-LEARNING AND EXPERIENCE REPLAY . We consider a Markov Decision Process ( MDP ) defined by a tuple { S , A , P , R , γ } , where S is a finite set of states , A is a finite set of actions , P is the transition function , R is the reward function , and γ ∈ [ 0 , 1 ] is the discount factor . A policy π of an agent assigns probability π ( a|s ) to each action a ∈ A given state s ∈ S. The goal is to learn an optimal policy that maximizes the expected discounted return starting from time step t , Gt = rt + γrt+1 + γ2rt+2 + ... = ∑∞ i=0 γ irt+i , where rt is the reward the agent receives at time step t. Value function vπ ( s ) is defined as the expected return starting from state s following policy π , and Q-function qπ ( s , a ) is the expected return on performing action a in state s and subsequently following policy π . According to Q-learning ( Watkins & Dayan , 1992 ) , the optimal policy can be learned through policy iteration : performing policy evaluation and policy improvement interactively and iteratively . For each policy evaluation , we update Q ( s , a ) , an estimate of qπ ( s , a ) , by Qnew ( s , a ) = Qold ( s , a ) + αTD ( s , a , r , s′ ) , where the TD error TD ( s , a , r , s′ ) = r + γmaxa′ Qold ( s′ , a′ ) − Qold ( s , a ) and α is the step-size parameter . Qnew and Qold denote the estimated Q-function before and after the update respectively . And for each policy improvement , we update the policy from πold to πnew according to the newly estimated Q-function , πnew = arg max a Qnew ( s , a ) . Standard Q-learning only uses each experience once before disregarded , which is sample inefficient and can be improved by experience replay technique ( Lin , 1992 ) . We denote the experience that the agent collected at time k by a 4-tuple ek = { sk , ak , rk , s′k } . According to experience replay , the experience ek is stored into the replay buffer and can be accessed multiple times during learning . 2.2 VALUE METRICS OF EXPERIENCE . To quantify the importance of experience , we derive three value metrics of experience . The utility of update on experience ek is defined as the value added to the cumulative discounted rewards starting from state sk , after updating on ek . Intuitively , choosing the most valuable experience for update will yield the highest utility to the agent . We denote such utility as the expected value of backup EVB ( ek ) ( Mattar & Daw , 2018 ) , EVB ( ek ) = vπnew ( sk ) − vπold ( sk ) = ∑ a πnew ( a|sk ) qπnew ( sk , a ) − ∑ a πold ( a|sk ) qπold ( sk , a ) ( 1 ) where πold , vπold and qπold are respectively the policy , value function and Q-function before the update , and πnew , vπnew , and qπnew are those after . As the update on experience ek consists of policy evaluation and policy improvement , the value of experience can further be separated to evaluation improvement value EIV ( ek ) and policy improvement value PIV ( ek ) by rewriting ( 1 ) : EVB ( ek ) = ∑ a [ πnew ( a|sk ) − πold ( a|sk ) ] qπnew ( sk , a ) ︸ ︷︷ ︸ PIV ( ek ) + ∑ a πold ( a|sk ) [ qπnew ( sk , a ) − qπold ( sk , a ) ] ︸ ︷︷ ︸ EIV ( ek ) , ( 2 ) where PIV ( ek ) measures the value improvements due to the change of the policy , and EIV ( ek ) captures those due to the change of evaluation . Thus , we have three metrics for the value of experience : EVB , PIV and EIV . 2.3 VALUE METRICS OF EXPERIENCE IN Q-LEARNING . For Q-learning , we use Q-function to estimate the true action-value function . A backup over an experience ek consists of policy evaluation with Bellman operator and greedy policy improvement . As the policy improvement is greedy , we can rewrite value metrics of experience to simpler forms . EVB can be written as follows from ( 1 ) , EVB ( ek ) = max a Qnew ( sk , a ) −max a Qold ( sk , a ) . ( 3 ) Note that EVB here is different from that in Mattar & Daw ( 2018 ) : in our case , EVB is derived from Q-learning ; while in their case , EVB is derived from Dyna , a model-based RL algorithm ( Sutton , 1990 ) . Similarly , from ( 2 ) , PIV can be written as PIV ( ek ) = max a Qnew ( sk , a ) −Qnew ( sk , aold ) , ( 4 ) where aold = arg maxaQold ( sk , a ) , and EIV can be written as EIV ( ek ) = Qnew ( sk , aold ) −Qold ( sk , aold ) . ( 5 ) 2.4 A MOTIVATING EXAMPLE . We illustrate the potential gain of value of experience in a “ Linear GridWorld ” environment ( Figure 1a ) . This environment contains N linearly-aligned grids and 4 actions ( north , south , east , west ) . The rewards are rare : 1 for entering the goal state and 0 elsewhere . The solution for this environment is always choosing east . We use this example to highlight the difference between prioritization strategies . Three agents perform Q-learning updates on the experiences drawn from the same replay buffer , which contains all the ( 4N ) experiences and associated rewards . The first agent replays the experiences uniformly at random , while the other two agents invoke the oracle to prioritize the experiences , which greedily select the experience with the highest surprise or EVB respectively . In order to learn the optimal policy , agents need to replay the experiences associated with action east in a reverse order . For the agent with random replay , the expected number of replays required is 4N2 ( Figure 1d ) . For the other two agents , prioritization significantly reduces the number of replays required : prioritization with surprise requires 4N replays , and prioritization with EVB only uses N replays , which is optimal ( Figure 1d ) . The main difference is that EVB only prioritizes the experiences that are associated with the optimal policy ( Figure 1c ) , while the surprise is sensitive to changes in the value function and will prioritize non-optimal experiences : for example , the agent may choose the experiences associated with south or north in the second update , which are not optimal but have the same surprise as the experience associated with east ( Figure 1b ) . Thus , EVB that directily quantifies the value of experience can serve as an optimal priority . 3 UPPER BOUNDS OF VALUE METRICS OF EXPERIENCE IN Q-LEARNING . PER ( Schaul et al. , 2016 ) greatly improves the learning efficiency of DQN . However , the underlying rationale is not well understood . Here , we prove that surprise is the upper bound of the value metrics in Q-learning . Theorem 3.1 . The three value metrics of experience ek in Q-learning ( |EVB| , |PIV| and |EIV| ) are bounded by α|TD ( sk , ak , rk , s′k ) | , where α is a step-size parameter . Proof . See Appendix A.1 . In Theorem 3.1 , we prove that |EVB| , |PIV| , and |EIV| are upper-bounded by the surprise ( scaled by the learning step-size ) in Q-learning . As surprise intrinsically tracks the evaluation and policy improvements , it can serve as an appropriate importance metric for past experiences . We will further study these relationship in experiments .

Summary: This work aimed to understand the prioritized experience replay, a widely used technique to improve learning efficiently for RL agents. The authors proposed three different value metrics to quantify the experience, and showed that they are upper bounded by the TD error (up to a constant). The extension to soft Q-learning was also presented. Finally, the authors showed in experiments that derived upper bounds hold in Maze and CartPole. They also demonstrated that a new variant based on the upper bound achieved better performance on a subset of Atari games.