Experience Replay with Likelihood-free Importance Weights

1 INTRODUCTION . Deep reinforcement learning methods have achieved much success in a wide variety of domains ( Mnih et al. , 2016 ; Lillicrap et al. , 2015 ; Horgan et al. , 2018 ) . While on-policy methods ( Schulman et al. , 2017 ) are effective , using off-policy data often yields better sample efficiency ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) , which is critical when querying the environment is expensive and experiences are difficult to obtain . Experience replay ( Lin , 1992 ) is a popular paradigm in off-policy reinforcement learning , where experiences stored in a replay memory can be reused to perform additional updates . When applied to temporal difference ( TD ) learning of the Q-value function ( Mnih et al. , 2015 ) , the use of replay buffers avoids catastrophic forgetting of previous experiences and improves learning . Selecting experiences from the replay buffers using a prioritization strategy ( instead of uniformly ) can lead to large empirical improvements in terms of sample efficiency ( Hessel et al. , 2017 ) . Existing prioritization procedures rely on certain choices of importance sampling ; for instance , Prioritized Experience Replay ( PER ) selects experiences with high TD error more often , and then down-weight the experiences that are frequently sampled in order to become closer to uniform sampling over the experiences ( Schaul et al. , 2015 ) . However , this might not work well in actorcritic methods , where the goal is to learn the value function ( or Q-value function ) induced by the current policy , and following off-policy experiences might be harmful . In this case , it might be more beneficial to perform importance sampling that reflects on-policy experiences instead . Based on this intuition , we investigate a new prioritization strategy for actor-critic methods based on the likelihood ( i.e. , the frequency ) of experiences under the stationary distribution of the current policy ( Tsitsiklis et al. , 1997 ) . In actor-critic methods ( Konda & Tsitsiklis , 2000 ) , we can estimate the value function of a policy by minimizing the expected squared difference between the critic network and its target value over a replay buffer ; an appropriate replay buffer should properly reflect the discrepancy between critic value functions . We treat a discrepancy as “ proper ” if it preserves the contraction properties of the Bellman operator , and consider discrepancies measured by the expected squared distances under some state-action distribution . In Theorem 1 we prove that the stationary distribution of the current policy is the only distribution in which the Bellman operator is a contraction ( i.e . being “ proper ” ) ; this motivates the use of the stationary distribution as the underlying distribution for the replay buffer . Intuitively , optimizing the expected TD-error under the stationary distribution addresses the TD-learning issue in actor-critic methods , as the TD errors in high-frequency states are given more weight . To use replay buffers derived from the stationary distribution with existing deep reinforcement learning methods , we need to be mindful of the following bias-variance trade-off . We have fewer experiences from the current policy ( using which results in high variance ) , but more experiences from other policies under the same environment ( using which results in high bias ) . We propose to find appropriate bias-variance trade-offs by using importance sampling over the replay buffer , which requires an estimate of the density ratio between the stationary policy distribution and the replay buffer . Inspired by recent advances in inverse reinforcement learning ( Fu et al. , 2017 ) and off-policy policy evaluation ( Grover et al. , 2019 ) , we use a likelihood-free method to obtain an estimate of the density ratio from a classifier trained to distinguish different types of experiences . We consider a smaller , “ fast ” replay buffer that contains near on-policy experiences , and a larger , “ slow ” replay buffer that contains additional off-policy experiences , and estimate density ratios between the two buffers . We then use these estimated density ratios as importance weights over the Q-value function update objective . This encourages more updates over state-action pairs that are more likely under the stationary policy distribution of the current policy , i.e. , closer to the fast replay buffer . Our approach can be readily combined with existing approaches that learn value functions from replay buffers . We consider our approach over three competitive actor-critic methods , Soft Actor-Critic ( SAC , Haarnoja et al . ( 2018 ) ) , Twin Delayed Deep Deterministic policy gradient ( TD3 , Fujimoto et al . ( 2018 ) ) and Data-regularized Q ( DrQ , Kostrikov et al . ( 2020 ) ) . We demonstrate the effectiveness of our approach over on 11 environments from OpenAI gym ( Dhariwal et al. , 2017 ) and DeepMind Control Suite ( Tassa et al. , 2018 ) , where both low-dimensional state space and high-dimensional image space are considered ; this results in 45 method-task combinations in total . Notably , our approach outperforms the respective baselines in 35 out of the 45 cases , while being competitive in the remaining 10 cases . This demonstrates that our method can be applied as a simple plug-and-play approach to improve existing actor-critic methods . 2 PRELIMINARIES . The reinforcement learning problem can be described as finding a policy for a Markov decision process ( MDP ) defined as the following tuple ( S , A , P , r , γ , p0 ) , where S is the state space , A is the action space , P : S × A → P ( S ) is the transition kernel , r : S × A → R is the reward function , γ ∈ [ 0 , 1 ) is the discount factor and p0 ∈ P ( S ) is the initial state distribution . The goal is to learn a stationary policy π : S → P ( A ) that selects actions in A for each state s ∈ S , such that the policy maximizes the expected sum of rewards : J ( π ) : = Eπ [ ∑∞ t=0 γ tr ( st , at ) ] , where the expectation is over trajectories sampled from s0 ∼ p0 , at ∼ π ( ·|st ) , and st+1 ∼ P ( ·|st , at ) for t ≥ 0 . For a fixed policy , the MDP becomes a Markov chain , so we define the state-action distribution at timestep t : dπt ( s , a ) , and the the corresponding ( unnormalized ) stationary distribution over states and actions dπ ( s , a ) = ∑∞ t=0 γ tdπt ( s , a ) ( we assume this always exists for the policies we consider ) . We can then write J ( π ) = Edπ [ r ( s , a ) ] . For any stationary policy π , we define its corresponding stateaction value function as Qπ ( s , a ) : = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] , its corresponding value function as V π ( s ) : = Ea∼π ( ·|s ) [ Qπ ( s , a ) ] and the advantage function Aπ ( s , a ) = Qπ ( s , a ) −V π ( s ) . A large variety of actor-critic methods ( Konda & Tsitsiklis , 2000 ) have been developed in the context of deep reinforcement learning ( Silver et al. , 2014 ; Mnih et al. , 2016 ; Lillicrap et al. , 2015 ; Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) , where learning good approximations to the Q-function is critical to the success of any deep reinforcement learning method based on actor-critic paradigms . The Q-function can be learned via temporal difference ( TD ) learning ( Sutton , 1988 ) based on the Bellman equation Qπ ( s , a ) = BπQπ ( s , a ) ; where Bπ denotes the Bellman evaluation operator BπQ ( s , a ) : = r ( s , a ) + γEs′ , a′ [ Q ( s′ , a′ ) ] , ( 1 ) where in the expectation we sample the next step , s′ ∼ P ( ·|s , a ) and a′ ∼ π ( ·|s ) . Given some experience replay buffer D ( collected by navigating the same environment , but with unknown and potentially different policies ) , one could optimize the following loss for a Q-network : LQ ( θ ; D ) = E ( s , a ) ∼D [ ( Qθ ( s , a ) − B̂πQθ ( s , a ) ) 2 ] ( 2 ) which fits Qθ ( s , a ) to an estimate of the target value B̂π [ Qθ ] ( s , a ) 1 . In practice , the target values can be estimated either via on-policy experiences ( Sutton et al. , 1999 ) or via off-policy experiences ( Precup , 2000 ; Munos et al. , 2016 ) . Ideally , we can learn Qπ by optimizing the LQ ( θ ; D ) to zero with over-parametrized neural networks . However , instead of minimizing the loss LQ ( θ ; D ) directly , prioritization over the sampled replay bufferD could lead to stronger performance . For example , prioritized experience replay ( PER , ( Schaul et al. , 2015 ) ) is a heuristic that assigns higher weights to transitions with higher TD errors , and is applied successfully in deep Q-learning ( Hessel et al. , 2017 ) . 3 PRIORITIZED EXPERIENCE REPLAY BASED ON STATIONARY DISTRIBUTIONS . Assume that d , the distribution the replay buffer D is sampled from , is supported on the entire space S ×A , and that we have infinite samples from π ( so the Bellman target is unbiased ) . Let us define the TD-learning objective for Q with prioritization weights w : S ×A → R+ , under the sampling distribution d ∈ P ( S ×A ) : LQ ( θ ; d , w ) = Ed [ w ( s , a ) ( Qθ ( s , a ) − BπQθ ( s , a ) ) 2 ] ( 3 ) In practice , the expectation in LQ ( θ ; d , w ) can be estimated with Monte-Carlo methods , such as importance sampling , rejection sampling , or combinations of multiple methods ( such as in PER ( Schaul et al. , 2015 ) ) . Without loss of generality , we can treat the problem as optimizing the mean squared TD error under some priority distribution dw ∝ d · w , since : arg min θ LQ ( θ ; d , w ) = arg min θ LQ ( θ ; d w ) , ( 4 ) so one could treat prioritized experience replay for TD learning as selecting a favorable priority distribution dw ( under which the LQ loss is computed ) in order to improve some notion of performance . In this paper , we propose to use as priority distribution dw = dπ , where dπ is the stationary distribution of state-action pairs under the current policy π . This reflects the intuition that TDerrors in high-frequency state-action pairs are more problematic than in low-frequency ones , as they will negatively impact policy updates more severely . In the following subsection , we argue the importance of choosing dπ from the perspective of maintaining desirable contraction properties of the Bellman operators under more general norms . If we consider Euclidean norms weighted under some distribution dw ∈ P ( S × A ) , the usual γ-contraction argument for Bellman operators holds only for dw = dπ , and not for other distributions .

Summary: This paper is on an experience replay approach, as applied to deep RL methods, that uses a density ratio between on-policy and off-policy experiences as the prioritization weights. The objective is to find appropriate bias-variance trade-offs for importance sampling from the replay buffer. In particular, there's the bias issue from replay experiences of other policies, and the variance issue from the recent on-policy experiences.