Experience Replay with Likelihood-free Importance Weights
Authors: Samarth Sinha
Jiaming Song
Animesh Garg
Stefano Ermon
Venue:  (2021)
Keywords: Experience Replay
Off-Policy Optimization
Deep Reinforcement Learning

Abstract: The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy, and justify this with a contraction argument over the Bellman evaluation operator. The resulting TD objective encourages small approximation errors on the value function over frequently encountered states. To balance bias and variance in practice, we use a likelihood-free density ratio estimator between on-policy and off-policy experiences, and use the ratios as the prioritization weights. We apply the proposed approach empirically on three competitive methods, Soft Actor Critic (SAC), Twin Delayed Deep Deterministic policy gradient (TD3) and Data-regularized Q (DrQ), over 11 tasks from OpenAI gym and DeepMind control suite. We achieve superior sample complexity on 35 out of 45 method-task combinations compared to the best baseline and similar sample complexity on the remaining 10. 

Under review as a conference paper at ICLR 2021
EXPERIENCE REPLAY WITH LIKELIHOOD-FREE IMPORTANCE WEIGHTS
Anonymous authors Paper under double-bling review
ABSTRACT
The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, a key component in deep reinforcement learning. In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy, and justify this with a contraction argument over the Bellman evaluation operator. The resulting TD objective encour- ages small approximation errors on the value function over frequently encountered states. To balance bias and variance in practice, we use likelihood-frre density ratio estimator between on-policy and off-policy experiences, and use the ratios as three
the prioritization weights. We apply the proposed approach empirically competitive methods, Soft Actor Critic (SAC). Twin Delayed Deep Deterministic policy gradient (TD3) and Data-regularized Q (DrQ), over 11 tasks from OpenAl gym and DeepMind control suite. We achieve superior sample complexity on 35 out of 45 method-task combinations compared the best baseline and similar sample complexity on the remaining 10.
INTRODUCTION
Deep reinforcement learning methods have achieved much success in a wide variety of domains (Mnih al., 2016; Lillicrap et al., 2015; Horgan al., 2018). While on-policy methods (Schulman et al., 2017) are effective, using off-policy data often yields better sample efficiency (Haamoja et al., 2018; Fujimoto et al., 2018), which is critical when querying the environment is expensive and experiences are difficult to obtain. Experience replay (Lin, 1992) is a popular paradigm in off-policy reinforcement learning, where experiences stored in a replay memory can be reused to perform additional updates. When applied to temporal difference (TD) learning of the Q-value function (Mnih et al., 2015), the use of replay buffers avoids catastrophic forgetting of previous experiences and improves learning. Selecting experiences from the replay buffers using prioritization strategy (instead of uniformly) can lead to large empirical improvements in terms of sample efficiency (Hessel et al., 2017).
Existing prioritization procedures rely certain choices of importance sampling; for instance, Prioritized Experience Replay (PER) selects experiences with high TD error more often, and then down-weight the experiences that frequently sampled in order become closer uniform sampling over the experiences (Schaul et al., 2015). However, this might not work well in actor- critic methods, where the goal is learn the value function (or Q-value function) induced by the current policy, and following off-policy experiences might be harmful. In this case, it might be more beneficial to perform importance sampling that reflects on-policy experiences instead.
Based on this intuition, we investigate a new prioritization strategy for actor-critic methods based on the likelihood (i.e., the frequency) of experiences under the stationary distribution of the current policy (Tsitsiklis et al., 1997). In actor-critic methods (Konda & Tsitsiklis, 2000), we can estimate the value function of policy by minimizing the expected squared difference between the critic network and its target value over replay buffer; an appropriate replay buffer should properly reflect the discrepancy between critic value functions. We treat a discrepancy as "proper" it preserves the contraction properties of the Bellman operator, and consider discrepancies measured by the expected squared distances under some state-action distribution. In Theorem we prove that the stationary distribution of the current policy is the only distribution in which the Bellman operator is contraction (i.e. being "proper"); )); this motivates the use of the stationary distribution as the underlying distribution for the replay buffer. Intuitively, optimizing the expected TD-error under the stationary distribution

---

Under review as a conference paper at ICLR 2021
addresses the TD- TD-leamning issue in actor-critic methods, as the TD errors in high-frequency states given more weight.
To use replay buffers derived from the stationary distribution with existing deep reinforcement learning methods, we need to be mindful of the following bias-variance trade-off. We have fewer experiences from the current policy (using which results in high variance), but more experiences from other policies under the same environment (using which results in high bias). We propose to find appropriate bias-variance trade-off's by using importance sampling over the replay buffer, which requires an estimate of the density ratio between the stationary policy distribution and the replay buffer. Inspired by recent advances in inverse reinforcement learning (Fu al., 2017) and off-policy policy evaluation (Grover et al., 2019), we use likelihood frree method to obtain an estimate of the density ratio from a classifier trained to distinguish different types of experiences. We consider a smaller, "fast" replay buffer that contains near on-policy experiences, and larger, "slow" replay buffer that contains additional off-policy experiences, and estimate density ratios between the two buffers. We then use these estimated density ratios as importance weights over the Q-value function update objective. This encourages more updates over state-action pairs that are more likely under the stationary policy distribution of the current policy, i.e., closer to the fast replay buffer.
Our approach can be readily combined with existing approaches that learn value functions from replay buffers. We consider approach over three competitive actor-critic methods, Soft Actor- CritiC (SAC, Haarnoja et al. (2018),, Twin Delayed Deep Deterministic policy gradient (TD3, Fujimoto et all (2018)) and rata-regularized Q (DrQ. Kostrikov al. (2020)). We demonstrate the effectiveness of our approach over on 11 environments from OpenAl gym (Dhariwal et al., 2017) and DeepMind Control Suite (Tassa al., 2018), where both low-dimensional state space and high-dimensional image space are considered; this results in 45 method-task combinations in total. Notably, our approach outperforms the respective baselines in 35 out of the 45 cases, while being competitive in the remaining 10 cases. This demonstrates that our method can be applied as a simple long-and-pla approach to improve existing actor-critic methods.
2 PRELIMINARIES
The reinforcement learning problem can be described as finding a policy for Markov decision process (MDP) defined as the following tuple (S, A, P,r, Po). where is the state space, A is the action space, P: PSS S x A P(S) is the transition kernel, r S x A + R is the reward function, E [0, 1) is the discount factor and po E P(S) is the initial state distribution. The goal is to learn a stationary policy π S P(A) that selects actions in A for each state E S, such that the policy maximizes the expected sum of rewards: [(T) = E. IE= ( 4t), where the expectation is over trajectories sampled from 8p po. ag (-1x)), and 8t-1 P(-]se, az) for t ' 0.
For a fixed policy, the MDP becomes a Markov chain, so we define the state-action distribution at timestep t: de (s, a). and the the corresponding (unnormalized) stationary distribution over states and a) (we assume this always exists for the policies we consider). We Ex- (r(s,a) a)]. For any stationary policy ग, we define its corresponding state- (*(ssa) (s,a) := Ex (Ei= 7'r(s,, at) |s0 = s. ag = a), its corresponding value function as (T(s) == (+1)) [QT a)] and the advantage function A (s, a) = Q" a) (s)... A large variety of actor-critic methods (Konda & Tsitsiklis, 2000) have been developed in the context of deep reinforcement learning (Silver et al., 2014; Mnih al., 2016; Lillicrap et al., 2015; Haarnoja et al., 2018; Fujimoto et al., 2018), where learning good approximations to the Qffunction is critical to the success of any deep reinforcement learning method based on actor-critic paradigms.
actions (s, a) = > t= 7'd7
can then write (TT]
action value function as Q"
The Q-function can be learned via temporal difference (TD) learning (Sutton, 1988) based on the Bellman equation QT a) = Q=QQ (s, a): where f denotes the Bellman evaluation operator
BT (s) = rrsrrssss a) + yE. a (Q(ss, a')),
where in the expectation we sample the next step, s' ~ P(-|s, a) and a' ~ T-|ss). Given some experience replay buffer D (collected by navigating the same environment, but with unknown and potentially different policies), one could optimize the following loss for a Q-network:
LQ(0: D) = E(SA)~D [(Qo(s.0) (B00)BB0Qosso Q)))
2

---

Under review as a conference paper at ICLR 2021
which fits Qo a) to estimate of the target value [*o a)' In practice, the target values be estimated either via on-policy experiences (Sutton et al., 1999) or via off-cy experiences (Pre- cup, 2000; Munos et al., 2016).
Ideally, we can learn Q" by optimizing the (0: D) to zero with over peeapaaametrized neural networks. However, instead of minimizing the loss (o D) directly, prioritization over the sampled replay buffer D could lead to stronger performance. For example, prioritized experience replay (PER, (Schaul al., 2015)) is a heuristic that assigns higher weights transitions with higher TD errors, and is applied successfully in deep Q-learning (Hessel et al., 2017).
3. PRIORITIZED EXPERIENCE REPLAY BASED ON STATIONARY DISTRIBUTIONS
Assume that d, the distribution the replay buffer D is sampled from, is supported on the entire space x A, and that we have infinite samples from π (so the Bellman target is unbiased). Let us define the TD-learnin objective for Q with prioritization weights S A - R+, under the sampling distribution d E P(S x A):
(0 d, w) = Ed [w(s, a) B Qo(s,
In practice, the expectation in LQ( d, w) can be estimated with Monte-Cario methods, such as im- portance sampling, rejection sampling, or combinations of multiple methods (such as in PER (Schaul et al., 2015)). Without loss of generality, we can treat the problem optimizing the mean squared TD error under some priority distribution d" x d w, since:
arg min Lg(0; d, w) arg min Lo d);"),,
so one could treat prioritized experience replay for TD learning as selecting a favorable priority dis- tribution d (under which the LQ loss is computed) in order to improve some notion of performance.
In this paper, we propose to use as priority distribution d d where d" is the stationary distribution of state-action pairs under the current policy л. This reflects the intuition that TD- errors in high-frequency state-action pairs are more problematic than in low-frequency ones, as they will negatively impact policy updates more severely. In the following subsection, we argue the importance of choosing dt from the perspective of maintaining desirable contraction properties of the Bellman operators under more general norms. If we consider Euclidean norms weighted under some distribution d' E PSS x A), the usual y-contraction argument for Bellman operators holds only for dh d=, and not for other distributions.
3.1 POLICY-DEPENDENT NORMS FOR BELLMAN BACKUP
The convergence of Bellman updates relies on the fact that the Bellman evaluation operator 8 is y-cotrrcction with respect to the for norm, i.e. AQ, Q' E Q, where Q = [Q (S x A) + RR is the set of all possible Q functions:
1B" Q - BTQ QFQQBBQ''I
<11QQ= Q'1
While it is sufficient to show convergence results, the Loc norm reflects a distance over two Q functions under the worst possible state-action pair, and is independent of the current policy. If two Q functions are equal everywhere except for a large difference on a single state-action pair (3, â) that unlikely under d= the Lx distance between the two Q functions is large. In practice, however, this will have little effect over policy updates as it is unlikely for the current policy to sample (3, 2).
Since our goal with the TD updates is learn Q", a distance metric that is related to a more suitable one for comparing different Q functions, reflecting the intuition that errors in frequent state-action pairs are more costly than on infrequent ones. Let consider the following weighted & distance between Q functions,
Q - Q' 12 = ((s.a)a Q'(s, a)
(6)
We also do not take the gradient over the target, which the more conventional approach.
3

---

Under review as a conference paper at ICLR 2021
where d E P(S A) is distribution state-action pairs. This can be treated the la norm but measured over stationary distribution d as opposed to the Lebesgue measure. This is closely tied to the Lo objective since
(o(0: - -
In the following statements, we show that 1 is only contraction operator when under the ||-Ila- norm; this supports use of instead of other distributions for the Le objective, as it reflects more reasonable measurement of distance between Q-functions for policy x.
Lemma 1. For all E (0, the Bellmun operator 8 is 7-contraction with respect to the Il-lla norm if d d= holds almost everywhere, i.e.,
d = d a.e. - 115T BBQQIl S >lQ Q'Ila, AQ, Q' EQ
Proof. In Appendix B. On high-level, we apply Jensen's inequality to f(x) x²
Theorem 1. For all 7 E (0, 1), the Bellman operator B" a 7-contraction with respect to the Il-Ila norm if and only if d = d= holds almost everywhere, i.e.
d d=, a.e.
115*Q B" Q'llaa ~111 Q'Ila, VQ, E
Proof. In Appendix B. On a high-level, whenever d = dt does not hold over some non-empty open set, we can perturb a constant Q-value function over this set to contradict 7-contraction.
Theorem highlights the importance of using d' the || norm specifically for measuring the distance between Q-functions: if we use any distribution other than d" the Bellman operator is not guaranteed to be a 7-contraction under that distance, which leads to worse convergence rates.
3.2 TD LEARNING BASED ON dm
The ||-I|a- norm also captures our intuition that errors in high-freee gigh-frequency state-action pairs are more problematic than low-frequency ones, they are likely to have larger effect in policy learning, For example, for the actor-critic policy gradient with Qo:
(T)) Es V & log T (als)
(Qa(s, -s,,))assaa) to be worse than the update with ground truth Q". Moreover, the gradient descent update over the objective
a²) is large for gigh-freuuency. (s, a) tuples, then the policy update is likely
- 0 VoL = -E ((ooss, Ta Qo(s, Vo
corresponds to batch version of TD update. This places more emphasis on TD errors for state-action pairs that occur more frequently under the current policy.
To illustrate the validity of using d=,, we consider chain MDP example (Figure la, but with 5 states in total), where the agent takes two actions that progress the state on the left on the right. The agent receives final reward of at the fight-most state and rewards of at other states. The policy takes the right action at each state with probability p. and takes the left action for with probability 1 p. We initialize the Q-function from [0, uniformly at random and consider p = 0.8 and 0.2.
We compare three approaches to prioritization with TD updates: uniform over all state-action pairs, prioritization over TD error (as done in Schaul et al. (2015)), and prioritization with d" we include more details in Appendix C. We illustrate the 1|- distance between the learned Q-function and the ground truth Q-function in Figure lb; prioritization with dm outperforms both uniform sampling and prioritization with TD error in terms of speed of converging to the ground truth, especially at the initial iterations. When be smaller than 1, while TD error takes 182 steps on average; this means that prioritization with d= helpful when we have limited update budget.
0.8, d" only takes 120 steps average decrease the expected error to
4

---

Under review as a conference paper at ICLR 2021
p 0.8
0.2
Expected erear
Expectec entor
d4
40
0.4
C2
Patry
Uniomm
weror
80
81
32
0
0.0
100
150
50
100
150
d1
d1
Rerations
Rerations
(a) Illustration of the chain MDP.
(h) Estimation error (F.g- (QQ. - )²) for different prioritization methods, including uniform sampling (Uniform), sampling based on TD error (TD error), and sampling based on d' (Policy).
Figure 1: Simulation of TD updates with different prioritization methods.
4 LIKELIHOOD-FREE IMPORTANCE WEIGHTING OVER REPLAY BUFFERS
In practice, however, there are two challenges with regards to using LQ (f:; d²) as the objective. On the one hand, an accurate estimate of d! requires many on-policy samples from d" and interactions with the environment, which could increase the practical sample complexity; on the other hand, if instead off- aff-policy experiences from the replay buffer, it would be difficult estimate the importance ratio w(s, a) := d' (s, a)/d"(s, a) when the replay buffer D is mixture of trajectories from different policies. Therefore. likelihood-frre density ratio estimation methods that rely only on samples (c.g. from the replay buffer) rather than likelihoods are more general and well-suited for estimating the objective function L (0; d=) with a good bias-variance trade-off. An appropriate choice of importance weights allows us to balance bias (which comes from replay experiences of other policies) and variance (which comes from a small number of on-policy experiences).
In this paper, we use the variational representation of f-divergences (Csiszar, 1964) to estimate the density ratios. For any convex, seniconnimisss function f [0, oo) + R satisfying f(1) = the f-divergence between two probabilistic measures P.Q E P(X) (where
assume P < Q, i... P is absolutely continuous w.r.t. Q) is defined as: (PI()) = -x f (dP(x)/ dQ(x)) dQ(x). A general variational method can be used to estimate f-divergences given only samples from P and Q
Lemma 2 (Nguyen et al. (2008)), Assume that has first order derivatives f' at [0, +20). VP, Q e P(X) such that P & Q and w - IRt,
(P(())) 2 (rlf(((z)) - (f
(8)
where f denotes the convex conjugate and the equality is achieved when w dP/dQ
We can apply this approach to estimating the density ratio w(ss a) d (s,, ad (s. a) with samples from the replay buffer. These ratios are then multiplied to the Q-function updates to perform importance weighting. Specifically, we consider sampling from two types of replay buffers. One is the regular (slow) replay buffer, which contains a mixture of trajectories from different policies; the other is smaller (fast) replay buffer, which contains only small of trajectories from very recent policies. After each episode of environment interaction, we update both replay buffers with the new experiences; the distribution of the slow replay buffer changes more slowly due the larger size (hence the name "slow").
The slow replay buffer contains off-policy samples from dD whereas the fast replay buffer contains (approximately) on-policy samples from dl (assuming the buffer size is small enough). Therefore, the slow replay buffer has better coverage of transition dynamics of the environment while being less on-policy. Denoting the fast and slow replay buffers as Dj and D. respectively, we estimate the ratio d /dD via minimizing the following objective over the network uhy (z:) parametrized by (the outputs (s a. are forced to be non-negative via activation functions):
L (vb) = ED. [s* (S' (ww(s, a)))]
(s,,
From Lemma 2, we can recover estimate of the density ratio from the optimal wy by minimizing the The (x)) objective. To address the finite sample size issue, we apply seff-normalization (Cochran, 2007) to the importance weights over the slow replay buffer Dj with a hyperparameter T:
ws os
(s, = Ep. [w (s, (1/T)
(10)
5

---

Under review as a conference paper at ICLR 2021
The final objective for TD learning over Q is then
LQ(0; ) x [Q(0; D,, w+)) == B(s,a))D, (z) (
(11)
where the target QT is estimated via Monte Carlo samples. We keep the remainder of the algorithm, such as policy gradient and value network update (if available) unchanged, so this method can be adapted for different off-policy actor-critic algorithms,, utilizing their respective advantages. We observe that using the weights to correct the policy updates does not provide much marginal improvements, so we did not consider this for comparison. We describe a general procedure of our approach in Algorithm (Appendix A), where one can modify from some "base" actor-critic algorithm to implement our approach. These algorithm cover both stochastic and deterministic policies, as our method does not require likelihood estimates from the policy. We consider our divergences to be Pennen-Shannon wy can be treated as probabilistic classifier.
5 RELATED WORK
Experience replay (Lin, 1992) is a crucial component in deep reinforcement learning (Hessel et al., 2017; Andrychowicz et al., 2017; Schaul et al., 2015), where off-policy experiences are utilized to improve sample efficiency. These experiences can be utilized on policy updates (such as in actor- critic methods (Konda & Tsitsiklis, 2000; Wang et al., 2016)), on value updates (such as in deep Q-learning (Schaul et al., 2015)) or evaluating TD update targets (Precup, 2000; Precup et al., 2001). For value updates, there are two sources of randomness that could benefit from importance weights (prioritization). The first source is the evaluation of the TD learning target for longer traces such as TD(A): importance weights can be used to debias targets computed from off-policy trajectories (Precup, 2000; Munos et al., 2016; Espeholt et al., 2018; Schmitt al., 2019), similar to its role in policy learning. The second source is the sampling of state-action pairs where the values are updated (Schaul et al., 2015), which addressed in this paper.
Numerous techniques have achieved superior sample complexity through prioritization of replay buffers. In masel-based plamning. Prioritized Sweeping (Moore & Atkeson, 1993; Andre et al., 1998; van Seijen& & Sutton, 2013) selects the next state updates according to changes in value. Prioritized Experience Replay (PER, (Schaul et al., 2015)) emphasizes experiences with larger TD) errors and is critical to the success of sample efficient deep Q-learning (Hessel et al., 2017). Remember and Forget Experience Replay (ReF-ER, (Novati & Koumoutsakos, 2018)) removes the experiences if differs much from choices of the current policy; this encourages sampling on-policy behavior which is similar to what we propose. Differing from ReF-ER, we do not require knowledge of the policy distribution. Distribution Correction (DisCor, Kumar al. (2020)) suggests against using on-policy experiences, which seems to be in contrast to what we have promoted. However, their analysis is based on the Bellman optimality operator, which aims find the optimal Q-value function, while ours is based on the Bellman evaluation operator, which aims to find the Q-value function under the current policy; this could partially explain why DisCor did not achieve superior performance than the baseline approach on OpenAl gym tasks.
Likelihood free density ratio estimation have been adopted imitation learning Ho & Ermon (2016), inverse reinforcement learning (Fu et al., 2017) and model- based off off-policy policy evaluation (Grover et al., 2019). Different from these cases, we do not use the weights to estimate the advantage function or to reduce bias in reward estimation; our goal is to improve performance of TD learning with function approximation. Dual representations of /-divergences are also leveraged in reinforcement learning (Nachum et al., 2019; Nachum & Dai, 2020), but it is used over a regularizer that encourages exploration to be closer to off-policy experiences; the importance weights are added to the reward function when computing the Q-value function but do not affect the replay experiences otherwise.
6 EXPERIMENTS
We combine the proposed prioritization approach over three popular actor-critic algorithms, namely Soft-Actor Critic (SAC, Haarnoja et al. (2018)), Twin Delayed Deep Deterministic policy gradient (TD3, Fujimoto et al. (2018)) and Data-regularized Q (DrQ, Kostrikov et al. (2020)). We compare our method with alternative approaches to prioritization; these include uniform sampling over the replay buffer (adopted by the original SAC and TD3 methods) and prioritization experience replay

---

Under review as a conference paper at ICLR 2021
Table Results on OpenAI Gym environments when trained with 500k steps. ERE is only designed for SAC, so its results on TD3 are not available.
SAC based
SAC
+PER
+ERE
+LFIW
Ant-v2
3193 + 404
2764 287
3331 298
3579 + 260
HalfCheetah-v2
8325 + 408
8111 + 341
8631 + 189
9045 = 222
Hopper-v2
2645 310
2871 + 214
2512 + 301
3109 = 244
Humanoid-v2
2033 + 199
1459
208
2466
147
3189 + 231
Walker2d-v2
2914 + 189
3071 + 109
2990 + 217
3221 = 149
TD3 based
TD3
+PER
+ERE
+LFIW
Ant-v2
2663 = 372
2610 = 128
2990 = 178
HalfCheetah-v2
7527 + 438
7310 339
8567 + 491
N/A
Hopper-v2
1801 + 206
2019 + 109
1937 = 250
Walker2d-v2
1306 + 257
1241 + 122
2113 + 310
Table 2: Results of SAC and TD3 trained from states on the DeepMind Control environments with and without LFIW after 100k and 250k environment steps. The results show significant improvements when the agents is trained with LFIW. Results are reported over 5 random seeds. The maximum possible score for any environment 1,000.
Training steps
100k
250k
TD3 based
TD3
+PER
+LFIW
TD3
+PER
+LFIW
Finger, Spin
315 = 49
306 = 68
467 = 49
654 + 16
587 + 32
788 + 43
Cartpole, Swing
464 +: 50
457 + 30
567
28
750
21
702 + 21
801 + 39
Reacher, Easy
404 + 73
351 + 109
592
90
711
+ 21
709 + 56
892 + 45
Cheetah, Run
108 + 20
98 23
113
18
381
22
390 46
414 20
Walker, Walk
349 - 22
336 + 18
451
45
765
31
724
+ 40
811 20
Ball in Cup, Catch
278 = 29
213 - 11
418 + 44
664
+
21
687
+ 24
872 + 27
Table 3: Results of SAC and TD3 trained from states on the DeepMind Control environments with and without LFIW after 100k and 250k environment steps. The results show significant improvements when the agents is trained with L.IWW Results are reported over 5 random seeds. The maximum possible score for any environment 1,000.
100k environment steps
SAC based
SAC
+PER
+PER+LFIW
+LFIW
Finger, Spin
482 +: 34
486 + 18
503 +: 27
523 + 16
Cartpole, Swing
700 + 51
689 39
726 + 14
789 + 27
Reacher, Easy
750 = 68
704
806 55
861 29
Chectah, Run
498 + 108
367 667 123
502 109
541 + 89
Walker, Walk
187 = 89
31
321 - 29
333 1 12
Ball in Cup, Catch
888 + 13
834 + 23
892 + 8
890 + 6
250k environment steps
SAC based
SAC
+PER
+PER+LFIW
+LFIW
Finger, Spin
806 +
814 45
860 + 23
901 + 14
Cartpole, Swing
825 + 8
811 + 15
823 + 31
873 - 23
Reacher, Easy
945 +
931 11 11
944
+
Cheetah, Run
638 + 32
618 41
631 + 56
709 + 11
Walker, Walk
895 - 47
881 23
917 - 20
911 - 12
Ball in Cup, Catch
974 - 13
978 7
970 770
981 + 19
based on TD-error (Schaul et al., 2015). We choose 5 continuous control tasks from the OpenAl gym

---

Under review as a conference paper at ICLR 2021
Table Results for DrQ (Kostrikov et al., 2020) on the image- RL on the DeepMind Control Suite. LFIW is applied to state-of-the ofrt--rr- image-based based RL algorithm in DrQ, and we are able to see consistent improvement over the DM Control Suite Benchmark.
100k steps
DrQ
DrQ+LFIW
500k steps
DrQ
DrQ+LFIW
Finger, Spin
838 - 58
909 + 28
Finger, Spin
918 + 49
922 = 28
Cartpole, Swing
748 + 50
801 + 22
Cartpole, Swing
875 + 6
893 + 8
Reacher, Easy
573 + 67
743 + 89
Reacher, Easy
945 + 25
939 12
Cheetah, Run
387 = 45
444 + 38
Cheetah, Run
574 104
581 = 112
Walker, Walk
639 = 99
718 + 86
Walker, Walk
901 - 35
909 = 38
Ball in Cup, Catch
901 + 17
901 + 25
Ball in Cup, Catch
970 +
968 + 8
benchmark (Brockman et al., 2016) and 6 tasks from the DeepMind Control suite (DCS, Tassa et al. (2018)). We consider state representations in all tasks and pixel representations from DCS.
Our method introduces some additional hyperparameters compared the vanilla approaches, namely the temperature T, the size of the fast replay buffer [Dr] and the architecture for the density estimator wg. To ensure fair comparisons against the baselines, we the same hyperparameters as the original algorithms when it available. For all environments we use the following default hyperparameters for likelihood-free free importance weighting: T 5, [Pr] 10a, [Dal 10s. We use f from the Jensen Shannon divergence for better numerical stability. We include more experimental details in Appendix C.
6.1 EVALUATION
We use (+LFIW) to denote our likelihood-frie importance weighting method, (+PER) to denote prioritization with TD error (Schaul et al., 2015)* and (+ERE) denote Emphasizing Recent Experience (ERE, Wang & Ross (2019)) for SAC only. Table shows the results on OpenAl gym (500k steps), whereas Tables and shows the results DMCS with state (100k and 250k steps) and image representations (100k and 500k steps) respectively. These steps are chosen to demonstrate both initial training progress and approximate performance convergence.
OpenAI Gym results Table demonstrates that in terms of performance 500k steps, our LFIW method is able to outperform baseline methods on most tasks (except Hopper-v2 with TD3). On the other hand, PER and ERE do not perform very favorably against uniform sampling: similar phenomenon has been observed by (Novati & Koumoutsakos, 2018) for PER on other actor-critic algorithms, such as DDPG (Lillicrap et al., 2015) and PPO (Schulman et al., 2017). We also considered combining PER with LFIW, but achieved little initial success. We believe this is the because PER is designed for Q-lemnnin instead of actor-critic methods, where learning the max Q-function is the objective.
DCS results Table and shows the results with SAC and TD3 on state representations and DrQ on pixel representations. Again, we observe improvements over the baselines in most cases, and comparable performance others. Notably, we achieve much higher performance with LFIW at 100k training steps, which demonstrates that biasing the replay buffer towards on-policy experiences able to achieve good policy performance more quickly.
6.2 ADDITIONAL ANALYSES
To illustrate the advantage our method, we perform further analyses over the classification accuracy of we and the quality of the Q-function estimates over the Humanoid-v2 environment trained with SAC and SAC + LFIW. In Appendix C, we also include additional ablation studies over the hyperparameters introduced by LFIW, including temperature T', replay buffer size [Dyl and number of hidden units in Wg. We observe that SAC + LFIW is insensitive to these hyperparameter changes, which suggests that the same LFIW hyperparameters can be applied readily to other tasks.
We 0.6,8 0.4 in PER.
8

---

Under review as a conference paper at ICLR 2021
Accuracy of We We we to discriminate two types of experiences; experiences sampled from the policy trained with SAC for 5M steps are labeled positive, and the mixture of experiences sampled from policies trained for 1M to 4M steps are labeled negative. With the wh predictions, obtain precision of 87.3% and an accuracy of 73. 1 1%. This suggests that the importance weights tends to be higher for on-poliey data as desired, and the weights indeed allows the replay buffer be closer to on-policy experiences.
Quality of Qeestimates We compare the quality of the Qesstimates between SAC and SAC+LFIW, where we sample 20 trajectories from each policy, and obtain the "ground truth" via Monte Carlo estimates of the true Q-value. We then evaluate the learned Q-function estimates and compare their correlations with the ground truth values. For the SAC case, the Pearson and Spearman correlations are 0.41 and 0.11 respectively, whereas for SAC+LFIW method they are 0.74 and 0.42 (higher is better). This shows how our Q-function estimates are much more reflective of the "true" values, which explains the improvements in sample complexity and the performance of the learned policy.
7 CONCLUSION
In this paper, we propose a principled approach to prioritized experience replay for TD-llarning of Q-value functions in actor-critic methods, where we re-weigh the replay buffer to be closer to on-policy experiences. We justify this by showing theoretically that when measure the discrepancy between function with the expected squared differences under state-action distributions, only the on-policy distribution ensures that the Bellman evaluation operator is a contraction for the resulting discrepancy. To achieve a good bias-variance trade-off in practice, we assign weights to the replay buffer based on their estimated density ratios against the stationary distribution. These density ratios are estimated via samples from fast and slow replay buffers, which reflect on-policy and off-policy experiences respectively. Our methods can readily applied to deep reinforcement learning methods based on actor-critic approaches. Empirical results on SAC, TD3 as well as DrQ on 11 environments and 45 method-task combinations demonstrate that our method based on likelihood- free importance weighting is able to achieve superior sample complexity most cases compared other methods, so our approach can be applied as a plug-and-play method to improve actor-critic methods.
REFERENCES
David Andre, Nir Friedman, and Ronald Parr. Generalized prioritized sweeping. In M I Jordan, M J Kearns, and Solla (eds.), Advances in Neural Information Processing Systems 10, pp. 1001-1007. MIT Press, 1998.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pp. 5048-5058, 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
William G Cochran. Sampling techniques. John Wiley & Sons, 2007.
I Csiszar. Eine informationstheorrttschhe ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. OpenAl baselines, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed Deep-RL with importance weighted Actor-Learner architectures. arXiv February 2018.
preprint arXiv: 00t..0002.0..10
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 1710. 1248, October 2017.

---

Under review as a conference paper at ICLR 2021
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in Actor-Critic methods. arXiv preprint arXiv: February 2018.
Aditya Grover, Jiaming Song, Alckh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using Likelihood-Free importance June 2019.
weighting. arXiv preprint
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- Policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint /811v:1801.012900 January 2018.
Mattco Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv: 1710.0228, October 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint 1803:000830 March 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv::412.690 December 2014.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, pp. 1008-1014, 2000.
llya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. arXiv preprint arXiv:2003.07335, 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /509. 02971, 2015.
Long-Ji Lin. Self- lmproving reactive agents based on reinforcement learning, planning and teaching. Machine learning. 8(3):293-3211 May 1992. ISSN 0885-6125, 1573-0565. doi: 10.1023/A: 1022628806385.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53) 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928-1937, 2016.
Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13(1):103-130, 1993.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G Bellemare. Safe and efficient Off- Policy reinforcement learning. arXiv preprint arXiv:1606 (eee:6006002777 June 2016.
Ofir Nachum and Bo Dai. Reinforcement learning via arXiv:2001.01866, January 2020.
duality. arXiv preprint
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv: 1912.0004, December 2019.
Xuanlong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. arXiv preprint arXiv:0809.0853, (1 (1)::5477-581,, September 2008. doi: 10.1 (09/TTT.2010.20688
10

---

Under review as a conference paper at ICLR 2021
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. arXiv preprint arXiv:1807.0582, July 2018.
D Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty, 2000.
Doina Precup, Richard $ Sutton, and Sanjoy Dasgupta. Off-policy temporal-differece learning with function approximation. In ICML, pp. 417-424, 2001.
Tom Schaul, John Quan, Ioamnis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.0594 November 2015.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-Policy Actor-Critic with shared experience replay. arXiv preprint arXiv: 1909. 1583, September 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06337, July 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International Conference on Machine Learning, pp. 387-395, January 2014.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):81-211, 1999. ISSN 0004-3702.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 2018.
John N Tsitsiklis, Member, and Benjamin Van Roy. An analysis of Temporal-Differnnce learning with function approximation. IEEE transactions on automatic control, 42(5), 1997. ISSN 0018-9286.
Harm van Seijen and Richard S Sutton. Planning by prioritized sweeping with small backups. arXiv
preprint 130iv:1301.22. 23433, January 2013.
Che Wang and Keith Ross. Boosting soft Actor-Critic: Emphasizing recent experience without forgetting the past. arXiv preprint 1r0.v:1906.04000 June 2019.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient Actor-Critic with experience replay, arXiv preprint arXiv:1611.01224. November 2016.

---

Under review as a conference paper at ICLR 2021
A ALGORITHM
Algorithm 1 Actor Critic with Likelihood-free Importance Weighted Experience Replay
1: repeat
2: for each environment step do
3: gather new transition tuples (s,a,r,*
4: update a, 8, 6') to D3 (slow replay buffer) and Dr (fast replay buffer)
5: end for
6: remove stale experiences in D.. Dr (Drl < (
7: if [D&ll exceeds some threshold then
8:
obtain samples from D, and Dr
9:
update wy with loss function (v)) (Eq. 9) assign Wyy according to Eq. 10
10:
else
12:
U.e = (no re-weighting)
end if
14:
obtain estimates for Bd B²Qo Qo with base algorithm
update Qo with loss function (0; D,, ú) (Eq. 11)
16:
update na and value network (if available) with base algorithm
17:
until Stopping criterion
18:
return Qo,
B PROOFS
Lemma 1. For all y E 1), the Bellman operator 13T is "ycontraction with respect to the ll norm if d = d= holds almost everywhere, i.e.,
d= d= a.e. = - s11Q Q'lla, VQ, Q' e
Proof. From the definitions of ||-I and 8", we have:
||8T - B5Q
(12)
[ (Qss' YE.r (Q'(d,)')))
- (E, ((6''a')) - ('(s',a']])
S 7 E, (Q(6'a)'
(13)
= (Q((s.) (s.
(14)
= Q'Ila 113
(15)
where 8' ~ a(. ~ (-11s) and
d'ss' a') Σ a)(d')s) 8,a
represents the state-action distribution of the next step when the current distribution is d. We use Jensen's inequality over the convex function (.)2 in Eq. 13. Since d= is the stationary distribution, ==d' = d" a.e., so the direction holds.
Theorem 1. For all 7 E (0. 1), the Bellman operator B" is y-contraction with respect to the ll- norm if and only if d = d holds almost everywhere, i.e..
d = d= a.e. + I BTQ lla S r11 - Q'lla, Illa,QQ. VQ, Q' e
Proof. The "if" case is apparent from the Lemma, only need to prove the "only if" case.
12

---

Under review as a conference paper at ICLR 2021
For the "only if" case, construct counter-example for all d * d'. Without loss of generality, assume Vs, a E S x A, (x) = 0. The following functionals of Q
h(Q) = Q - Q' ((Q) ||Q = (
(a) (Bs (Q(s', a')))
corresponds the quantities at the two ends of the contraction argument. Our goal is find some Q E @ such that ((Q) -{(() > O, which would complete the contradiction. We can evaluate the functional derivatives for fi(Q) and g(Q):
dh (s' a') Σ d(s, dls.aa dQ
dg dQ
(8.0) a) ==(===d(ss))) ({)(,a))))))))))))))))))))))))))
where E(s, a) = the current step at a). Now us consider some Qo such that for some constant q ys, a E S x A, Qo(s, a)
[Q( (s d")] is the expected Q function of the next step when
Let us then evaluate both functional derivatives at Qo:
dh
2
Σ d(s ()
2d 2d'(s', d
dQ
Qo
dg
(8,
2d( 8 a)
aQ
where {(s, a) = under the current Qo- Because d' and d are not equal almost everywhere (from the assumption of d not being the stationary distribution), there must exist some non-empty open set E where ( (s, a) - -d(s, a)) ds da > 0. We can then add function e S x A - such that E(s, a) = - I[(ss,), a) E r)) where is the indicator function and v is an infinitesimal amount. Now let us evaluate (h g) at (Qo + +):
(h (Qo +
dh dg
(h g)Qu= +
c
dQ
dQ
= =(q² - 9²) 29 / (s, d(s, a(s.a) o(v) > 0
Therefore, the proposed function (Q + 6) is the contradiction we need.
C ADDITIONAL EXPERIMENTAL DETAILS
C.1 SETUP ON CHAIN MDP
The chain MDP considered has deterministic transitions, so we make the policy stochastic make sure that stationary distribution exists. In each epoch over all the state-action pairs, we use the following TD learning update over tabular data to simulate the effect of weighting with fixed learning rate
(s, + Q(s, + (1 (1-(1 (1- (BF (BFC)))))B) a) Q(s, a))
(16)
where wis,, a) the weight (uniform, TD eπrоr or d' which simulates the number of TD updates with learning rate 77. The weights are normalized to have mean value of 1 which makes number of updates per epoch the same across different methods.
C.2 ABLATION STUDIES
To demonstrate the stability of our method across different hyperparameters, we conduct further analyses over the key hyperparameters, including the temperature in Eq. 10, the size of the fast replay buffer [Dr), and the number of hidden units in the classifier model We. We consider running the SAC+LFIW method on the Walker-v2 environment with 1000 episodes using all the default hyperparameters unless explicitly changed.
13

---

Under review as a conference paper at ICLR 2021
Table 5: Additional hyperparameters for SAC (Haarnoja al., 2018)
Parameter
Value
optimizer
Adam Kingma & Ba (2014)
learning rate discount
3 x 10
0.99
number of samples per minibatch nonlinearity
256
ReLU 10
target smoothing coefficient
Table 6: Additional hyperparameters for TD3 (Fujimoto al., 2018)
Parameter
Value
optimizer
Adam Kingma & Ba (2014)
learning rate discount
10-3
0.99
number of samples per minibatch nonlinearity exploration policy
256
ReLU
N (0,, 1)
Temperature The temperature T affects variances of the weights assigned; larger makes the weights more similar to each other, while smaller 7 relies more the outputs of the classifier. Since we are using finite replay buffers, using a larger temperature reduces the chances of negatively impacting performance due wy overfitting the data. We consider T 1, 2.5, 5, 7.5, 10 in Figure 2a; all cases have similar sample efficiencies except for T 1. Similarly, we also perform a similar analysis Humanoid-v2 with SAC in Figure 3. We observe similar dependency significantly worse that for the other hyperparameters
T as in Walker
where the sample efficiency with 7 considered, which shows that overfitting the data can easily be avoided by using a higher temperature value even in nigher-dimensional state-action distributions.
Replay buffer sizes [D;] The replay buffer sizes 'Dr affects the amount of experiences we treat as "on-polic" "on-policy". Larger [Drl reduces the risk of overfitting while increasing the chances of including more off-policy data. We consider 'Dil = 1000, 10000, 50000, 100000, corresponding to to 100 episodes. We note that [Da] = 106, so even for the largest D,, D. is significantly larger. The performance relatively stable despite a small drop for DD = 100000.
Hidden units of wy The number of hidden units each layer affects the expressiveness of the neural network. While networks with more hidden units are more expressive, they are easier to overfit the replay buffers. We consider hidden layers with 128, 256 and 512 neurons respectively. While the smaller network with 128 units is able to achieve superior performance initially, the other configurations are able catch up around 1000 episodes.
14

---

Under review as a conference paper at ICLR 2021
Table 7: Additional hyperparameters for DrQ (Kostrikov et al., 2020)
Parameter
Value
optimizer
Adam Kingma & Ba (2014)
learning rate
10 10-3
discount
0.99
critic target update frequeney actor log stddev bounds
2
[-10, 2]
actor update frequency
2
number of samples per minibatch nonlinearity image dimensions
512
ReLU
84 x 84
3500
3500
3000
Epicode Rewarts
2500
2500
2000
2000
1500
1000
2.5
1000
1000
19000
500
500
200
400
600
800
1000
200
400
600
800
1000
Aipcodee Number
Eptsode Number
(a) Temperature
(h) Fast replay buffer size [De]
3000
TAM!
Rewards
2500
2000
1500
128
500
200
400
600
800
1000
Number
(c) Hidden layer size of
Figure Hyperparameter sensitivity analyses on Walker2d-v2 with SAC.
8000
7000
Episode Rewards
6000
5000
4000
3000
2000
10
1000
2000
4000
6000
8000
10000
Episode Number
Figure 3: Temperature sensitivity on Humanoid-v2 with SAC
15


Reviews:
[{"review_id": "w77RHYBMvEE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose likelihood-free importance weights applied to\nexperience replay buffers in reinforcement learning. When the sampling\ndistribution is not the on-policy distribution the RL agent may learn\na biased estimate. This could happen in a number of ways, but two\nobvious ones are data being generated off-policy or prioritized\nexperience replay (PER). In both cases the sampling distribution is\nnot the one we wish to minimize over. In PER this is partly fixed by\nthe importance sampling correction. This work replaces this\ncorrection, in the off-policy setting, with a learned ratio.\n\nAlthough exposition and some of the early details about related work\nand background seem problematic, the experimental results are strong\nand encouraging. The fundamental idea is broadly applicable, although\nsome of the related work touches on related ideas this particular\napplication of the idea appears novel and potentially impactful.\n\n-- Pros:\n\nBroadly useful technique for LFIW, and specific application using the\ntwo replay buffers is quite nice.\nRelatively strong empirical results\n\n-- Cons:\n\nIssues with the background / related work (see comments below)\nExperiments could be presented and discussed more clearly\n\n-- Comments:\n\"However, such a heuristic could be highly sub-optimal in actor-critic\nmethods...\"\nThis sentence and the rest of the paragraph seem to be missing\nsomething essential. Prioritized experience replay is an importance\nsampling technique. That is, while sampling under a biased\ndistribution, they use importance weights to correct for this sampling\nbias (only partly in the case of PER in practice). As well, it is not\njust actor-critic methods that require the expected loss to be under a\nparticular distribution over states.\n\nThis misunderstanding reappears just before section 3, where the\nauthors discuss PER as a heuristic for improved performance by\nchanging the expectation in Equation 2. This is, indeed, incorrect. In\nterms of Equation 2, we could accurately represent PER by saying that\nD is already reweighted by the priorities (which is how the authors\ncurrently interpret it), and further adding a reweighting where inside\nthe expectation we divide by something like p_D(s, a), to convert the\nexpectation into one over the uniform distribution. The most accurate\nway obviously would be the seperate these two out explicity: a\nsampling distribution (data generated by a behavior policy and put\ninto replay) and an importance sampling distribution (the prioritized\ndistribution), and multiplying inside the expectation by their ratio.\n\nIt is strange that these errors would crop up considering the primary\ncontribution of the work is about essentially doing this without\nexplicitly computing this ratio of likelihoods. The lack of correction\nre-appears in equation 3 as well\n\nFinally, this comes full circle just before section 3.1 with the\nstatement that the 'prioritized' distribution needs to be d^\\pi. Of\ncourse it has to be d^\\pi when you do not correctly correct for the\nsampling distribution.\nOverall, I don't want to attribute this to an actual\n'misunderstanding' by the authors, I think section 4 clearly shows the\ndepth of their understanding. So instead, I think of this as an issue\nwith the writing and how things are explained. However, how PER is\ncommunicated here does make me fairly wary about the empirical results\nfor PER.\n\nRegarding Theorem 1, one direction of this has been known for quite a\nlong while. That is, the contraction holds when the distribution is\nd^\\pi. So, it might be better to spend more time emphasizing the other\ndirection and discussing its proof.\n\n\"In actor-critic methods, the critic objective typically estimates the\nQ-value function\"\nArguably the most typical estimate for actor-critic would be the value\nfunction (V) not the action-value function (Q). Not that AC methods\ndon't ever use Q-values, but this doesn't seem like the typical case.\n\n\n-- Minor suggestions / Typos:\n\nPage 2, just above equation 1: \"based on Bellman equation Q\", missing a \"the\"\n\nFootnote 1 on page 3 should probably come earlier, during\npreliminaries as it is not specific to PER.\n\nPage 5, last paragraph, \"We observe that using the weights to correct\nthe policy updates does not demonstrate provide much...\" (remove\neither 'demonstrate' or 'provide')\n\nI struggled to find where the authors specify the f-divergence they use.\n\n-- Questions:\n\nDid you consider using your technique combined with prioritized\nreplay? This would mean changing the 'slow' replay buffer\nprobabilities to be prioritized probabilities, but otherwise would be\nidentical to what is currently shown in the paper. More specifically,\nalthough the authors compare against PER as though this is an\nalternative to PER, it seems more fitting to think of this as an\nalternative to the IS correction being used by PER (among many other\nmethods).\n\nThe authors report the accuracy of w_\\psi used as a discriminator,\nwhich is great because I was going to ask about this exactly before I\ngot to that part of the results. However, I wonder about a few\nadditional details here. Have you looked at how this\nprecision/accuracy varies over training? In particularly, you might\nexpect that it becomes nearly impossible to discriminate as the policy\nconvergences and thus the two distributions slowly become identical.\n\nAlso, how robust is performance to degradations in this accuracy? This\nseems like something you might have looked at. If so, do you have any\ninsights? The reported accuracy is low enough, with empirical\nperformance strong enough, that I suspect it *is* quite robust.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "b26Ui1Epo6S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is on an experience replay approach, as applied to deep RL methods, that uses a density ratio between on-policy and off-policy experiences as the prioritization weights. The objective is to find appropriate bias-variance trade-offs for importance sampling from the replay buffer. In particular, there's the bias issue from replay experiences of other policies, and the variance issue from the recent on-policy experiences. \n\nIt's not entirely clear to the reviewer about the necessity of maintaining two replay buffers (slow and fast). Instead of maintaining two replay buffers, it seems one can simply retain the standard single-buffer strategy, and evaluate how likely the experience is with respect to the current policy. Then the likelihood can be used as the weight for prioritized experience replay. This simple strategy also takes the bias-variance trade-offs coming from on-policy and off-policy experiences. The reviewer is curious about the advantage of the developed approach that uses two buffers over what's described above. \n\nIn the paper, the slow buffer is considered for maintaining off-policy experience, and the fast buffer is for on-policy experience. Accordingly, the sizes of those two buffers are supposed to be very important parameters. For instance, if the two buffers have similar sizes, then the developed approach are expected to function like standard deep RL (on policy or off policy depending on the buffer size). However, Figure 2 (b) shows that the performance is not sensitive to the size of the fast (on-policy) buffer. The result is counter-intuitive, though it's clear that the results are supposed to show the insensitivity to such parameters. \n\nExperiments were conducted by combining the experience replay approach (called LFIW, likelihood-free importance weighting) with three existing deep actor-critic methods, and then comparing the combinations with their originals. The results look good, and demonstrate the effectiveness of the developed approach. It's unclear why the results were presented in tables instead of curves (minor point), which can be potentially more better for readability. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EIqg7wKvdeU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a generally applicable modification to experience sampling in the context of actor-critic algorithms using a Q function as a critic. The modification is called \"Likelihood-free Importance Weights\" (LFIW). The authors describe the approach in Appendix A in the form of pseudocode.  Comparing to a generic actor-critic algorithm, the changes include the keeping of two replay buffers (\"fast\" and \"slow\") and inclusion of an additional re-weighting function w which in turn is used in the update of the Q function. The paper includes a thorough performance comparison on MuJoCo and DM Control Suite. \n\nThe results are good, but the authors seem to use a weak implementation of SAC. For comparison, I am referring to SAC as implemented in https://github.com/tensorflow/agents/blob/v0.6.0/tf_agents/agents/sac/sac_agent.py#L62-L634  Using this implementation I am getting, e.g. for the Humanoid-v2 environment,  at 500K steps results above 4000, compared to 3189 (SAC+LFIW) and 2033 (SAC) reported in the paper. Hence it is hard for me to assess whether LFIW offers a real improvement of the SOTA or perhaps fixes some problems of the underlying implementation of SAC.\n\nThe mathematical analysis contained in Theorem 1 is interesting, but in my opinion, it is written confusingly. It would be better to decompose it into two separate statements:\n- the first statement stating the inequality with a simple proof based on the convexity of the square function,\n- the second statement proposing a counter-example in the form of Q+epsilon for appropriately small epsilon. \n\nAlso, the statement of the theorem is slightly weaker than I would like: can we just prove, that the counterexample exists regardless of gamma? The current statement says that the mapping is not a gamma-contraction, but one can imagine, that relaxing gamma would still lead to a contractive mapping.\n\nThe example presented in Figure 1 is interesting, though in my opinion, somewhat detached from the main focus of the paper which in my opinion is the analysis of Algorithm 1. Also, the three-state MDP may seem too simple to conclude about the performance of the sampling method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B9cWGRtsTD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a method to address the distribution mismatch problem in off-policy learning. Its method is to estimate the density ratio between the stationary distribution under the target policy and the data distribution in the replay buffer, and then correct the state distribution with the ratio so that it can use experience from the replay as if they were generated from the target policy.\n\nAlthough I have many questions about the paper, the following two are most important currently and are hoped to be addressed first.\n\nIt seems not clear to me whether this paper considers the tabular case or the function approximation case. It wrote in section 5 that \"our goal is to improve performance of TD learning with function approximation\", but its preliminaries (Section 2), its example (Figure 1), and its main theoretical result (Theorem 1) are all for the tabular case. In fact, the state distribution mismatch only matters when multiple states share the same parameter. For the tabular case, as each state has its own parameter, there is no interference between them, and thus there is no need to make the state distribution on-policy.\n\nEmpirically, all the experiments are designed to solve continuous control problems, where both state and action are continuous. However, the density ratio d^\\pi(s, a) / d^D(s, a) is only defined under the discrete setting. In fact, because both state and action spaces are continuous, all elements in the replay buffers are different from others and this ratio makes no sense. Therefore this algorithm doesn't seem appropriate to me for continuous control tasks. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new experience replay approach, applicable to deep RL methods. Two reviewers suggested acceptance and two did rejection. The first negative reviewer R1 raised a concern on continuous vs. discrete issue, but AC thinks that the authors' response is not fully convincing enough.  The second negative reviewer R2 pointed out that the reported performance of SAC is poor compared to the existing implementation (although authors claim a different set of hyperparameters is used), which AC thinks is a critical weakness to judge the value of the experiments. Two other positive reviewers (even R4) shows mixed opinions. Overall, AC thinks this is a borderline paper, a bit toward rejection.
Decision: Reject