Regret Minimization Experience Replay in Off-Policy Reinforcement Learning

Abstract: In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed. In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous prioritized sampling algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.


Reviews and author responses:

The paper presents a new experience sampling method by formulating the problem as regret minimization. They try to minimize the gap between the policy given all the data so far and the optimal policy. The paper provides a solution to such a regret minimization problem. Based on the solution, the paper claims that the optimal prioritization strategy indicates that we should pay more attention to experiences with higher hindsight TD error, better on-policiness, and more accurate Q value. Then the paper proposes methods to implement such sampling methods and present experiments in both continuous and discrete control domains to check their methods’ performances. 

Strength: 
1. The presentation is reasonably clear.
2. The problem formulation is novel to me. 
3. The theory provides some insight into sampling experiences in RL.
4. The experiments contain both challenging continuous and discrete control domains.   Originality. It seems novel to me. 

Clarity. The presentation is reasonably clear. 

Significance. The topic itself is important. However, the result is not convincing, as I pointed out in the quality part below. 

Quality. See concerns below. 

1. Section 3.1. I would not consider PER and DisCor are two representative algorithms. In some sense, PER also leads to a more accurate Q value faster. Please see the missing citation later. In fact, one can definitely argue that a regular ER is also trying to learn more accurate Q values faster. Without ER, one may use a single experience to update the network/value function, which is definitely slower. 

Further, L101-L104 is simply not justified. Using a target different from maximizing the expected return is common, from policy-based methods to value-based methods. L103, “for example, when the Bellman target is inaccurate, … ” For any stochastic optimization algorithm, there is no guarantee that each single updating step can lead to an improvement in the reduction of the loss function. Hence, such an example is meaningless. 

The motivating example in Fig 1 is invalid because of both lack of details and the design itself.  

a. Why compare PER to uniform? The latter has access to all the states, and hence I guess it also has access to the model?  Does PER have access to all the states and the model? 
If not, the comparison is unfair and pointless. First, PER does not have the same information as Uniform. Second, Uniform across state space is usually not doable in continuous domains. Are you trying to claim uniform sampling is what you want to design? 
If yes, PER should learn really quickly, as it should learn s_0 to s_3’s values really quickly, and when at s_0, it would realize that the right action has a larger value. 
b. On such a simple domain, the learning rate should be finely swept rather than fixed as a constant 0.1. 
c. The domain should be more useful to demonstrate the exploration strategy. What is the exploration strategy used?  

2. The theoretical result is not rigorous. 

First, the formulation of minimizing regret is problematic. According to objective (1),  the optimal Q_k should simply be B^\ast Q_{k-1}, then the constraint is satisfied. Or do you mean it should not be a constraint; it should also be part of the objective? Furthermore, are there any constraints on the magnitude of the weights w_k? I would be surprised if there is no such constraint. 

How can a theorem be informal? Would you please provide a clear answer? The assumption and important technical lemmas should be moved to the main body. 

Further, I believe assumption 1 in the appendix is invalid. First, what is H? It is an undefined notation. How large could it be? Do you concern about the finite horizon setting? If so, it is different from the background section in the main body. 

And, such an assumption is counter-intuitive. It says once a state-action pair gets visited, the chance of going back to it is low. It seems contrary to the traditional ergodic/recurrency assumption. The paper also says, “the assumption is trivial in continuous spaces”; however, the theorem is designed for the tabular case. 

The result of theorem 1 requires access to Q^\ast. How can this be true?

3. The proposed approach. The proposed approach to approximate the terms in Theorem 1 is not convincing. First, approximating |Q_k - Q^\ast| is difficult. Does it require learning an environment model? Second, it is hard to believe an algorithm that needs so many approximations can work well. The algorithm should be very fragile. This can also be seen in the experiments as I discuss below. 

4. What are the relationships between ReMERT and ReMERN? Which one should be preferred in what cases? I cannot see a clear message from the paper. Is ReMERT the ultimate algorithm you propose? And ReMERN is just an intuitive baseline? 

5. The experiments. Why compare with SAC? Aren’t your methods generic to be used with many algorithms? (As indicated from L53-54: “Similar to PER, ReMERN, and ReMERT can be a plug-in module to all off-policy RL algorithms 54 with a replay buffer”) I expect the comparison should be DDPG-PER, DDPG-ReMERN, DDPG-REMERT, etc.… Anyway, PER should also be included. 

In addition to PER, an intuitive baseline is to use PER with both long-term and short-term buffers, according to the paper’s theoretical result. 

The variance is too high to distinguish different algorithms. It is not clear how the parameter sweep is done for all these algorithms. ReMERN/T should have many more parameters to tune. 

6. Important citation missing: Beyond Prioritized Replay: Sampling States in Model-Based Reinforcement Learning via Simulated Priorities. The work discusses PER in detail and points out its limitations and solutions. 

Minors: 

1. L26, “For example, minimizing TD error is different from the objective of RL, which is to maximize the expected return of the policy.” This statement is not accurate. First, Q learning does find the optimal policy in the tabular case. And the statement seems to reject all value-based methods. Second, RL has both policy evaluation and control problems. The former does not seek to maximize expected returns. It just evaluates a policy. 

2. The notations do not differentiate random variables and realized r.v. 

3. L72, the Bellman optimal operator is incorrect. It seems r(s, a) is an r.v., and it should be inside the expectation. The notation of function f can be replaced by Q. 

4. some references are messy. Editors are mixed with publication venue.  Please see the review section. 

 Thanks a lot for the authors' detailed response. I may have misunderstood things the first time I read this paper. 

I adjusted my score based on both the author's response and other reviews. The main concerns are: 
1. the theoretical contribution seems not adequate
2. the rigorousness of the experiments

 I thank the authors for their response.

 Thank you for your thoughtful comments. We provide discussions and explanations on your concerns as follows.

**Q: Why is optimizing regret a better objective than optimizing value error? Why is the resulting weight different from DisCor?**

A: Lower value error may not imply a better policy. For example, action $a_1$ and $a_2$ can be taken in state $s$, and we have Q function $Q_1$, $Q_2$, and the optimal Q function $Q^*$, as illustrated in the table.

||$(s,a_1)$ |$(s,a_2)$  |
|------|:-----:|:------:|
|  $Q_1$ |     1    |    2      |           
|  $Q_2$ |     10   |    0      |       
|  $Q^*$ |     2    |    1      |   

It can be seen that $Q_1$ has lower value error than $Q_2$, but the policy induced by $Q_2$ is better.  
    
DisCor approximates the distribution $d^\pi(s,a)$ with uniform distribution. We handle this term more carefully and obtain an extra term regarding on-policy reweight. 

**Q: I think the authors should (1) mention that the origin of Eqn (1) is from DisCor and (2) discuss the differences and/or show why these changes are desirable.**

A: (1) We respectfully disagree with your argument. The motivation behind Eq. (1) is entirely different from DisCor. DisCor focuses on "lack if corrective feedback issue" while we focus on the objective mismatch. Based on the different motivation, we derive the optimization problem as shown in Eq. (1), which has a different formulation from DisCor, except the equation $Q_k=\arg\min_Q (Q-\mathcal{B}^\ast Q_{k-1})^2$. However, such a constraint is from fitted Q iteration rather than DisCor. (2) Our motivating example and the results of experiments demonstrate why these changes are desirable. We will add more concrete discussion in the revised paper. 

**Q: Eq. (3) is identical to the solution of DisCor, where DisCor's $\lambda$ is the density ratio.**

A: Eq. 3 is different from the solution of DisCor. DisCor doesn't have the term $d^{\pi_k}(s,a)$. In DisCor, $\lambda$ is the Lagrange multiplier.  In order to calculate $p_k/\mu$, DisCor uses $q_k$ as a projection of $p_k$ but still close to $\mu$ under the metric of KL divergence. But our method don't need to do such an approximation.


**Q: Figure 1 is used to describe the issues with PER and DisCor. But the explanation doesn't really explain why DisCor is suboptimal. Furthermore, the authors should compare their proposed approach in this same setting to demonstrate that it solves this suboptimality.**

A: To get the optimal policy, we indeed need to focus on the left action. However, when the Q value for the left action is accurate enough, attention on the left action may not help. In this stage, we need to add higher prioritization weight to the right action, and DisCor fails to do this.

We did comparative experiments on ReMERN and other baseline methods. The empirical results are shown in the following table.
    
|        Timestep      | 50 | 100 | 150 | 200 | 250 | 300 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | 
|   Reward (No weight) | 2  |  2  |  2  |  2  |  2  |  5  |
| TD error (No weight) | 10 | 5.7 | 3.3 | 1.9 | 1.1 | 0.7 |
|   Reward (PER)       | 2  |  2  |  2  |  2  |  2  |  4  |
| TD error (PER)       | 10 | 5.2 | 3.1 | 1.8 | 1.0 | 0.6 |
|   Reward (DisCor)    | 2  |  2  |  2  |  2  |  2  |  5  |
| TD error (DisCor)    | 10 | 5.3 | 3.0 | 1.9 | 1.2 | 0.8 | 
|   Reward (ReMERN)    | 2  |  2  |  2  |  2  |  5  |  5  |
| TD error (ReMERN)    | 10 | 6.6 | 5.4 | 4.5 | 3.4 | 3.1 |


    
As shown by the results, ReMERN can learn the optimal policy with fewer environment interactions than baseline methods like Q learning without prioritization weights, PER and DisCor. We can also provide intuitive explanations on the efficiency of ReMERN. As the reviewer commented, ReMERN will first upweight transitions that go left. However, once the Q value of going right surpasses that of going left, the on-policy correction takes place and upweights the transitions going right. For PER and DisCor, transitions that go right have small TD-error and large $|Q_k-Q^*|$, so they will both downweight these transitions, resulting in a slower training process.


**Q: The limitations of ReMERT: One novel aspect of the paper is ReMERT which considers the estimation of value error by distance to the terminal state. However, this solution doesn't seem very general as its reliant on the horizon and the existence of terminal states. This approach appears to have some small benefits in MuJoCo but isn't even applicable in meta-world where these algorithms seem the most useful.**
    

A: The constraint is not very strict. ReMERT can be applied to not only MuJoCo tasks but also some Atari games, as shown in the experiments. Besides its benefits in applicable environments, it has lower computational complexity. Comparisons of the computation time required for SAC, DisCor, PER, ReMERN and ReMERT are shown in the following table. Experiments are conducted with one NVIDIA 2080Ti GPU on the Ant-2d environments. 

|         Algorithm        |  SAC  |  PER  | DisCor | ReMERN | ReMERT |
| --------- | --------- | ----------- | ----------- | ----------- | ------------ | 
|    Time/h for 1M step    |  4.88 |  6.73  |  6.87  |  8.13  |  6.45  |

For the inapplicable environments, such as the Meta World Suite, we can use ReMERN as an alternative.  We will give a more clear explanation on the limitation of ReMERT in the conclusion of the revised paper.


**Q: "Our analysis gives a theoretical explanation for some prioritization methods, including PER, LFIW and DisCor" (line 325). I don't think this statement is true at all based on the contents of the paper.**

A: We kindly inquiry about any specific disadvantages in our paper that make this statement being "not true at all". There are terms similar to, if not the same as, heuristics used in PER, LFIW and DisCor in the solution to our optimization problem. Therefore, we find these heuristics can play important roles in the regret minimization of the policy.

**Q: The limitations and the role of approximations to the optimal strategy.**

A: From the construction of TCE, ReMERT is suitable for episodic MDP with terminal states. The approximation of ReMERT is accurate if the distance to end of a state is relatively fixed. This can be satisfied when the environment has limited randomness. We will give a more clear explanation on the limitations in the Conclusion section.

The role of approximations for TCE is discussed in appendix. It shows the approximation of TCE is accurate. 

**Q: No baselines which actually use prioritized sampling anywhere in the paper. PER outperforms ReMERT on 4/5 of the Atari environments.**

A: First, [Kumar et al., 2020] and [Sinha et al., 2020] have already shown in their experiments that PER performs worse than SAC in continuous control tasks like MuJoCo and Meta World, so we do not include PER as a baseline in these tasks. We do include PER as baseline in Atari benchmarks and show the results in Appendix D. In fact, empirical results show that ReMERT outperforms PER on 8/13 of Atari environments. Second, DisCor can be regarded as an approach of prioritized sampling, and we include DisCor as our baseline in continuous control tasks.

**Q: Does ReMERT compare favorably against DisCor in Atari environments? The authors should also include ReMERN results.**

A: We do not include DisCor because we can not reproduce the empirical results in the DisCor paper in the Atari environments. ReMERN is based on DisCor and is not included either.

**Q: Lack of code is disappointing for reproducibility.**

A: We will open source the project when the response period finishes.

**References**

[Kumar et al., 2020] Kumar, A., Gupta, A., and Levine, S. (2020). Discor: Corrective feedback in reinforcement learning via distribution correction.

[Sinha et al., 2020] Sinha, S., Song, J., Garg, A., and Ermon, S. (2020). Experience replay with likelihood-free importance weights.

 Thank you for your thoughtful and inspiring comments. We provide discussions and explanations about your concerns as follows.

**Q: It is quite obvious that it's better to update the value from the finish line backward. Moving from Theorem 2 to Eq. (11) includes several approximations and bounds, so I get the feeling that not much of the theory propagates to the actual algorithm.**

A: Though intuitively it is better to update the value function from the finish line backward, we can not determine the exact weight assigned to each state-action pairs, while our theory can provide numerical guidance. As to the approximations we make, we verify the existence of stable temporal structure in Appendix D.1, and show that the approximation is tight. In fact, approximations and bounds are necessary. We have to relax or ignore some factors irrelevant to prioritization weights, or 
quantities which are computationally expensive to obtain during training.

**Q: The claim that TCE is an approximation of $|Q_k - Q^{\ast}|$ is problematic on its own since if this is the case why not just use it to calculate $Q^{\ast}$.**

A:  If we use TCE to compute $Q^{\ast}$, the approximation error will be propagated directly to the policy and make it very unstable. This is disastrous since unstable policy can negatively impact the quality of the collected data, and in turn the policies afterwards. In contrast, prioritization weights are more robust to approximation error, because a batch of data is sampled at each update and the variance can be reduced. Also, the errors can only indirectly influence the policy afterwards.
% Second, even if we know $|Q_k-Q^{\ast}|$, we don't know whether $Q_k$ is overestimated or underestimated.

**Q: There seem to be some kind of equivalence between TCE and reward shaping with distance to goal.**

A: Distance to goal is only applicable to goal-based RL tasks, which is a subset of tasks that TCE can handle. Also, reward shaping requires extensive human effort and has limited theoretical guarantee.

**Q: ReMERT is but another example of how to use prior knowledge on the environment.**

A: It is true that using distance-to-end to estimate $|Q_k-Q^\ast|$ is based on prior knowledge, but it has formal and theoretical justifications on its validity. We can infer when ReMERT will fail from the theory, and provide guidance on the choice of algorithms.

**Q: ReMERT is dedicated towards a specific set of problems with finite horizons.**

A: The set of MDPs with finite horizons can cover many of the problems in the real world. We also provide ReMERN for tasks with infinite horizons.

 **Q: The variance is too high to distinguish different algorithms. It is not clear how the parameter sweep is done for all these algorithms. ReMERN/T should have many more parameters to tune.** 

A: Experiments on Meta-World and Atari show significant improvements. Previous prioritization methods like PER and DisCor struggle to outperform SAC in the MuJoCo benchmark, but we have made reasonable empirical improvements in MuJoCo compared with these methods. Experiments on Meta-World and Atari are even more significant. We only need to tune one or two hyperparameters when using the algorithms. 

**Q: "For example, minimizing TD error is different from the objective of RL, which is to maximize the expected return of the policy." This statement is not accurate. First, Q learning does find the optimal policy in the tabular case. And the statement seems to reject all value-based methods. Second, RL has both policy evaluation and control problems.** 

A: The objective of minimizing TD error is specifically designed for prioritization strategies. This doesn't contradict to the fact that we can find the optimal policy by minimizing TD error in training stage. TD error minimization is a good method for training in RL, but we can improve it by using prioritization strategy. Second, we will revise "the objective of RL" to "the objective of policy improvement in RL". 

**Q: The notations do not differentiate random variables and realized r.v.. The Bellman optimal operator is incorrect. It seems r(s, a) is an r.v., and it should be inside the expectation. The notation of function f can be replaced by Q.** 

A: Thanks for your suggestions on improving the clarity of paper. We will revise the notations.

 Thank you for your comments. We provide discussions and explanations about your concerns as follows.

**Q: I would not consider PER and DisCor are two representative algorithms. In fact, one can definitely argue that a regular ER is also trying to learn more accurate Q values faster. Without ER, one may use a single experience to update the network/value function, which is definitely slower.**

A: Here we focus on the algorithms that aims to improve performance by sampling each data point in the ER with different prioritization. Despite its potential mistakes, PER is still the most widely used algorithms and it inspires many prioritized methods based on TD error. PER has also been chosen as a representative prioritization algorithm in many previous researches, e.g., [Kumar et al., 2020], [Sinha et al., 2020] and [Li et al., 2021]. DisCor, on the other hand, consider this problem from "lack of corrective feedback issue". We select DisCor as representative algorithm because of its new perspective of viewing the prioritization. 

We would like to include [Jincheng et al., 2020] in the related work section as an existing analysis on prioritization methods.

**Q: L101-L104 is simply not justified. Using a target different from maximizing the expected return is common, from policy-based methods to value-based methods.**

A: The motivating example is especially proposed to justify L101-L104, as shown by the results. The training process of Q learning is indeed slowed down by PER and DisCor, whose criteria of prioritization is different from regret minimization. Also, being common does not mean being optimal. PER is a common approach for prioritized sampling, but [Kumar et al., 2020] and [Sinha et al., 2020] have shown that it has poor performance in tasks other than Atari.


**Q: For any stochastic optimization algorithm, there is no guarantee that each single updating step can lead to an improvement in the reduction of the loss function. Hence, such an example is meaningless.**

A: Though single updating step does not necessarily reduce the loss, in the expectation the loss reduction is guaranteed for any effective algorithm. Our motivating example is designed to show the difference of objectives between current prioritization and policy improvement, not to study the process of stochastic optimization. 

**Q: The motivating example in Fig 1 is invalid because of both lack of details and the design itself.**

A: We describe details of the example as follows. The MDP example in section 3.1 is specially designed to show the suboptimality of previous heuristics for prioritization. Specifically, the estimated Q values are stored in a table and prioritized Q learning is used to update the value estimations of each state-action pair, as described by the equation $Q_k(s,a)=Q_{k-1}(s,a)+\alpha w(s,a)[r(s,a) + \gamma\mathbb{E}\_{s'}\max_{a'}Q_{k-1}(s', a') - Q_{k-1}(s,a)]$, where $\alpha$ is the learning rate and $w$ is the prioritization weight. However, with unconstrained learning, accurate Q values can be obtained in few iterations, which is not the case in reality with function approximations. To simulate the approximation error in complex environments, we initialize all Q table entries with $0$ and set the learning rate $alpha$ (or step size) as 0.1. The reward scale is set to be around 1. The Q value of certain state-action pairs can be more aggressively updated by assigning larger prioritization weight.

In fact, RL algorithms and prioritization strategies is applicable to the designed MDP. That is, this is the problem these algorithms need to solve. Their failure demonstrates their shortage. Furthermore, there are plenty of papers involving similar motivating examples for illustration, e.g. [Kumar et al., 2020] [Li et al., 2021] [Sutton et al., 2015].


**Q: Why compare PER to uniform? The latter has access to all the states, and hence I guess it also has access to the model? Uniform across state space is usually not doable in continuous domains**

A: We perform Q iteration in this experiments, so both algorithms have access to all the states. Q iteration will certainly involve the transition probability of the MDP.  Also, uniform means no prioritization, and PER is built on it.


**Q: Are you trying to claim uniform sampling is what you want to design? What is the exploration strategy used?**

A: No. Uniform sampling is the baseline. This example demonstrates that prioritization based on heuristics can lead to worse performance in certain environments. Exploration is unnecessary with Q iteration on all state-action pairs.

**Q:  The formulation of minimizing regret is problematic. The optimal $Q_k$ should simply be $B^\ast Q_{k-1}$, then the constraint is satisfied. Or do you mean it should not be a constraint; it should also be part of the objective? Furthermore, are there any constraints on the magnitude of the weights $w_k$?**

A:  $Q_k=\mathcal{B}^\ast Q_{k-1}$ can not provide optimal $\pi_k$ for the objective (1), as clarified in [Kumar et al., 2020]. If $|Q_{k-1}-Q^*|$ is large, $|B^\ast Q_{k-1}-Q^{\ast}|$ will be larger, resulting in a suboptimal policy. Our theory does not involve constraints on the magnitude of $w_k$, but in practice we add clip to it to reduce variance from estimation error.   

**Q: I believe assumption 1 in the appendix is invalid. What is H? How large could it be? Do you concern about the finite horizon setting?**

A: $H$ in assumption 1 is a large constant, such that $\gamma^H$ is negligible. We consider the infinite horizon setting. 

**Q: Assumption 1 is counter-intuitive. It says once a state-action pair gets visited, the chance of going back to it is low. It seems contrary to the traditional ergodic/recurrency assumption. The paper also says, “the assumption is trivial in continuous spaces”; however, the theorem is designed for the tabular case.**

A: Our assumption is about the probability of an agent going back to the visited state within $H$ steps, while the ergodic/recurrency assumption says that we can reach a certain state almost surely without constraining the recurrent time. For non-recurrent MDPs, assumption 1 holds obviously. It also holds when recurrent time is large. MDP with large recurrent time is common. Also, this theorem can be generalized to continuous action spaces, where the assumption holds.
     
**Q: The $Q^{\ast}$ in theorem 1 and the informal statement.**

A: The objective of optimization problem in theorem 1 is to minimize the difference between the return of the current policy and the optimal policy, while the optimal policy is determined by $Q^{\ast}$. So $Q^\ast$ appears in theorem 1.

Informal form of a theorem is common when the theorem is complex. Informal means we omit the exact statement of conditions in the theorem, because they are less intuitive and less important. For theorem 1, we omit assumption 1 which states once a state is visited, it has low probability for the agent to go back to this in $H$ steps, where $H$ is a large constant. For theorem 2, we also omit assumption on the independence and some terms that tends to zero. 


**Q:  First, approximating $|Q_k - Q^{\ast}|$ is difficult. Does it require learning an environment model? Second, it is hard to believe an algorithm that needs so many approximations can work well.**

A: First, we don't need an environment model to approximate $|Q_k-Q^*|$. We have proposed two methods to approximate $|Q_k-Q^{\ast}|$. One uses an error network. Eq. (6) shows the update rule of neural network. It is similar to Bellman equation if $|Q_k-\mathcal{B}^*Q_{k-1}|$ was substituted with the reward. The other uses the distance of a state to terminal state to approximate $Q_k-Q^{\ast}$, as shown in Eq. (11).

Second, the validity of the approximation has been verified. Experiments on $\Delta_k$ has been done in [Kumar et al., 2020], and the experiments on TCE can be found in Appendix D.1.


**Q: What are the relationships between ReMERT and ReMERN? Which one should be preferred in what cases? Is ReMERT the ultimate algorithm you propose? And ReMERN is just an intuitive baseline?**

A: They are both practical algorithms that compute prioritization weights based on theorem 1. Their difference is that ReMERN use neural networks to estimate a specific term in the theory, while ReMERT exploits a common temporal structure in MDPs. ReMERT is preferred in environments with stable temporal structure, while ReMERN is a more general algorithm. They are both the algorithms that we propose.


**Q: Why compare with SAC? Aren’t your methods generic to be used with many algorithms? PER should also be included.**

A: We choose SAC because it is a SOTA algorithm in continuous control tasks. [Kumar et al., 2020] and [Sinha et al., 2020] have already shown in their experiments that PER performs worse than SAC in continuous control tasks like MuJoCo and Meta World, so we do not include PER as a baseline in these tasks. We do include PER as baseline in Atari benchmarks and show the results in Appendix D. 



**Reference:**

[Kumar et al., 2020] Kumar, A., Gupta, A., and Levine, S. (2020). Discor: Corrective feedback in reinforcement learning via distribution correction.

[Jincheng et al., 2020] Jincheng, M., Yangchen, P. (2020). Beyond Prioritized Replay: Sampling States in Model-Based Reinforcement Learning via Simulated Priorities. 

[Sinha et al., 2020] Sinha, S., Song, J., Garg, A., and Ermon, S. (2020). Experience replay with likelihood-free importance weights.

[Li et al., 2021] Li, A. A., Lu, Z., and Miao, C. (2021). Revisiting prioritized experience replay: A value perspective.

[Sutton et al., 2015] Sutton, R. S., Mahmood, A. R., and White, M. (2015). An emphatic approach to
the problem of off-policy temporal-difference learning. 

 
Thank you for your thoughtful and inspiring comments. We provide explanations to your concerns as follows.

**Q: The novelty of the paper is somewhat on the lower end, and the proposed ReMERN method only adds an additional on-policy correction to the prior method (DisCor).**

A: The novelty of this paper can be summarized into three aspects. First, it points out the issue of objective mismatch in prior methods, and proposes a direct objective for prioritization strategies. Second, we make several theoretical improvements despite that our optimization problem has a similar form to DisCor. The relationship between $w_k$ and $d^{\pi_k}$ is complex. DisCor ignores this by relaxing $d^{\pi_k}$ to uniform distribution, while we keep $d^{\pi_k}$ throughout our derivation.  Third, we propose ReMERT by using a new method to estimate $|Q_k-Q^*|$. The experiment results verify our claim that a more direct objective for prioritization strategy is better and provide a promising direction for algorithm design in the future. 

**Q: What would the proposed method ReMERN do in the example in Figure 1(a)? Is the proposed method failing in the same mode as DisCor in this example?**

A: We conduct comparative experiments on ReMERN and other baseline methods. The empirical results are listed in the following table.
    
| Timestep | 50 | 100 | 150 | 200 | 250 | 300 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- |
|   Reward (No weight) | 2  |  2  |  2  |  2  |  2  |  5  |
| TD error (No weight) | 10 | 5.7 | 3.3 | 1.9 | 1.1 | 0.7 |
|   Reward (PER)       | 2  |  2  |  2  |  2  |  2  |  4  |
| TD error (PER)       | 10 | 5.2 | 3.1 | 1.8 | 1.0 | 0.6 |
|   Reward (DisCor)    | 2  |  2  |  2  |  2  |  2  |  5  |
| TD error (DisCor)    | 10 | 5.3 | 3.0 | 1.9 | 1.2 | 0.8 | 
|   Reward (ReMERN)    | 2  |  2  |  2  |  2  |  5  |  5  |
| TD error (ReMERN)    | 10 | 6.6 | 5.4 | 4.5 | 3.4 | 3.1 |


    
As shown by the results, ReMERN can learn the optimal policy with fewer environment interactions than baseline methods like Q learning, PER and DisCor. ReMERN will first upweight transitions that go left. Once the Q value of going right surpasses that of going left, the on-policy correction dominates and upweights the transitions that go right. For PER and DisCor, transitions that go right have small TD-error and large $|Q_k-Q^*|$, so they will both downweight these transitions, resulting in a slower training process.
    
    
**Q: Regarding the temporal property of TD error, I find it hard to assume that we can ignore the TD error. TD error in practice is never zero and in fact grows during training and saturates. How can we make this optimization error 0?**

A:  In Section 3.4.1, $|Q_k-Q^*|$ is split into two terms with triangle inequality: $|Q_{k}-Q^{\ast}| \leq |Q_{k}-\mathcal{B}^{\ast} Q_{k-1}|+| \mathcal{B}^{\ast} Q_{k-1}-Q^{\ast}|$. The second term is usually very large at the early stage of training because the value function has not been trained with data collected by a good policy. For example, in the Ant-v2 task in MuJoCo, $| \mathcal{B}^\ast Q_{k-1}-Q^\ast|$ can be as large as 4000 at timestep 50k, while the TD error $|Q_k-\mathcal{B}^{\ast} Q_{k-1}|$ is often around 100. Intuitively, the TD error is negligible compared with $| \mathcal{B}^{\ast} Q_{k-1}-Q^\ast|$. Also, this intuition is mentioned only to introduce our idea of estimating $|Q_k-Q^{\ast}|$ with TCE. We give theoretical justifications afterwards that do not depend on this intuition.

**Q: The distance to the end is a heuristic, especially on cyclic MDPs like the gym MDPs. So, why does ReMERT still work in these domains?**

A: The distance to end has a theoretical justification as shown in theorem 2. Cyclic MDPs contradict to the conditions of theorem 2, so TCE might not give a proper estimation of $|Q_k-Q^*|$. However, the estimation of other terms remains unaffected, so ReMERT still works in these domains. Nonetheless, ReMERN performs better in such domains. 

**Q: Is there an ablation of TCE without the importance weights? Importance weights have been shown to add variance only, and prior methods have not found it to really help especially when off-policy replay buffers are used.**

A: We have done the ablation analysis to study the role of the on-policy reweight (only on-policy) and TCE (only TCE) in ReMERT. The results are shown in the following table. The values are the total returns after 1M timesteps in Fish-swim and Ant-v2, and after 4M timesteps in Humanoid-v2. We run each experiment with four random seeds and report their average returns.

|  Environment  |  Fish-swim  |  Ant-v2  |  Humanoid-v2  |
| ---- | ---- | ---- | ---- |
|      SAC      |     179     |    4923  |      5820     |
| Only On-policy|     183     |    5274  |      7047     |
|   Only TCE    |     161     |    5601  |      6289     |
|     ReMERT    |     287     |    5873  |      8167     |

As shown by the empirical results, both on-policy reweight and TCE play an important role in the overall performance of the ReMERT algorithm. The good performance of on-policy reweighting can be attributed to the introduction of slow and fast buffer. They reduce the variance of importance sampling as mentioned in [1].

**Q: What can be done to handle hindsight Bellman error? What happens if a PER estimator (historical Bellman error) is used for this term?**

A: We have tried to add PER to ReMERN and ReMERT and the empirical result in MuJoCo shows few improvements, but we notice performance improvements in Atari. The improvement is not significant overall. Also, currently we are not able to give theoretical analysis on the difference between hindsight Bellman error and historical Bellman error. Therefore, we do not include PER in our algorithm. However, we do think hindsight Bellman error is interesting and would like to explore this idea in future work.

**Q: In RL we are typically interested in optimizing the cumulative regret, not the instantaneous regret. Then why is the regret minimization formulation complete and interesting? Maybe using this formulation in tasks with static datasets is more informative? Is it possible to analyze this aspect on a gridworld, or does an experiment already do it?**

A: Instantaneous regret minimization focuses on exploiting a constructed buffer. Prioritization methods on the buffer add prior knowledge to the learning process, and different methods can make a big difference in the performance of RL agents. So we think instantaneous regret minimization is interesting. As to tasks with static datasets or Offline RL, we tried to add our prioritization weight on top of several of traditional offline RL algorithms, but the improvement is limited. This may be attributed to the extrapolation error in the offline setting, which is not considered in the derivation. 

**Q: Limitations are not discussed. Maybe it is worth discussing more details of when the proposed method fails (e.g., when is the estimation of importance weights hard?)**

A: First, as you mentioned, ReMERT is not a good choice for cyclic MDP. Second, according to the design of TCE, we can only apply ReMERT to episodic MDPs with terminal states. Third, the estimation of importance weight is hard in pixel-based environments like Atari, or when the policy changes quickly due to large variance. We will add discussions of the limitations to the Conclusion of our revised paper.

**Reference**

[1] Sinha, S., Song, J., Garg, A., and Ermon, S. (2020). Experience replay with likelihood-free importance weights.


This paper proposes a method for prioritized sampling of a replay buffer to make it easy for temporal difference learning methods to learn faster. The paper builds on prior work, DisCor (Kumar et al. 2020), and derives a more inclusive prioritization estimator which includes accuracy of targets and more on-policyness. Empirical results indicate that the proposed method leads to improved performance on a number of tasks including MetaWorld, Atari and Gym tasks.     Overall the paper has interesting ideas. While on one end, it seems to heavily build on analysis and techniques from DisCor (i.e, the novelty is somewhat on the lower end), and the proposed ReMERN method only adds an additional on-policy correction to the prior method (DisCor), the practical method ReMERT does seem to be a bit better empirically since it uses a different estimator for accuracy of target values. However, since the main ideas seem heavily inspired by prior work, the amount of novelty is a bit low. The results on domains that are harder (e.g, Meta-World and Atari) do seem good, which makes this method empirically interesting.

My main concerns/questions are:

1. What would the proposed method ReMERN do in the example in Figure 1(a)? Let's say the policy at the current instant takes all left actions. In this case, would the proposed method upweight transitions that also go left since it is a product of DisCor and importance weights, which would both suggest upweighting transitions that go left? So, is the proposed method failing in the same mode as DisCor in this example? 

2. Regarding the temporal property of TD error, I find it hard to assume that we can ignore the TD error. TD error in practice is never zero and infact grows during training and saturates. How can we make this optimization error 0? Moreover, no evidence of this is shown in the paper either (that this was indeed the case in the experiments)? Second, the distance to the end is a heuristic, especially on cyclic MDPs like the gym MDPs. So, why does ReMERT still work in these domains? 

3. Is there an ablation of TCE without the importance weights? Importance weights have been shown to add variance only, and prior methods have not found it to really help especially when off-policy replay buffers are used. An ablation to clearly disentangle TCE and importance weights is somewhat essential to verify this. This can also be thought of as a version of the DisCor idea instantiated differently, In my opinion, this is important to understand where the gains are coming from in the proposed method. 

4. What can be done to handle hindsight Bellman error? What happens if a PER estimator (historical Bellman error) is used for this term? 

5.  In RL we are typically interested in optimizing the cumulative regret, not the instantaneous regret. Then why is the regret minimization formulation complete and interesting? Maybe using this formulation in tasks with static datasets is more informative?  Is it possible to analyze this aspect on a gridworld, or does an experiment already do it?

Overall, I am voting for weak acceptance. It would be good to add the ablations listed above and experiments to make sure the claims of the paper are empirically valid, and for more understanding to readers.  I would like to add that the novelty in the paper is relatively on the lower side, but the idea overall is interesting and empirically it seems promising.

The paper is clearly written and understanding the main ideas is easy from the writeup. Limitations are not discussed. Maybe it is worth discussing more details of when the proposed method fails (e.g., when is the estimation of importance weights hard?)

The authors derive the optimal prioritization strategy for maximizing return of the greedy policy. From this, the authors introduce two approximations of this optimal strategy as re-weighting strategies when learning the value function. The algorithms are evaluated on several MuJoCo tasks, meta-world and Atari.  My biggest concern is with the novelty of the proposed approach over DisCor. 
- DisCor is derived from a very similar formulation to eqn (1), it's not clear to me why the change from optimizing regret is necessarily better than optimizing value error ($|Q - Q^*|$). In fact, the relaxation in the proof also relies on minimizing this value error (Eqn (12) in the supplementary). So why is this the resulting weight (Eqn (2)) different from DisCor? I think the authors should (1) mention that the origin of Eqn (1) is from DisCor and (2) discuss the differences and/or show why these changes are desirable.
- It's worth pointing out that Eqn (3) is identical to the solution of DisCor, where DisCor's $\lambda$ is the density ratio. 
- Figure 1 is used to describe the issues with PER and DisCor. But the explanation doesn't really explain why DisCor is suboptimal. Focusing on the value estimate of the left action is not necessarily wrong (if for example, the left action is over-estimated, and the right action is correct, then we would need to focus on the left action). Furthermore, the authors should compare their proposed approach in this same setting to demonstrate that it solves this suboptimality. 

One novel aspect of the paper is ReMERT which considers the estimation of value error by distance to the terminal state. However, this solution doesn't seem very general as its reliant on the horizon and the existence of terminal states. This approach appears to have some small benefits in MuJoCo but isn't even applicable in meta-world where these algorithms seem the most useful. 

Clarity could be improved. 
- As mentioned, Section 3.1 could better describe the issues with DisCor. 
- I think small algorithm boxes which summarize the algorithms (i.e. steps 6/7 in Algo 1/2 in the supplementary) would improve the presentation of the algorithms. 
- Eqn (4) doesn't define $f^*$ and $f'$. 

Given the paper claims to introduce a prioritization method, I do find it unusual that there are no baselines which actually use prioritized sampling anywhere in the paper. A quick glance at the PER paper suggests PER outperforms ReMERT on 4/5 of the Atari environments. Does ReMERT compare favorably against DisCor in this setting? The authors should also include ReMERN results. 

Lack of code is disappointing for reproducibility. 

Minor Comments:
- "Our analysis gives a theoretical explanation for some prioritization methods, including PER, LFIW and DisCor" (line 325). I don't think this statement is true at all based on the contents of the paper. 

Overall, I think the proposed approach is interesting, but the authors need to better distinguish the benefits of the approach over DisCor. I'm not sure the authors have discussed limitations. For example, some discussion on the role of the approximations to the optimal strategy would improve the paper. 

The paper analyzes the desired sampling from an experience replay buffer and proposes two new methods for weighting the samples in the buffer.
The paper's contributions are:
1. An detailed expression of the optimal weights.
2. An algorithm ReMERN for finding the weights using a neural network.
3. An algorithm ReMERT for finding the weights using a temporal property.  Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

The methods are novel. I especially liked Theorem 1 and how it explains the common intuition beyond previous methods. 

Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

I am bothered by the results regarding ReMERT. In the example of Figure 1, as in any initial example of RL, it is quite obvious that it's better to update the value from the finish line backward. This result is not surprising and can explain the success of ReMERT in all experiments in the paper. Moving from Theorem 2 to Eq. (11) includes several approximations and bounds, so I get the feeling that not much of the theory propagates to the actual algorithm. Also, the claim that TCE is an approximation of |Qk - Q*| is problematic on its own since if this is the case why not just use it to calculate Q* - in which case there seem to be some kind of equivalence between TCE and reward shaping with distance to goal. To summarize, I think the ReMERT related results are technically sound but misleading to the reader as similar experimental results could have been reached with simple logic, and ReMERT is dedicated towards a specific set of problems with finite horizons.

Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

Yes.

Figure 1(b), (c) X-label is missing. 

Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

Theorem 1 is a very significant result I think other researchers are likely to build on. As with most algorithms that require training another neural network for a marginal improvement, it's unlikely ReMERN will be used in practice though I do think it's a significant result. ReMERT is but another example of how to use prior knowledge on the environment, though to the best of my knowledge using it in the experience replay buffer is novel (even though this is not how ReMERT was presented in the paper). 
Overall the paper provides a unique theoretical result with experimental consequences. NA

Meta-review: We have a wide range of opinions for this paper. On the positive side, most reviewers agree that the problem addressed by the paper is important. But then, there are several concerns too:

- One of the main issues is the novelty, both theoretical and algorithmic, compared to DisCor (Kumar et al., "DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction," 2020). Some of the reviewers believe the theoretical novelty is not significant compared to DisCor, while others believe that the theoretical result, especially Theorem 1, is significant and insightful.
Since this is a point of contention, I suggest the authors to be more explicit in comparing their results with the prior work.

- The other issue is regarding the empirical studies, and whether the improvements are significant or not. The paper uses 3 or 4 random seeds in its experiments. The standard deviation of results is often large enough that we have a lot of overlap between curves in figures or the values in the table (for Atari).

- There is also a concern regarding Theorem 2.
The upper bound in Theorem 2 is "$\gamma R_\text{max} / (1 - \gamma) + ...$". This additive form makes the upper bound almost vacuous, as $R_\text{max}/(1 - \gamma)$ is equal to $Q_\text{max}$ and the quantity that is bounded is $|Q_k - Q^*|$, which is trivially smaller than 2 $Q_\text{max}$.
I should note that after bringing up this issue in the discussions, the authors provided another upper bound. That requires a careful verification by the reviewers.

- Some of the theoretical results are not presented precisely. For example, Theorem 1 provides a guarantee about a relaxation of an optimization problem in Eq. (1). But the relaxation is not specified in the main body. It appears to be Eqs. (12-13) in the appendix, which are presented as a part of the proof of an auxiliary lemma. This is simply too remote from where the result is stated.
It might be OK if the result is about a relaxed optimization problem instead of the original one, but the paper should be very clear and precise about it.

Overall, I think this paper might be interesting for the community, but it can benefit a lot from some major revisions. Given that two out of four reviewers are on the negative side, and that one of the positive reviewers is only recommending a weak accept, and all concerns mentioned above, unfortunately I cannot recommend the acceptance of this paper in its current form.

Decision: accept