Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: EDIT: The statements about ERO clarify the contribution considerably. 6-->7

The authors propose an adaptively sampling mechanism optimized for policy improvement (NERS). By incorporating minibatch-wide information into the sampling score (while maintaining permutation invariance), they are able to out-perform reasonable baselines on a wide range of tasks.

While NERS rarely beats other methods decisively, it has a strong showing across continuous and discrete action tasks and with a variety of off-policy learners. However, a few things would strengthen the empirical results. Some notation of spread should be reported on all of the Tables (e.g. standard error). Number of random seeds should also be mentioned. The reasoning behind the task selection should also be made explicit. The Atari subset used here is a bit unusual, particularly the choice to not use frame-skip. Investigating the sampling decisions of NERS is attempted in Figure 4, but further work should be done to provide evidence to the 'diversity of samples' claim. Ideally, NERS wouldn't just trade off TD error and Q-value over time, but also within each batch. Reporting something like the average minibatch Q-value/ TD-error standard deviation on NERS vs other methods would be nice. A qualitative evaluation akin to Figure 1 would also help guide intuition.

The off-policy RL related works section is a bit over-long, discussing things like the dueling architecture which don't seem to be overtly related apart from coming from the same sub-field. On the other side of things, having skimmed the ERO paper it is definitely the most-closely related, and as such deserves a bit more time spent on discussing the differences. For example, it is a bit unclear if the two sampling reward functions are different. An uncharitable reading of this paper would be that it is just an architecture tweak on top of ERO, and while the empirical results help dispel this idea, I think a more explicit comparison would still be useful.

A related point is that the reward function for the sampler is quite unclear (Equation 6). How are these expectation evaluated in practice? I'd assume it'd just be the difference of value functions before and after the update, but the appendix suggest a more involved computation that doesn't appear to have been made explicit anywhere.

Final small point, towards the end a bi-GRU is mentioned as being used and I can't see where that'd come into play. Perhaps just a typo?

Overall, I like this paper. Evaluating an idea across a variety of learning algorithms, observation and action spaces is no small feat, and the results are very solid. With a few tweaks and explanations this would be a very strong paper.







review_title: A sound idea and evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Observing that the existed ER-based sampling method may introduce bias or redundancy in sampled transitions, the paper proposes a new sampling method in the ER learning setting. The idea is to take into consideration the context, i.e. many visited transitions, rather than a single one, based on which one can measure the relative importance of each transition. Specifically, the weights of transitions are also learned through a Reinforce agent and hence the sampling distribution is learned to directly improve sample efficiency. 

Clarity. The presentation is clear in most places. But I do feel the core part of updating the sampling policy needs clarification. 

Quality. Please see the questions below.

Originality/Significance. The method is novel and is potentially interesting to the RL research community. 

In the original prioritized ER paper, they use a ratio to mitigate the biased sampling issue, did the authors ever visualize what the result would be (say in Figure 1) if you use that ratio? 

Eq (2) the input feature can be highly non-stationary/unstable. For example, some of the variables may decrease all the time, and some others may increase all the time. Intuitively, training with such data should be very challenging. It looks like that in order to resolve the issue of biased sampling, the authors introduce an even more difficult task. Do the authors have some comments about this? 

I am confused about how the sampling network is updated. In Algorithm 1, my understanding is that if the current time step is the end of an episode, then update the sampling network. Is it correct? Note that Algorithm 1 indicates that the actor, critic are updated at each time step. But the paper also says that the NERS is updated at each evaluation step and this means that throughout the evaluation episode the policy should be fixed to estimate (6). Can the author further explain how the network is updated? 

Using evaluation to learn parameters seems unrealistic, the evaluation may happen only one time in practice. By evaluation, my understanding is that how much we can gain if we deploy such a policy. If the updating NERS requires to use evaluation data, this largely limited usability. 

I am also confused by the statement “The replay reward is interpreted as measuring how much actions of the sampling policy help the learning of the agent for each episode.” Eq (6) says that the reward is actually how much it improved from the current evaluation to the previous one. No matter you use the special sampling method or not, it should make an improvement. So this difference does not indicate “how much the sampling distribution can help.”

In the empirical study, NERS does not show a clear benefit from the learning curves. I believe it is better to average over a smoothing window before averaging over random seeds. Or do more runs. Doing more runs should not be that computationally expensive at least on Pendulum and LunarLander. And one question, how do you implement the Prioritized ER? Do you also use the importance ratio to anneal the bias as described in section 3.4 in that paper (https://arxiv.org/pdf/1511.05952.pdf)? Do you ever tune a bit the parameter beta in that formulae? 

An additional note about related work. 
The sampling distribution is an important problem in RL and is not well/rigorously studied. I really think it worths a more complete discussion of related work. For example, the author might also discuss the Langevin dynamics Monte Carlo sampling method in RL (Frequency-based search control in Dyna by Pan et al.), as their sampling distribution is supported by intuition and suggestive theoretical evidence and they show their method is better than prioritized ER and ER. 
review_title: The paper proposes an interesting idea to design sampling distribution to improve sample efficiency of deep RL algorithms. But there are several nontrivial issues to be resolved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:
This paper proposes to improve sampling from the experience replay buffers that weights samples by their "relatively usefulness".  

This paper proposes to use two encoders - one global that encodes across the current batch of experience replay sample, and one local that encodes each selected timestep. Using the encodings, a scorer scores the experiences and weights the actor and critic losses in proportion to that score. The sampler is trained to maximize the probability that it chooses high reward timesteps on average. 


 
##########################################################################

Pros: 

Experience Replay is a widely used technique and improving on the naive random sample method makes sense. The analysis on other sampling methods is insightful. 
The idea to have a global feature pooled from the sample transitions is unique. 
Reweighing each sample's loss at training time is simple and effective.

##########################################################################

Cons: 

The paper's idea could be flawed. The sampler is trained so that it prioritizes high reward timesteps as in eq 6.
But this is dubious. What if we need to sample the N timesteps right before a high-reward timestep, even though those preceding timesteps do not have high return themselves? And how do you explain why"NERS focuses on sampling transitions with high TD-errors in the beginning, ... as the timestep progresses, it samples transitions with both high TD-errors and Q-values (diverse)", given that it's trained with a single objective to maximize sampled reward? "using various features in an advanced manner." is not a satisfactory explaination. 


The experiment section is not convincing. 
The model has not been trained till convergence. e.g. SAC and ERO has all been trained with 1e6+ steps at least. In addition, what is the reason to do Hopper from Mujoco instead of Hopper-V2 from the OpenAI gym? With the latter, you can compare with the numbers published in ERO.

##########################################################################

Questions during rebuttal period: 
Please address and clarify the cons above 


review_title: Interesting idea after clarifying the misunderstanding.

Meta-review: All reviewers agree that this paper is worth publishing. It investigates a novel idea on how to adaptively prioritise experiences from replay based on relative (within-batch) importance. The empirical investigation is thorough, and while the performance improvements are not stunning, the benefit is surprisingly consistent across many environments.
Decision: Accept (Poster)