Benchmarking Sample Selection Strategies for Batch Reinforcement Learning
Authors: Yuwei Fu
Di Wu
Benoit Boulet
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Experience Replay

Abstract: Training sample section techniques, such as prioritized experience replay (PER), have been recognized as of significant importance for online reinforcement learning algorithms. Efficient sample selection can help further improve the learning efficiency and the final learning performance. However, the impact of sample selection for batch reinforcement learning algorithms, where we aim to learn a near-optimal policy exclusively from the offline logged dataset, has not been well studied. In this work, we investigate the application of non-uniform sampling techniques in batch reinforcement learning. In particular, we compare six variants of PER based on various heuristic priority metrics that focus on different aspects of the offline learning setting. These metrics include temporal-difference error, n-step return, self-imitation learning objective, pseudo-count, uncertainty, and likelihood. Through extensive experiments on the standard batch RL datasets, we find that non-uniform sampling is also effective in batch RL settings. Furthermore, there is no single metric that works in all situations. Our findings also show that it is insufficient to avoid the bootstrapping error in batch reinforcement learning by only changing the sampling scheme.

Under review as a conference paper at ICLR 2022
BENCHMARKING SAMPLE SELECTION STRATEGIES FOR BATCH REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Training sample selection techniques, such as prioritized experience replay (PER), have been recognized as of significant importance for online reinforcement learn- ing algorithms. Efficient sample selection can help further improve the learning efficiency and the final performance. However, the impact of sample selection for batch reinforcement learning algorithms, where we aim to learn a near-optimal policy exclusively from the offiin logged dataset, has not been well studied. In this work, we investigate the application of non-uniform sampling techniques in batch reinforcement learning. In particular, we compare six variants of PER based on various heuristic priority metrics that focus on different aspects of the offline learning setting. These metrics include -rrpoorr-difffrence error, n-step re- turn, self-imitation learning objective, pscudo-count, uncertainty, and likelihood. Through extensive experiments non-uniform sampling is also effective batch RL. settings. Furthermore, there is no single metric that works in all situations. Our findings also show that it is in- sufficient to avoid the bootstrapping error in batch reinforcement learning by only changing the sampling scheme.
the standard batch RL datasets, we find that
INTRODUCTION
A key question in machine learning is to select the suitable training samples (Katharopoulos & Fleuret, 2018). Many prior works proved that an appropriate sample selection strategy, i.e., removing redundant data or selecting samples according to their hardness, usually significantly improves the learning efficiency and final performance (Bengio al., 2009; Schaul et al., 2015; Fan al., 2017). Similarly, sample selection also plays crucial role in reinforcement learning (RL) (De Bruin et al., 2018). A notable example is the sample selection problem for experience replay (ER) in off-policy RL (Fedus al., 2020), where agent reuses stored experiences from a buffer while interacting with the environment. For example, Prioritized Experience Replay (PER) (Schaul et al., 2015), which samples high error transitions more frequently, is now widely used in different state-of-the- ant (SOTA) off-policy RL algorithms (Barth-Maron et al., 2018; Hessel et al., 2018; Kapturowski al., 2018).
Batch RI,, also known as offline RI., refers to the problem of learning a near-optimal policy from a fixed offline buffer (Lange et al., 2012). Due the wide availability of logged-data and the increas- ing computing power, batch RL holds the promise for successful real-world applications (Levine al., 2020). Especially for the scenarios where collecting online data is time-consuming, dan- gerous or unethical, i.e., robotics, self-driving cars and medical treatments (Gulcehre et al., 2020). While most off- oif-colicy RL algorithmss are applicable in the offline setting, they usually suffer from the bootstrapping error (Fujimoto et al., 201 8b; Kumar et al., 2019) due to out-of-distiibution (OOD) state-action pairs. Different solutions are proposed to mitigate this problem, i.e,, adding constraints (Fujimoto et al., 2018b; Wu al., 2019), imitating behavior policy (Chen et al., 2019; Zolna et al., 2020), learning dynamics models (Yu et al., 2020; Kidambi et al., 2020; Argenson & Dulac- Amnold, 2020), incorporating uncertainties (Wu et al., 2021), learning ensembles (Agarwal al., 2020), or learning pessimistic value functions (Kumar et al., 2020; Buckman et al., 2020; Jin et al., 2021).
Unlike the wide application of PER in online off off-policy RL, the non-uniform sampling strategy is largely ignored in recent batch RL algorithms. Inspired by the success of PER (Schaul et al., 2015) in the online setting, one natural question to ask is that what is the counterpart of PER in batch RL.

---

Under review as a conference paper at ICLR 2022
This problem is appealing for several reasons: (1) In some real-world applications, the size of the offline dataset is usually increasing though we have no access to the real environment. For example, would have ever-growing medical records from the hospitals (Raghu, 2019), or recorded videos from dash cams (Yu et al., 2018). (2) As the D4RI. offline benchmark (Fu et al., 2020) shows that for most existing methods, more samples do not necessarily lead to better performance. That is, batch RL agent sometimes under-performs in a large combined buffer. Given the success achieved by PER in online RL, we are curious that if similar technique could help to develop more robust batch RL agents (Fujimoto et al., 2020).
Some prior works proposed different sample selection strategies in batch RL. For example, Optimal Sample Selection (OSS) (Rachelson et al., 201 1) introduced meta-leaming algorithm which se- lects optimal samples according to a cross entropy search method for tree-based Fitted Q-Iteration (FQI) (Ernst et al., 2005) with a known dynamics model. Recently, Best-Action Imitation Learning (BAIL) (Chen et al., 2019) proposed to select high- performing samples with a learned value func- tion in behavior cloning. Another related line of research is to reweight sampled transitions. For example, Advantage-Weighee Weighted Regression (AWR) (Peng et al., 2019) and advantage-weightee Be- havior Model (ABM) (Siegel et al., 2020) both used weighte regression (Peters et al., 2010) to learn the policy. Further, Uncertainty Weighted Actor Critic (UWAC) (Wu et al., 2021) adopted papooutuuncertann estimation method (Gal & Ghahramani, 2016) and reweighted samples ac- cording to their estimated uncertainties. However, it is unclear which sample selection strategy is preferred in batch RL, thereby demanding more investigations.
To this end, in this work, we study the sample selection problem in batch RL (De Bruin et al., 2018). We follow the PER framework by assigning samples with different priorities (Schaul et al., 2015). Crudely, there are two types of metrics to evaluate sample importance. Firstly, we can design a heuristic metric based on our prior knowledge. i.e., temporal-different can use an end-to-end approach to learn a metric for each sample, for example, we can use off- policy evaluation (OPE) methods (Voloshin et al., 2019; Fu et al., 2021) to evaluate the goodness of current policy as the metric. However, existing OPE methods usually need to learn a model for each evaluation (Le et al., 2019), which makes the learning- based metric approach to be computationally expensive. Therefore, in this paper, focus on the heuristic metric-based approach and leave the learning-base based metric approach for future work. In particular, we benchmark six variants of PER based on different heuristic priority metrics in order to understand which sample selection strategy might be preferred in batch RL.
(TD) error. Secondly, we
2
PRELIMINARIES
2.1 BATCH REINFORCEMENT LEARNING
We consider the standard Markov Decision Process (MDP) (Puterman, 2014) M (S, S and A denote the state and action spaces. T(s'ls, a) and r(s,a) a) represent the dynamics and reward function, and y E [0, 1) is the discount factor. A policy <{als) defines a mapping from state to distributions over actions. The goal of an RL agent is to lear policy r[ass) that maximizes the expected cumulative discounted reward [(T) == E. so
/ rrr.. The performance of the policy can be defined by the Q-function Q (s, a) := E, Ei= 1'r:|50 = 8, 4g = a) and value function (s) : E,, /E= 7're|s0 = s). where [+] is the expected result when following the policy m. Once given the optimal Q-function (* a) = arg max= Q" (0(( a), we can derive an optimal policy as π" (a (ss = arg max, Q"(ss, a) (Sutton & Barto, 2018).
In (tabular) Q-learning, we solve for the Q' by iterating the Optimality Bellman Operator T*, de- fined as T'Q(s, a) r y max. ((s',a) a)) (Bertsekas & Tsitsiklis, 1996). To solve problems with large state space, we can use parameterized Q-function (s a) to approximate Q'. In practice, we optimize the parameters by a ju-weighted L2 projection (Q) (Fu et al., 2019), which mini- mizes the empirical Bellman error loss: 1I, (Q) = ming E(s,a, [[T (s((s a) - Q0(s, a)
Batch RL, also known as offline RL, aims to learn a near-optimal policy from a fixed dataset (Lange et al., 2012) D. representing a series of timestep tuples (s,, ae, Te. $841). Furthermore, the dataset can be collected by agents with different policies from different control tasks, including RL policies, such as human demonstrations (Levine et al., 2020). Some early works such as Fitted Q- Iteration (FQI) (Ernst et al., 2005) and Neural Fited Iterraiion (NFQ) (Riedmiller, 2005), which
2

---

Under review as a conference paper at ICLR 2022
formulate the original RL problem as sequence of supervised regression problem, are shown to be sample efficient in solving various real-world problems (Pietquin et al., 2011; Cunha et al., 2015). On the other hand, some recent studies show that current deep off-policy RL algorithms usually fail in challenging batch RI. problems due to bootstrapping error (Fujimoto et al., 201 2018h; Kumar al., 2019). That is, the OOD action a' might lead to unrecoverable over-estimation error through max operator in the Bellman backup. The over-estimation problem is particularly detrimental in the offline setting where the agent has no access interact with the real environment get the feedback to fix the estimation error (Kumar et al., 2020).
2.2 NON-UNIFORM SAMPLING WITH EXPERIENCE REPLAY
Experience replay (ER) (Lin, 1992) has been de facto component for modern deep RL algorithms. By reusing previous collected experiences from the replay buffer, ER helps to reduce sample com- plexity and stabilize training in off-policy RL (Mnih et al., 2013; Lillicrap et al., 2015). For some real-world problems where collecting online data is expensive time consuming, L.e., robotics or self-driving cars, the ability to learn good policies from pre-collected data is crucial for successful real-world applications (Cabi et al., 2019).
A number of works (Schaul et al., 2015; Andrychowicz et al., 2017; Liu et al., 2019; Sun et al., 2020; Fujimoto et al., 2020) show that applying different non-uniform sampling strategies in ER can significantly improve the learning efficiency. Especially for problems where there are many redundant transitions (Schaul al., 2015), or the reward signal sparse (Andrychowiez et al., 2017). A notable example is the Prioritized Experience Replay (PER) (Schaul et al., 2015), where the probability of sampling a certain transition (s,, at. "t. St+1)) proportional the absolute TD error. However, it is still an open question that which priority metric is optimal to value the importance of samples (De Bruin al., 2018).
RELATED WORK
3.1 SAMPLE SELECTION WITH EXPERIENCE REPLAY
Many prior works have sought to analyze the mechanism of experience replay, both empirically (De Bruin et al., 2018; Fedus et al., 2020) and theoretically (Fujimoto et al., 2020; Li et al., 2021). Similar to our work, (De Bruin et al., 2018) investigated number of proxies, i.e., age, TD error, and exploration noise, to decide which experience to store in the replay buffer and how to sample from the replay buffer. Likewise, (Fu al., 2019) used "unit-testing" framework to study Q-learning with function approximators and found that a sampling scheme with wider coverage improves per- formance. Further, (Fedus et al., 2020) conducted systematic analysis of experience replay in Q-learning methods and provided two insights - (1) Increasing the buffer capacity is preferable, because has broader data coverage. (2) Decreasing the age of the oldest policy improves the performance, because it contains more high-quality on-policy data. While these insights help us to understand the mechanism of experience replay, they are less practical in the batch RL setting, where the given offline dataset is fixed (Lange et al., 2012).
number of variants of ER have been introduced to further improve the learning efficiency (Schaul et al.. 2015; Andrychowicz et al., 2017: Novati & Koumoutsakos, 2019; Liu et al., 2019; Sun et al., 2020). One of the most popular variants is the Prioritized Experience Replay (PER) (Schaul al., 2015), which proposed to use the absolute TD error (5(i) as the priority metric and the probability of sampling the i-th transition is:
DS p()) Σ,pg' P. = 1800|+ or P rank(1)'
(1)
where is hyper parreee is small positive constant to avoid zero priority. priority is the value of 16()) or the inverse rank of [6()). addition, Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) proposed to re-label visited state as goal states to overcome hard ex- ploration problems with sparse rewards. Competitive Experience Replay (CER) Liu et al. (2019) later introduced an automatic exploratory curriculum by formulating exploration competition be- tween two agents. On the other hand, Remember and Forget Experience Replay (ReF-ER) (Novati & Koumoutsakos, 2019) classified samples "near-policy" and "far-policy" by the importance weight
3

---

Under review as a conference paper at ICLR 2022
p= [[als)/u((as)) between current policy π and the behavior policy μ4, and compute gradients only with near near-policy samples. Similarly, Attentive Experience Replay (AER) (Sun et al., 2020) selects samples according to the similarities between the transition state and current state.
Recently, Loss-Adjusted Prioritized (LAP) experience replay (Fujimoto et al., 2020) built the con- nection between the non-uniform sampling scheme in PER and loss functions. It shows that any loss function , evaluated with uniform sampling (i ~ Di) is equivalent to another loss function L2 that is evaluated with non-uniformly sampled data (i D2):
PD. V VeCI(6(1) = E,DD [VCC2(((1).]
Ea [VL((((0) = EawDa
(2)
PD,
where 5()) is the TD error of the i-th sample and the two loss functions follows VQL2(0(i)) = po, (+)
PD (2) VeLI((()) Moreover, Valuable Experience Replay (VER) (Li et al., 2021) proved that the absolute TD error [6(1) is an upper-bound of different value metrics of experiences in Q-learning.
3.2 SAMPLE SELECTION IN BATCH REINFORCEMENT LEARNING
A pioneering work that applied sample selection strategy in batch RL is the Optimal Sample Selec- tion (OSS) method (Rachelson et al., 2011). More specifically, OSS a model-based RL (MBRL) approach (Sutton & Barto, 2018) where known dynamics model is available to generate Monte Carlo rollouts for policy evaluation. Moreover, OSS introduced a meta-lcarning algorithm to se- lect optimal samples according to the cross entropy search method (Rubinstein & Kroese, 2004) for tree-based Fitted Q-tertiion (FQI) (Ernst et al., 2005). Recently, Best-Action Imitation Learning (BAIL) (Chen et al., 2019) proposed to learn special value function (s), called upper envelope, that upper bounds the cumulative discounted return G3 = Ei= r "r from starting from state s; to the end of the episode (max horizon 2"). The learned upper envelope (S) is then used to filter high-quality samples to train a behavior cloning policy (Pomerleau, 1991).
Another related line of research is to reweight samples (Tirinzoni et al., 2018; Peng et al., 2019; Wu et al., 2021). Unlike previous methods that actively select samples from the buffer, these methods still adopt uniform sampling while assigning different weights to each sample compute the loss Saaaantgg-eeightteee Regression (AWR) (Peng et al., 2019) first formulated the RL problem as a supervised regression problem, and then used a learned value function train the policy iT (a]ss) via reward- wwighte regression (Peters et al., 2010), which assigns higher weights to samples with large advantage values. Similarly, Advantage-weighted Behavior Model (ABM) (Siegel et al., 2020) adopted reward-weighted regression in policy training to focus more on good actions. On the other hand, Uncertainty Weighted Actor Critic (UWAC) (Wu et al., 2021) used Monte Carlo Dropout (Gal& & Ghahramani, 2016) approximate the epistemic uncertainty (Kendall & Gal, 2017) for samples batch RL dataset. The goal of UWAC is to assign lower weights to samples with higher epistemic uncertainty in order to mitigate the bootstrapping error caused by OOD state-action pairs (Fujimoto et al., 2018b; Kumar et al., 2019).
function. For example,
METHODOLOGY
4.1 BACKBONE ALGORIHTM
In this work, we select TD3BC (Fujimoto & Gu, 2021) and PER (Schaul et al., 2015) as the backbone algorithms for benchmarking sample selection strategies in batch RL. TD3BC is a minimalist batch RL algorithm which simply adds a behavior cloning term to the TD3 algorithm (Fujimoto et al., 2018a). While being simple, TD3BC achieves comparable performance w.r.t. other SOTA batch RL algorithms (Kostrikov al., 2021; Kumar al., 2020) on the standard batch RL benchmark (Fu et al., 2020). Moreover, TD3BC is able to run significantly faster than previous methods by removing additional computations overheads.
In particular, TD3BC made two small modifications upon the origin TD3 algorithm (Fujimoto & Gu, 2021). Firstly, it adds behavior cloning regularization term the policy update in order to keep the learned policy π to stay close to the behavior policy μc:
= arg max E(e.a)DD (()) - (I(() - a)?]
(3)
4

---

Under review as a conference paper at ICLR 2022
where À is parameter trade-off RL and imitation. Secondly, it normalizes the states s in the offline dataset by = (s sp)/s, where 8y and 8 are the mean and stander deviation.
In terms of the non-uniform sampling strategy, follow the PER framework (Schaul et al., 2015) in which the probability to sample transition is p(i) = pf / Pf. where p; is the priority of transition i and parameter a determines how much prioritization is used. In this paper, we investigate the problem of how the choice of priority metric matters in batch RL. We use both the proportional PER and rank-based PER in the experiment depending on the used priority metric. Proportional PER is more popular baseline, while rank-based PER is more robust outliers especially when different metrics have inconsistent scales.
4.2 PROPOSED METRICS
Here, we introduce six different priority metrics that we use in the experiment. We select these metrics based on prior insights of what data might be preferred in batch RL. Depending on whether the metric changes the experiment, we could further divide them into static and dynamic metrics. A summary of these metrics is shown in Table 1.
Table 1: List of proposed metrics. Depending on if the value is fixed during training, we divide them into static and dynamic metries. Some metrics are more computationally expensive, i.e., requiring dynamics model or behavior policy.
Metric
Type
Motivation
Prioritization
Extra computation
TD-Error
Dynamic
Reducing redundant samples
Proportional PER
N-step Return
Statie
Selecting good samples
Rank PER
GSIL
Dynamic
Selecting good samples
Proportional PER
A second buffer
Pseudo-coum count
Static
Avoiding OOD samples
Rank PER
Hash table
Uncertainty
Statie
Avoiding OOD samples
Rank PER
Probabilistic ensemble
Likelihood
Static
Being on-policy
Rank PER
Behavior policy
TD error. We first select the TD error as primary baseline, and test how well does the naive PER (Schaul et al., 2015) perform in the batch RL setting. We adopt the most popular proportional PER, and the priority for the i-th transition is pt 18(i) + e, where (8(i) is the absolute TD error and e is a small positive constant to avoid zero priority. The motivation of using TD error to select samples that small absolute TD error samples contain less information for our model to learn from (Moore & Atkeson, 1993).
N-step return. Secondly, select the n-step return as the proxy evaluate the goodness of sam- ples. Uncorrected n-step return has been shown to be an effective technique that greatly improves performances (Hessel al., 2018; Fedus al., 2020; Rowland al., 2020). Similar the idea of BAIL (Chen al., 2019), we hypothesis that samples with higher n-step return is more likely to be high-quality samples. Unlike Monte Carlo return which requires a full trajectory, n-step return are much more practical in real-world problems, where usually only have partial trajectories without N- terminal state. Specifically, the n-step return for transition is RN > 7 7'rise, where N is the horizon length. Notice that, the n-step return is fixed during the experiment which we only need compute once in the data pre-processing step. Given the different reward scales across tasks, use rank based PER in the experiment P. = 1/rank(R )..).
Our third metric is inspired by the Self-Imitation Learning (SIL) (Oh al., 2018), which exploits past good experiences. In particular, SIL imitates past good experiences by optimiz- ing following actor-critic loss function:
Generalized SIL.
Cii palue
[ (o
Cail policy = - log Te [R
Vo(s))
(4)
2
where = E 'r,, is the cumulative discounted return starting from state after taking action a, and [xr] max r). The motivation of SIL. is intuitive that policy to(als) should imitate action
5

---

Under review as a conference paper at ICLR 2022
if it is high-performing, such that R > 2020) later extends the original SIL to deterministic actor-critic setting with n-step TD-learning We follow GSIL to set the priority for the i-th transition to be p; [R Q0(s; a:)] te, where R is the n-step return and is small positive constant.
Vo(s). Generalized Self-Imitation Learning (GSIL) (Tang,
Although batch RL do not concern the exploration problem (Osband et al., 2016). We can still borrow some insights from exploration perspective distinguish useful samples. For example, in the experiment, we test the efficacy of Pseudo-count (Ostrovski et al., 2017; Tang et al., 2017) for sample selection in batch RL. We follow the #Exploration (Tang et al., 2017) model to use locality-sensitiv hashing (LSH) method, i.e., SimHash (Charikar, 2002). to convert continuous state 8 E R discrete k-dimension hash codes:
Pseudo-count.
(5) where g(.) is an optional preprocessing function and E R**D is a random matrix. In the batch RL dataset, small pseudo-count N, of state-action pair (8),a)) means the dataset has insufficient coverage around these data. Hence, it would be more likely to suffer from the OOD sample problem when learning from these samples. In the experiment, use a rank-based PER and set the priority of the i-th sample to be p; = 1 (N,,)
<(s) = (ggn(A(())) E (-1,133
Uncertainty. Inspired by UWAC (Wu al., 2021), we also attempt to use the epistemic un- certainty to evaluate the sample importance. UWAC adds a dropout layer before every weight layer (Gal & Ghahramani, 2016) and approximate uncertainty of state-action pair (s,a) by the variance of predicted Q(s, a). To exactly analyze how does the uncertainty-based sampling in- fluence the performance, we adopt the probabilistic ensemble (Chua al., 2018) method in- stead of change the origin TD3BC model. Specifically, we first train an ensemble of M prob- abilistic dynamic models TT..
Tm} (Pineda et al., 2021), where each dynamic model (1 88,, ax) = [ (st, 0z), Eo, (st, ax)) outputs a Gaussian distribution with diagonal covari- ances parameterized by 0, For the i-th transition (8,, 02, Ft, 8;+1), we approximate its epistemic uncertainty by the standard deviation of o; std (!ue, (5), a.), Mess (5,, a)))). We use a rank- based PER to assign higher priority to samples with smaller uncertainty, that is p3 1/rank(a,,))
(s),0)),
Likelihood. The last metric we test in the experiment is the likelihood of the behavior model (Kostrikov et al., 2021). Similar to previous constrained based batch RL algorithms (Fujimoto et al., 2018b; Wu et al., 2019; Kumar et al., 2019), we want to make the learned policy T[as)) to stay close to the behavior policy u(als). Therefore, we first learned behavior policy with mixture of Gaussian model (Kostrikov et al., 2021) 1) and used the likelihood as the priority. We use a rank-based PER where p; 1/rank p ((aa)s))) is the priority for the i-th sample.
EXPERIMENT
In this section, we compare different PER variants with the proposed metrics on a variety of batch RL continuous control tasks (Fu al., 2020). We seek to address the following questions in the experiments: (1) Does non-uniform sampling scheme also help to improve the performance in hatch RL? (2) Which priority metric is preferred in the batch RL setting?
Datasets. We evaluate different sample selection strategies on the widely-used D4RL gym Mu- joco benchmark (Todorov et al., 2012; Fu et al., 2020), including three environments (halfcheetah, hopper, and walker2d) and five dataset types (random, medium, medium-replay. medium-exper expert), yielding total of 15 datasets. These datasets differ in many aspects, e.g., number of tran- sitions, quality of behavior policy, and data coverage. We seek to validate the robustness of each sample selection strategy in different domains.
Experiment setup. For the backbone algorithm, we use the author-provide implementation for TD3BC 1. We maintain two replay buffers for the GSIL metric as in the origin paper' (Tang, 2020), where the first buffer stores single-step transitions train TD3 and the second buffer stores n-step transitions to compute the GSIL. loss. addition, we use the MBRL-Lib (Pineda et al., 2021) to
'attss//ithubb.oo//tttjn//// BC
2 hhttps//githhub... hhtpss//gituu..commooboni wtt/:///i/t////////oooeooeee https//gggnnoooooo/cccccceeeeete

---

Under review as a conference paper at ICLR 2022
train the probabilistic ensemble, and implement the SimHash according to EPG (Houthooft al., 2018). Parameters for the PER are taken from the original paper (Schaul et al., 2015). We follow exactly the same experimental setup as (Fujimoto & Gu, 2021), in which train for 1 million time steps and evaluate every 5000 time steps for 10 episodes. More details are in the Appendix.
Results. We report the final performance of different priority metrics in Table 2 and plot the learn- ing curves in Figure 1. We make several observations: (1) Non-uniform sampling strategy is also effective in batch RL, for example, the most performant method in each environment usually non-uniform sampling strategy. (2) There is no single metric that is consistently the best performer. (3) In some environments, such as Hopper-Medium Medium and Hopper- Expert, different sampling schemes perform very similar. In light of these results, we conclude that offline datasets are quite compli- cate and multiple factors can influence the sample priority. In environments with relatively low dimensions, such as Hopper, the learned policy is less affected by the sampling scheme.
Table 2: Performance of different priority metrics in the D4RL datasets. We report the average normalized score over the final 10 evaluations over seeds (+t standard deviation).
Uniform
TD-Error
Nstep-Return
GSII
Pseudo
Uncertiinty
Likelibood
Random
HallCheetah
11.2 + 1.3
11.1 + 111
10.3 + 0.6
9.1 2.0
11.3 + 1.3
11.4 + 1.2
11.0 + 0.6
Hopper
11.0 1 0.0
109 1 0.1
11.0 i: 0.0
10.9 1: 0.0
11.1 0.0
10.8 1 0.1
11.0 1 0.0
Walker2d
0.9 0 0.6
1.7 + 1.1
2.6
5.1 + 0.3
24 0.6
233
Medium
HalfCheerah
42.9 0.1
42.8 + 0.3
43,9 + 0.5
43.2 1:2::00 0.2
43.3
42,9 + 0.4
42.4 + 0.1
Hopper
0.1
99.6000 0.4
99.4 + 0.6
99.8 0.1
0.1
99.8 + 0.1
99.8 = 0.2
Walker2d
0.
78.2 1.0
77.3 - 1.2
77.9 1.3
7.7
76.9 + 0.6
79.4 = 0.6
Modium Replay
HalfCheetah
43.1 + 0.4
43.3 0.1
43,5 + 0.5
42.8 42.800.2
43.3 0.5
43.4 0.2
43.3 + 0.0
Hopper
32.1 1: 1.3
30,3 1 0.8
31.4 4: 0.7
30.6 1:61:10 1.9
31.9 0.3
31.1 4: 1.8
31.7 + 1.6
Walker2d
24.3 1: 4.6
23.8 1: 2.5
17.4 2.7
15.2 9.4
29.0 1 3.6
26.4 1: 1.0
24.8 1: 1.1
Medium
Expor
HalfCheetah
92.4 1.5
96.9 1.7
87.8 3.4 3.4
9.1
91.3
88.0
84.4 1: 4.0
Hopper
0,
106.2 + L.I
110.7 1.7
111.6
112.2 + 0.0
109.8 10 2.2
111.4 0.6
Walker2d
4.2
96.9 = 3.0
90.8 + 2.6
103.1 4.1
96.9 1.9::.0 4.6
103.9 + 2.1
81.4 = 23.6
Expert
HalfCheetah
0.7
103.5 L.5
102.4 1.6
105.4 +
104.0 + 1.2
103.8
104.5
Hopper
112.3
112.2 + 0.1
112.2 0.1
112.1 +
112.2 + 0.1
111.8 1 0.6
44.6 47.9
Walker2d
105.0
104.5 + 188
105.5 1.7
103.8 1: 1.3
104.2 1 1.1
105.9 0.4
104.1 1 3.3
haseline
Sampling scheme and bootstrapping error. We also plot the learned Q1 function in TD3 (Figure 2) to check for the bootstrapping error problem (Fujimoto et al., 201 8b; Kumar et al., 2019). We can see that all the sampling schemes learn explosive Q values in the Walker2d-Random environment, which implies that non-uniform sampling schemes fail to avoid the bootstrapping error. In addition, we can observe that the sampling scheme affects the learned Q function, where the likelihood metric usually learns relatively smaller Q values and N-step return metric learns relatively higher Q values. This corresponds to the inductive bias of each metric. For example, a transition with higher N-step return is more likely to have large single-step return, which leads to higher Q values in the Bellman backup. On the other hand, in the experiment, a transition with high likelihood is more likely to have lower reward. This may because most transitions the dataset are early stage of a trajectory, which have lower rewards. In addition, from figure 2, we can also observe that the bootstrapping error not the only problem which prevents us learn a good offline policy. For example, we did not suffer from severe bootstrapping error in HalfCheetah Randoom and Hopper- Random environment, but we still fail to exploit the offline dataset to learn performant policies. This maybe the fixed behavior cloning parameter to be overly-constrained in such datasets with low data-quality. Thus, we believe that there is promise in further improving performance by relaxing the fixed behavior cloning parameter through a dynamic evaluation of current data-quality.
Some problems of the proposed metrics. Here, we summarize some shortcomings the heuristic metric-based sample selection strategies. Firstly, we may need extra computations to compute the priority metrics (Table 1). For example, training the probability ensemble to estimate the sample

---

Under review as a conference paper at ICLR 2022
Uniform
Raturn N
Psoudo Count
Uncertainty
HorChactah-Aancco
Holtthhotah-Moddue Modium
HaltCheetah Nedium Ropay
HalChhotth Medium Expert
HolfChoctah- Expart
100
102
Normaizze
02
04
6.5
0.8
02
84
0.6
0.8
0.
0.4
06
0.8
0.
0.4
86
0.8
02
044
0.6
08
Hosper-aaddom
Hoppar&Nedium
Hoppar Mediu Redlay
Hoopors Med Med-uu-ddderttttt
Hopper-Expart
120
135
200
102
8C
Normalzed
09.6
20.4
04
6/6
0.6
0.8
06
0.8
0.8
Wellee22--andoo
Voller26-Medium
Modiim--apllz Rapley.
Expert
Normaizzd
22
10
Steps (1o6)
Stops (1:6)
Steos (1e5)
Steps (1c6)
Stops (1o6)
Figure 1: Results are averaged across 3 random seeds with shaded areas representing the standard deviation. We can observe that non-uniform sampling strategies is also effective in batch RI, and there is no priority metric that works all situations.
Paeudo_Coumt
Uncertainty MallCheettth-eecuu--ee
heliCheetah-Rardee Raniimmm----mmmmmmm
HellCheedah-teddumm
MedChheeth--ellm--eee Replyy
HalfC welli/reeaah-Eppee
391
2002
00
600
800
200
400
600
0
0.2
84
3.6
0.8
0.2
0.1
0.6
0.8
0.2
0.1
0.6
0.8
0.2
0.1
0.6
C.
0.2
0.1
0.6
0A
HopperRanddm
Hopper-Medium
Hoppor Medium Replay
Hopper Medium
Hodpar-Medium Export
1000
Hooper-Expeit
344
250
200
30
150
400
100
102
2003
0.2
0.4
0.8
0.5
0.2
0.4
0.6
0.5
0.2
0.4
06
0.8
0.2
0.4
0.6
0.8
0.2
044
0.6
0.
1.5
Whl Walcer2d--Pi Rendom
Welker2d Medium
1000
Walkeerd Medium Replay
Walker20 Medium Expert
400
Welker2d Expert
200
150
102
400
100
100
0.2
0.2
0.4
066
0.2
0.4
06
0.8
0.2
0.8
Time Staps (1.6)
Time Stops (1.6)
Time Stops (1e6)
Time Stops (1e6)
Time Stops (1o5)
Figure? 2: Learned Q1 function in TD3. We can observe that non-uniform sampling schemes fail to avoid the bootstrapping error in the Walker2d-Randon environment.
8

---

Under review as a conference paper at ICLR 2022
uncertainty would be quite time-consuming In addition, these extra models require further parame- ter tuning which may be problematic in the offline setting. Secondly, another critical problem of the wetric-bbsee sample selection method is that how exact is the metric for selecting good sample. For example, a transition with low uncertainty or high likelihood is not necessarily a good sample for policy learning. Instead, it might be better In particular, a low uncertainty transition may not be good sample, but a high uncertainty transition more likely to be bad one. Thirdly, in the experiment, we compute the priority metric metric for transition (s;, as ri; 81+1) 1) based the current state-action pair (s;, Qz). However, in the batch RL set- ting. the bootstrapping error comes from the OOD bate-action pair ($ 0.+1). Therefore, it might be more effective compute the priority metric based on the next state-action pair (8i-1: 0i+1). We leave these shortcomings for future work.
these metrics as thresholds to filter bad samples.
6 CONCLUSION AND FUTURE WORK
In this paper, we perform empirical analysis on non-uniform sample selection strategies in batch reinforcement learning (RL). In partieular, we compare different variants of Prioritized Experience Replay (PER) based on various heuristic sample priority metrics, including temporal-differeen ror, n-step return, self-imitation learning objective, pseudo-count, uncertainty and likelihood. Our experiments show that con-uniform sampling is also effective in the batch RL setting. However, there is no single priority metric that work in all situations, which shows that the offline datasets are quite complicate and multiple factors that the proposed metric only focus on current state-action pairs and requires extra computations. A future direction is learn a priority metric end-to-end with off-policy policy evaluation (OPE) methods. Another interesting future direction is to utilize unsupervised representation learning, i.e., self-supervised learning, extract useful hidden representations to assist the sample selection task in batch RL.
er-
influence the sample priority. A shortcoming of our work
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, arXiv preprint arXiv:1707.01449 2017.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008.055 05556, 2020.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint IrXiv:
2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009,
Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
Jacob Buckman, Carles Gelada, and Mare Bellemare. The importance of pessimism in fixed- dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov. Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data- driven roboties with reward sketching and batch reinforcement learning. arXiv preprint
1999:10909.12200, 2019.
Moses Charikar, Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium Theory of computing, pp. 380-388, 2002.

---

Under review as a conference paper at ICLR 2022
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, Qing Deng, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint MrXiv:1910.12179 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leamn- ing in a handful of trials using probabilistic dynamics models. arXiv preprint 1805. 2018.
Joao Cunha, Rui Serra, Nuno Lau, Luís Seabra Lopes, and Antóio JR Neves. Batch reinforce- ment learning for robotic soccer using the q-batch update-rule. Journal of Intelligent & Robotic Systems, 80(3):385-339 2015.
Tim De Bruin, Jens Koher, Karl Tuyls, and Robert Babuska. Experience selection in deep reinforce- ment learning for control. Journal of Machine Learning Research, 19, 2018.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Yan Liu. Learning what data learn. arXiv 2017.
preprint
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Con- ference on Machine Learning, pp. 3061-3071. PMLR, 2020.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. In International Conference Machine Learning, pp. 2021-2030. PMLR, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off- policy evaluation. arXiv preprint arXiv:2103.16599 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. arXiv preprint 1rXiv:2106606860 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. 2018b.
Scott Fujimoto, David Meger, and Doina Precup. An equivalence between loss functions and non- uniform sampling in experience replay. arXiv preprint wxXi:20077060049 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pp. arXiv-2006, 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint /
2018.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2022
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent perience replay in distributed reinforcement learning. International conference on learning representations. 2018.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977 2017
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05951 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning
with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5883. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Off- deep reinforcement learning without exploration. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint M:Xi::1906.00944 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto-
rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01641, 2020.
Ang A Li, Zongqing Lu, and Chenglin Miao. Revisiting prioritized experience replay: A value perspective. arXiv preprint arXiv:2102.03261, 2021.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint drXiv:1509.02971, 2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.
Hao Liu, Alexander Trout, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprini arXiv:1902.00522 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13(1):103-130, 1993.
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. Interna- tional Conference on Machine Learning, pp. 4851-4860 PMLR, 2019.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference Machine Learning, pp.
PMLR, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026 2016.

---

Under review as a conference paper at ICLR 2022
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage--eiighted regression: Simple and scalable off- off-policy reinforcement learning, arXiv preprint arXiv: 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth
AAAI Conference on Artificial Intelligence, 2010.
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. Sample- efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):1211, 2011.
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O Lambert, and Roberto Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. arXiv preprint arXiv:2104.00155 2021.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88-97) 97, 1991.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Emmanuel Rachelson, François Schnitzler, Louis Wehenkel, and Damien Ernst. Optimal sample selection for batch-mode reinforcement learning. In Proceedings of the 3rd International Confer- ence on Agents and Artificial Intelligence (ICAART 2011), 1), 2011.
Aniruddh Raghu, Reinforcement learning for sepsis treatment: Baselines and analysis. 2019.
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural reinforce- ment learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offs in off-policy learning. In International Conference Artificial Intelligence and Statistics, pp. 34-44 PMLR, 2020.
Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: unified approach monte carlo simulation, randomized optimization and machine learning. Information Science & Statis- fics, Springer Verlag, NY, 2004.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv: 15| 115511109552 2015.
Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
Peiquan Sun, Wengang Zhou, and Houqiang Li. Attentive experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5900-5907, 2020.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Yunhao Tang. Self-imitation learning via generalized lower bound q-learning. arXiv preprint arXiv:2006.07442, 2020.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted trans- fer samples in reinforcement learning. In International Conference on Machine Learning, pp- 4936-4945 PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2022
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning, arXiv preprint arXiv:1911.06854 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint /91.:: 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv:1805.04689 2(5):6, 2018.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint 10Xiv 2020.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience, arXiv preprint 2..xi:2001.138805 2020.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
Here, we introduce some details of our experiments. For the PER model, we use the hyperparameters recommended in the origin paper (Schaul et al., 2015), and return metric, we select the N to be 20. For the GSIL metric, we follow the same experiment setup as in the origin GSIL paper (Tang, 2020). For the pseudo-count metric, we select the key dimension for Simhash by computing the 25% quantile and 50% quantile number (see Table 3). A small key dimension would lead to too many collisions while a large key dimension would lead to sparse collisions. We highlight the selected parameter for each environment we used in the experiment. For the uncertainty metric, we train an probabilistic ensemble with 7 models with early stopping- We use the default training parameters in the MBRL-LIB (Pineda et al., 2021) package. For the likelihood metric, we use the official FBRC (Kostrikov et al., 2021) code to learn the behavior policy.
0.6, 3 0.4. For the N-step
Table 3: Quantile number of the pseudo-count of each state-action pair in the offline dataset.
Key Dimension
16
24
32
48
64
128
Quantile
25%
50%
25%
50%
25%
50%
25%
50%
25%
50%
25%
50%
Medium Random
HalfCheetah
29
1
1
Hopper
1068
4377
321
1697
62
343
6
42
11
1
Walker2d
56
2
3
HalfCheetah
670
3288
112
188
Hopper
562
1931
125
626
41
208
7
43
13
1
1
Walker2d
99
443
6
40
1
7
1
Medium
Replay
HalfCheetah
7
29
1
1
1
1
Hopper
57
205
8
2
10
Walker2d
6
18
1
1
1
Medium
Expert
HallCheetah
838
4309
192
1241
23
210
4
43
8
1
Hopper
405
1354
73
326
26
135
Walker2d
218
1069
14
90
3
23
1
2
2
Expert
HalfCheetah
638
3831
149
947
23
198
2
12
Hopper
1032
3683
175
718
25
123
4
24
8
1
Walker2d
160
655
17
103
4
25
3
1
14


Reviews:
[{"review_id": "rnciN3kDKHR", "reviewer": "Reviewer_8KRS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1.\tThe paper is clearly written and easy to follow. It is self-contained and provides a good overview of existing literature.  \n \nWeaknesses\n1.\tI think the findings in this paper are not novel and significant. I guess the paper uses different sampling strategies to run TD3BC until convergence or for a fixed number of steps. In this case, isn’t the only difference between different sampling strategies the weighting on the loss function? In the online setting, different sampling strategies not only affect the learning at each time step but also the transitions the agent would see in the future. However, in the offline setting, the dataset is fixed, so it seems like the only impact is the weighting on the loss function? If this is the case, the paper basically shows that different weighting results in different performance and I don’t think that is a novel or significant finding.  \n2.\tThe empirical study is inconclusive. The paper mentions that the D4RL dataset is too complicated to draw any clear conclusion. Since the goal of the paper is not to propose a SOTA algorithm on “deep” RL environments, why not consider some smaller offline dataset or even discrete toy environments and see if any clear conclusion can be drawn? Moreover, in a small discrete environment, one might be able to show some relationship, e.g., using metric A performs well in environment B because it does C. I think that would be more informative than running on large complicated environments. \n\nQuestions: \n1.  I am not sure what is the reason to use N-step return? It seems like N-step TD error might be a better choice since it matches the reasoning with PER. Also, what are the reasons for using Likelihood? It does not make sense to use only the state-action pairs closed to the behavior policy. \n2. I am not sure what is the bootstrapping error in Section 5 and how it is connected to $Q_1$ in Figure 2? \n3. The term “sampling selection” is confusing to me. Given a fixed dataset, even if we put a small sampling probability to a particular sample, we might still use the sample a sufficient number of times. I think it might be interesting to see a harder “sampling selection”: we drop a sample if the metric of the sample is below a threshold.  \n\n", "summary_of_the_paper": "The paper empirically investigates several sample selection strategies in offline RL based on TD3BC and the PER framework, including TD errors, N-step return, Generalized SIL, Pseudo-count, Uncertainty, and Likelihood. The paper finds that some sampling strategies improve the performance on D4RL dataset but they fail to avoid the bootstrapping error. ", "summary_of_the_review": "I am leaning towards a recommendation to reject the paper since the empirically findings in the paper are inconclusive and not significant as mentioned in the Main Review section. \n\n---\nAfter Rebuttal: Thank you for the response. My concerns are not addressed and I think the paper is not ready for publication. I hope the authors can keep improving the paper since it is an interesting topic. I will keep my original score and evaluation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oJam7W__XJu", "reviewer": "Reviewer_VhFt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\n- Sensible question: is a widely used method in online RL also useful in offline RL?\n- While always inspired by online RL, I think at least some variants considered are not straightforward but rather an adaptation of an idea that works in online RL outside of PER. One example is using an idea that favors exploration in online RL to develop a sample weighting strategy.\n- Empirical study are interesting imho.\n- The paper offers interesting perspectives for further work.\n\n**Weak points:**\n- Imho, this paper assumes that the reader is extremely familiar with the state of the art in experience replay. I think this makes the paper perhaps harder than necessary to read. On the other hand, that type of paper must necessarily introduce a lot of algorithms quickly. I could not come up with a solution. Here are a few concrete examples:\n   - Unless I missed it, some terms are never defined, such as behavior policy or temporal difference.\n   - A lot of algorithms and variations are discussed and some are only introduced with a line of text. \n- The empirical experiments are well done, but limited in scope. The paper considers only one algorithm and three environments.\n- The paper requires significant additional proofreading. A few typos are listed below.\n- I saw no mention that the code would be provided, although the paper strongly emphasizes the empirical validation.\n\n**Questions:**\n- When reading the beginning of the paper, I thought it was an empirical evaluation of existing online RL strategies in batch RL. However, it seems that some new methods are also considered, such as Pseudo-Count for example. I am surprised this is not highlighted better as a contribution of the paper.\n\n**Typos:**\n- end of page 3: is could be the value\n- p5: samples with higher n-step return is\n- p5: optimizing following actor-critic loss functions\n- p5: motivation of SIL is intuitive that", "summary_of_the_paper": "This paper empirically studies six variants of prioritized experience replay, typically used in online RL, in a batch RL setting. The comparison is performed using TD3BC on three D4RL Mujoco benchmark environment times 5 data sets. The experiments study the performance and bootstrapping errors. Among other things, it is shown that non-uniform sampling strategies are also interesting in a batch RL setting. The paper also discusses some shortcomings of these approaches and future directions.", "summary_of_the_review": "Although mostly an empirical study, the research topic is interesting. I think the paper undersells a bit its contribution actually, as some variants considered are not trivial. It also provides interesting future directions. On the other hand, it is as bit hard to read, requires more proof-reading and the scale of the experiments is not big. Thus I believe it is a bit below the acceptance limit.\n\n*After Rebuttal*: Thank you for your response. It seems that few changes have been made to the manuscript. I will keep my score unchanged. I wish the authors the best to improve the manuscript.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gqGrn3zY_ti", "reviewer": "Reviewer_iELi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Overall the questions asked in this paper are worthwhile; I don't believe that PER has been applied to the offline setting before, so examining the efficacy of this is certainly an interesting direction.\n\nCons:\n* This paper in its current state doesn't meet the standard for publication at a top venue as it lacks rigor for the following reasons:\n   * Only one offline model-free RL algorithm was tested. More algorithms (e.g., CQL, MOPO, SAC etc.) should have been used. Perhaps using PER with a model-based method (and prioritising different hallucinated `(s,a,r,s')` tuples) could ameliorate the requirement of having a pessimistic MDP + reward penalty? Perhaps PER could be useful during model training?\n   * Limited choice (6) of prioritization strategies; there should be more than this\n   * Very low number of seeds (this should be at least 5)\n   * No significance testing is done; are the blue highlighted results significant under a statistical test (e.g., Wilcoxon signed rank test)?\n* The results themselves are not that compelling, and it doesn't appear that PER seems to help over uniform sampling much at all. I think a more directed piece of work that aims at creating a PER scheme that addresses distributional shift issues in offline RL, and show strong performance, would have been more suitable. Or perhaps work that seeks to explain why PER doesn't help that much, perhaps with some illustrative examples.\n* The choice of citations is inappropriate at times. For instance, when model-based RL is introduced, the authors' cite recent work by Kaiser et al., but this is not a canonical nor an archetypal algorithm. Instead something like Dyna [1] or a review paper would be more appropriate. There are other examples of this littered throughout (e.g., citing Chua et al. for epistemic uncertainty), so the authors would do well to address this.\n\nNits:\n* There are a few grammatical errors in this paper, for instance the use of 'pioneer' v.s. 'pioneering' at the top of section 3.2, and the repetition of 'metric' on page 9.\n\n-------------------------------------\n\nI have read the author's response but did not see much of what I requested in the updated manuscript (e.g., significance testing, additional algorithms, more compelling results, etc.). Therefore my rating stays the same.\n\nRefs:\n\n[1] \"Integrated Architectures for Learning, Planning and Reacting based on Dynamic Programming\", R. Sutton 1990.", "summary_of_the_paper": "In this paper the authors' investigate the application of prioritized experience replay (PER) applied to offline/batch RL. They trial a variety of different priority metrics from the literature on the TD3+BC model-free algorithm and report results on the D4RL suite.", "summary_of_the_review": "Overall this paper asks an interesting question, but fails in its execution for the reasons I have listed above. I believe it would be more suitable for publication at a workshop.\n\nBased on this I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fhAfbRokgxV", "reviewer": "Reviewer_YFrP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- Table 1 (List of proposed metrics) is well-explained and clarifies the considered sampling variants.\n\n\nWeakness\n\n- As shown in the numerical results, sampling in offline RL may not be a critical issue compared with online learning. Unlike online learning, most of the samples in a fixed dataset may be visited.\n\n- The proposed method lacks novelty. This paper does not propose any new algorithm for offline RL settings. Instead, this work combines TD3BC with the existing six variants of metric.\n\n- While TD3BC has an advantage in its simplicity in implementation, other backbone algorithms for model-free offline RL should also be considered. \n\n\nQuestion\n\n- What will be the result if the following metric is used with proportional PER instead of rank PER: N-step Return, Uncertainty, and Likelihood? ", "summary_of_the_paper": "This paper investigated the effect of non-uniform sampling in an offline RL setting. Using TD3BC (Fujimoto and Gu, 2021) as a backbone offline RL algorithm, the authors applied prioritized experience replay (PER) to the sampling of TD3AC with variants of priority metric, including standard TD error, rank-based return, pseudo-count using a hash table, and the other three metrics. The authors insist that non-uniform sampling can be helpful in offline RL compared with usual uniform sampling. They also found that there is no one outperforming metric for prioritized sampling in offline RL settings.", "summary_of_the_review": "Although this work raised an interesting question, this paper combines the existing algorithm and sampling metrics, which lack both novelty and empirical significance.\n\n\n-------------------- Post Rebuttal ---------------------\n\nThanks for the response. I read the response carefully, and I maintain my original score. I hope the authors to further improve this work in the future.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper empirically benchmarks multiple sample selection strategies for offline RL based on the prioritized experience replay framework, including TD errors, N-step return, Generalized SIL, Pseudo-count, Uncertainty, and Likelihood. These are all benchmarked for the base algorithm TD3BC. The experiments study the performance and bootstrapping errors. Among other things, it is shown that non-uniform sampling strategies are also interesting in a batch RL setting. The authors show that non-uniform sampling can be helpful in offline RL compared to uniform sampling but they fail to avoid bootstrap error. They also found that there is no one outperforming metric for prioritized sampling in offline RL settings.

The reviewers are in agreement that the question studied is a sensible and interesting one - Are PER strategies which are effective in online RL also useful for batch RL? The overall study conducted by the paper is clear and well presented. 

While the study/benchmark and the results presented is clear, the reviewers point out the following shortcomings
1. The study is not comprehensive for this work to become a definitive exploration of this space of ideas. Only algorithm has been tested with these ideas. 
2. The results of the study are unfortunately inconclusive - while there are benefits these are achieved via different strategies and as mentioned by the paper no clear conclusions can be drawn. 

Since the paper is targeted purely as a benchmark, the originality aspect of the paper is naturally low. For benchmark papers in that case the impact factor squarely falls on comprehensiveness of the study and the emergence of some clear conclusions to further research in that area. The reviewers unanimously believe the paper falls short in both respects and therefore the decision. 

Hopefully the authors can consider the feedback provided and incorporate it to improve the paper.
Decision: Reject