Benchmarking Sample Selection Strategies for Batch Reinforcement Learning

1 INTRODUCTION . A key question in machine learning is to select the suitable training samples ( Katharopoulos & Fleuret , 2018 ) . Many prior works proved that an appropriate sample selection strategy , i.e. , removing redundant data or selecting samples according to their hardness , usually significantly improves the learning efficiency and final performance ( Bengio et al. , 2009 ; Schaul et al. , 2015 ; Fan et al. , 2017 ) . Similarly , sample selection also plays a crucial role in reinforcement learning ( RL ) ( De Bruin et al. , 2018 ) . A notable example is the sample selection problem for experience replay ( ER ) in off-policy RL ( Fedus et al. , 2020 ) , where an agent reuses stored experiences from a buffer while interacting with the environment . For example , Prioritized Experience Replay ( PER ) ( Schaul et al. , 2015 ) , which samples high error transitions more frequently , is now widely used in different state-of-theart ( SOTA ) off-policy RL algorithms ( Barth-Maron et al. , 2018 ; Hessel et al. , 2018 ; Kapturowski et al. , 2018 ) . Batch RL , also known as offline RL , refers to the problem of learning a near-optimal policy from a fixed offline buffer ( Lange et al. , 2012 ) . Due to the wide availability of logged-data and the increasing computing power , batch RL holds the promise for successful real-world applications ( Levine et al. , 2020 ) . Especially for the scenarios where collecting online data is time-consuming , dangerous or unethical , i.e. , robotics , self-driving cars and medical treatments ( Gulcehre et al. , 2020 ) . While most off-policy RL algorithms are applicable in the offline setting , they usually suffer from the bootstrapping error ( Fujimoto et al. , 2018b ; Kumar et al. , 2019 ) due to out-of-distribution ( OOD ) state-action pairs . Different solutions are proposed to mitigate this problem , i.e. , adding constraints ( Fujimoto et al. , 2018b ; Wu et al. , 2019 ) , imitating behavior policy ( Chen et al. , 2019 ; Zolna et al. , 2020 ) , learning dynamics models ( Yu et al. , 2020 ; Kidambi et al. , 2020 ; Argenson & Dulac-Arnold , 2020 ) , incorporating uncertainties ( Wu et al. , 2021 ) , learning ensembles ( Agarwal et al. , 2020 ) , or learning pessimistic value functions ( Kumar et al. , 2020 ; Buckman et al. , 2020 ; Jin et al. , 2021 ) . Unlike the wide application of PER in online off-policy RL , the non-uniform sampling strategy is largely ignored in recent batch RL algorithms . Inspired by the success of PER ( Schaul et al. , 2015 ) in the online setting , one natural question to ask is that what is the counterpart of PER in batch RL ? This problem is appealing for several reasons : ( 1 ) In some real-world applications , the size of the offline dataset is usually increasing though we have no access to the real environment . For example , we would have ever-growing medical records from the hospitals ( Raghu , 2019 ) , or recorded videos from dash cams ( Yu et al. , 2018 ) . ( 2 ) As the D4RL offline benchmark ( Fu et al. , 2020 ) shows that – for most existing methods , more samples do not necessarily lead to better performance . That is , a batch RL agent sometimes under-performs in a large combined buffer . Given the success achieved by PER in online RL , we are curious that if similar technique could help to develop more robust batch RL agents ( Fujimoto et al. , 2020 ) . Some prior works proposed different sample selection strategies in batch RL . For example , Optimal Sample Selection ( OSS ) ( Rachelson et al. , 2011 ) introduced a meta-learning algorithm which selects optimal samples according to a cross entropy search method for tree-based Fitted Q-Iteration ( FQI ) ( Ernst et al. , 2005 ) with a known dynamics model . Recently , Best-Action Imitation Learning ( BAIL ) ( Chen et al. , 2019 ) proposed to select high-performing samples with a learned value function in behavior cloning . Another related line of research is to reweight sampled transitions . For example , Advantage-Weighted Regression ( AWR ) ( Peng et al. , 2019 ) and Advantage-weighted Behavior Model ( ABM ) ( Siegel et al. , 2020 ) both used reward-weighted regression ( Peters et al. , 2010 ) to learn the policy . Further , Uncertainty Weighted Actor Critic ( UWAC ) ( Wu et al. , 2021 ) adopted a dropout-uncertainty estimation method ( Gal & Ghahramani , 2016 ) and reweighted samples according to their estimated uncertainties . However , it is unclear which sample selection strategy is preferred in batch RL , thereby demanding more investigations . To this end , in this work , we study the sample selection problem in batch RL ( De Bruin et al. , 2018 ) . We follow the PER framework by assigning samples with different priorities ( Schaul et al. , 2015 ) . Crudely , there are two types of metrics to evaluate sample importance . Firstly , we can design a heuristic metric based on our prior knowledge , i.e. , temporal-difference ( TD ) error . Secondly , we can use an end-to-end approach to learn a metric for each sample , for example , we can use offpolicy evaluation ( OPE ) methods ( Voloshin et al. , 2019 ; Fu et al. , 2021 ) to evaluate the goodness of current policy as the metric . However , existing OPE methods usually need to learn a model for each evaluation ( Le et al. , 2019 ) , which makes the learning-based metric approach to be computationally expensive . Therefore , in this paper , we focus on the heuristic metric-based approach and leave the learning-based metric approach for future work . In particular , we benchmark six variants of PER based on different heuristic priority metrics in order to understand which sample selection strategy might be preferred in batch RL . 2 PRELIMINARIES . 2.1 BATCH REINFORCEMENT LEARNING . We consider the standard Markov Decision Process ( MDP ) ( Puterman , 2014 ) M = 〈S , A , T , r , γ〉 . S and A denote the state and action spaces . T ( s′|s , a ) and r ( s , a ) represent the dynamics and reward function , and γ ∈ [ 0 , 1 ) is the discount factor . A policy π ( a|s ) defines a mapping from state to distributions over actions . The goal of an RL agent is to learn a policy π ( a|s ) that maximizes the expected cumulative discounted reward J ( π ) : = Eπ [ ∑∞ t=0 γ trt ] . The performance of the policy can be defined by the Q-function Qπ ( s , a ) : = Eπ [ ∑∞ t=0 γ trt|s0 = s , a0 = a ] and value function V π ( s ) : = Eπ [ ∑∞ t=0 γ trt|s0 = s ] , where Eπ [ · ] is the expected result when following the policy π . Once given the optimalQ-functionQ∗ ( s , a ) = arg maxπ Qπ ( s , a ) , we can derive an optimal policy as π∗ ( a|s ) = arg maxaQ∗ ( s , a ) ( Sutton & Barto , 2018 ) . In ( tabular ) Q-learning , we solve for the Q∗ by iterating the Optimality Bellman Operator T ∗ , defined as T ∗Q ( s , a ) ← r+ γmaxa′ Q ( s′ , a ) ( Bertsekas & Tsitsiklis , 1996 ) . To solve problems with large state space , we can use a parameterized Q-function Qθ ( s , a ) to approximate Q∗ . In practice , we optimize the parameters by a µ-weighted L2 projection Πµ ( Q ) ( Fu et al. , 2019 ) , which minimizes the empirical Bellman error loss : Πµ ( Q ) = minθ E ( s , a , r , s′ ) ∼µ [ ( T ∗Qθ ( s , a ) −Qθ ( s , a ) ) 2 ] . Batch RL , also known as offline RL , aims to learn a near-optimal policy from a fixed dataset ( Lange et al. , 2012 ) D , representing a series of timestep tuples ( st , at , rt , st+1 ) . Furthermore , the dataset can be collected by agents with different policies from different control tasks , including non-RL policies , such as human demonstrations ( Levine et al. , 2020 ) . Some early works such as Fitted QIteration ( FQI ) ( Ernst et al. , 2005 ) and Neural Fitted Q-Iteration ( NFQ ) ( Riedmiller , 2005 ) , which formulate the original RL problem as a sequence of supervised regression problem , are shown to be sample efficient in solving various real-world problems ( Pietquin et al. , 2011 ; Cunha et al. , 2015 ) . On the other hand , some recent studies show that current deep off-policy RL algorithms usually fail in challenging batch RL problems due to bootstrapping error ( Fujimoto et al. , 2018b ; Kumar et al. , 2019 ) . That is , the OOD action a′ might lead to unrecoverable over-estimation error through max operator in the Bellman backup . The over-estimation problem is particularly detrimental in the offline setting where the agent has no access to interact with the real environment to get the feedback to fix the estimation error ( Kumar et al. , 2020 ) . 2.2 NON-UNIFORM SAMPLING WITH EXPERIENCE REPLAY . Experience replay ( ER ) ( Lin , 1992 ) has been a de facto component for modern deep RL algorithms . By reusing previous collected experiences from the replay buffer , ER helps to reduce sample complexity and stabilize training in off-policy RL ( Mnih et al. , 2013 ; Lillicrap et al. , 2015 ) . For some real-world problems where collecting online data is expensive or time consuming , i.e. , robotics or self-driving cars , the ability to learn good policies from pre-collected data is crucial for successful real-world applications ( Cabi et al. , 2019 ) . A number of works ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Liu et al. , 2019 ; Sun et al. , 2020 ; Fujimoto et al. , 2020 ) show that applying different non-uniform sampling strategies in ER can significantly improve the learning efficiency . Especially for problems where there are many redundant transitions ( Schaul et al. , 2015 ) , or the reward signal is sparse ( Andrychowicz et al. , 2017 ) . A notable example is the Prioritized Experience Replay ( PER ) ( Schaul et al. , 2015 ) , where the probability of sampling a certain transition ( st , at , rt , st+1 ) is proportional to the absolute TD error . However , it is still an open question that which priority metric is optimal to value the importance of samples ( De Bruin et al. , 2018 ) . 3 RELATED WORK . 3.1 SAMPLE SELECTION WITH EXPERIENCE REPLAY . Many prior works have sought to analyze the mechanism of experience replay , both empirically ( De Bruin et al. , 2018 ; Fedus et al. , 2020 ) and theoretically ( Fujimoto et al. , 2020 ; Li et al. , 2021 ) . Similar to our work , ( De Bruin et al. , 2018 ) investigated a number of proxies , i.e. , age , TD error , and exploration noise , to decide which experience to store in the replay buffer and how to sample from the replay buffer . Likewise , ( Fu et al. , 2019 ) used a “ unit-testing ” framework to study Q-learning with function approximators and found that a sampling scheme with wider coverage improves performance . Further , ( Fedus et al. , 2020 ) conducted a systematic analysis of experience replay in Q-learning methods and provided two insights – ( 1 ) Increasing the buffer capacity is preferable , because it has a broader data coverage . ( 2 ) Decreasing the age of the oldest policy improves the performance , because it contains more high-quality on-policy data . While these insights help us to understand the mechanism of experience replay , they are less practical in the batch RL setting , where the given offline dataset is fixed ( Lange et al. , 2012 ) . A number of variants of ER have been introduced to further improve the learning efficiency ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Novati & Koumoutsakos , 2019 ; Liu et al. , 2019 ; Sun et al. , 2020 ) . One of the most popular variants is the Prioritized Experience Replay ( PER ) ( Schaul et al. , 2015 ) , which proposed to use the absolute TD error |δ ( i ) | as the priority metric and the probability of sampling the i-th transition is : p ( i ) = pαi∑ j p α j , pi = |δ ( i ) |+ or pi = 1 rank ( i ) , ( 1 ) where α is a hyper-parameter , is a small positive constant to avoid zero priority , priority pi is the value of |δ ( i ) | or the inverse rank of |δ ( i ) | . In addition , Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) proposed to re-label visited state as goal states to overcome hard exploration problems with sparse rewards . Competitive Experience Replay ( CER ) Liu et al . ( 2019 ) later introduced an automatic exploratory curriculum by formulating an exploration competition between two agents . On the other hand , Remember and Forget Experience Replay ( ReF-ER ) ( Novati & Koumoutsakos , 2019 ) classified samples as “ near-policy ” and “ far-policy ” by the importance weight ρ = π ( a|s ) /µ ( a|s ) between current policy π and the behavior policy µ , and compute gradients only with near-policy samples . Similarly , Attentive Experience Replay ( AER ) ( Sun et al. , 2020 ) selects samples according to the similarities between the transition state and current state . Recently , Loss-Adjusted Prioritized ( LAP ) experience replay ( Fujimoto et al. , 2020 ) built the connection between the non-uniform sampling scheme in PER and loss functions . It shows that any loss function L1 evaluated with uniform sampling ( i ∼ D1 ) is equivalent to another loss function L2 that is evaluated with non-uniformly sampled data ( i ∼ D2 ) : Ei∼D1 [ ∇QL1 ( δ ( i ) ) ] = Ei∼D2 [ pD1 ( i ) pD2 ( i ) ∇QL1 ( δ ( i ) ) ] = Ei∼D2 [ ∇QL2 ( δ ( i ) ) , ] ( 2 ) where δ ( i ) is the TD error of the i-th sample and the two loss functions follows ∇QL2 ( δ ( i ) ) = pD1 ( i ) pD2 ( i ) ∇QL1 ( δ ( i ) ) . Moreover , Valuable Experience Replay ( VER ) ( Li et al. , 2021 ) proved that the absolute TD error |δ ( i ) | is an upper-bound of different value metrics of experiences in Q-learning .

Summary: This paper investigated the effect of non-uniform sampling in an offline RL setting. Using TD3BC (Fujimoto and Gu, 2021) as a backbone offline RL algorithm, the authors applied prioritized experience replay (PER) to the sampling of TD3AC with variants of priority metric, including standard TD error, rank-based return, pseudo-count using a hash table, and the other three metrics. The authors insist that non-uniform sampling can be helpful in offline RL compared with usual uniform sampling. They also found that there is no one outperforming metric for prioritized sampling in offline RL settings.