Benchmarks for Deep Off-Policy Evaluation

1 INTRODUCTION . Reinforcement learning algorithms can acquire effective policies for a wide range of problems through active online interaction , such as in robotics ( Kober et al. , 2013 ) , board games and video games ( Tesauro , 1995 ; Mnih et al. , 2013 ; Vinyals et al. , 2019 ) , and recommender systems ( Aggarwal et al. , 2016 ) . However , this sort of active online interaction is often impractical for real-world problems , where active data collection can be costly ( Li et al. , 2010 ) , dangerous ( Hauskrecht & Fraser , 2000 ; Kendall et al. , 2019 ) , or time consuming ( Gu et al. , 2017 ) . Batch ( or offline ) reinforcement learning , has been studied extensively in domains such as healthcare ( Thapa et al. , 2005 ; Raghu et al. , 2018 ) , recommender systems ( Dudík et al. , 2014 ; Theocharous et al. , 2015 ; Swaminathan et al. , 2017 ) , education ( Mandel et al. , 2014 ) , and robotics ( Kalashnikov et al. , 2018 ) . A major challenge with such methods is the off-policy evaluation ( OPE ) problem , where one must evaluate the expected performance of policies solely from offline data . This is critical for several reasons , including providing high-confidence guarantees prior to deployment ( Thomas et al. , 2015 ) , and performing policy improvement and model selection ( Bottou et al. , 2013 ; Doroudi et al. , 2017 ) . The goal of this paper is to provide a standardized benchmark for evaluating OPE methods . Although considerable theoretical ( Thomas & Brunskill , 2016 ; Swaminathan & Joachims , 2015 ; Jiang & Li , 2015 ; Wang et al. , 2017 ; Yang et al. , 2020 ) and practical progress ( Gilotte et al. , 2018 ; Nie et al. , 2019 ; Kalashnikov et al. , 2018 ) on OPE algorithms has been made in a range of different domains , there are few broadly accepted evaluation tasks that combine complex , high-dimensional problems ∗Equally major contributors . †Policies and evaluation code are available at https : //github.com/google-research/deep_ ope . See Section 5 for links to modelling code . commonly explored by modern deep reinforcement learning algorithms ( Bellemare et al. , 2013 ; Brockman et al. , 2016 ) with standardized evaluation protocols and metrics . Our goal is to provide a set of tasks with a range of difficulty , excercise a variety of design properties , and provide policies with different behavioral patterns in order to establish a standardized framework for comparing OPE algorithms . We put particular emphasis on large datasets , long-horizon tasks , and task complexity to facilitate the development of scalable algorithms that can solve high-dimensional problems . Our primary contribution is the Deep Off-Policy Evaluation ( DOPE ) benchmark . DOPE is designed to measure the performance of OPE methods by 1 ) evaluating on challenging control tasks with properties known to be difficult for OPE methods , but which occur in real-world scenarios , 2 ) evaluating across a range of policies with different values , to directly measure performance on policy evaluation , ranking and selection , and 3 ) evaluating in ideal and adversarial settings in terms of dataset coverage and support . These factors are independent of task difficulty , but are known to have a large impact on OPE performance . To achieve 1 , we selected tasks on a set of design principles outlined in Section 3.1 . To achieve 2 , for each task we include 10 to 96 policies for evaluation and devise an evaluation protocol that measures policy evaluation , ranking , and selection as outlined in Section 3.2 . To achieve 3 , we provide two domains with differing dataset coverage and support properties described in Section 4 . Finally , to enable an easy-to-use research platform , we provide the datasets , target policies , evaluation API , as well as the recorded results of state-of-the-art algorithms ( presented in Section 5 ) as open-source . 2 BACKGROUND We briefly review the off-policy evaluation ( OPE ) problem setting . We consider Markov decision processes ( MDPs ) , defined by a tuple ( S , A , T , R , ρ0 , γ ) , with state space S , action space A , transition distribution T ( s′|s , a ) , initial state distribution ρ0 ( s ) , reward function R ( s , a ) and discount factor γ ∈ ( 0 , 1 ] . In reinforcement learning , we are typically concerned with optimizing or estimating the performance of a policy π ( a|s ) . The performance of a policy is commonly measured by the policy value V π , defined as the expected sum of discounted rewards : V π : = Es0∼ρ0 , s1 : ∞ , a0 : ∞∼π [ ∞∑ t=0 γtR ( st , at ) ] . ( 1 ) If we have access to state and action samples collected from a policy π , then we can use the sample mean of observed returns to estimate the value function above . However , in off-policy evaluation we are typically interested in estimating the value of a policy when the data is collected from a separate behavior policy πB ( a|s ) . This setting can arise , for example , when data is being generated online from another process , or in the purely offline case when we have a historical dataset . In this work we consider the latter , purely offline setting . The typical setup for this problem formulation is that we are provided with a discount γ , a dataset of trajectories collected from a behavior policy D = { ( s0 , a0 , r0 , s1 , . . . ) } , and optionally the action probabilities for the behavior policy πB ( at|st ) . In many practical applications , logging action propensities is not possible , for example , when the behavior policy is a mix of ML and hard-coded business logic . For this reason , we focus on the setting without propensities to encourage future work on behavior-agnostic OPE methods . For the methods that require propensities , we estimate the propensities with behavior cloning . The objective can take multiple flavors , as shown in Fig . 1 . A common task in OPE is to estimate the performance , or value , of a policy π ( which may not be the same as πB ) so that the estimated value is as close as possible to V π under a metric such as MSE or absolute error . A second task is to perform policy selection , where the goal is to select the best policy or set of policies out of a group of candidates . This setup corresponds to how OPE is commonly used in practice , which is to find the best performing strategy out of a pool when online evaluation is too expensive to be feasible . 3 DOPE : DEEP OFF-POLICY EVALUATION . The goal of the Deep Off-Policy Evaluation ( DOPE ) benchmark is to provide tasks that are challenging and effective measures of progress for OPE methods , yet is easy to use in order to better facilitate research . Therefore , we design our benchmark around a set of properties which are known to be difficult for existing OPE methods in order to gauge their shortcomings , and keep all tasks amenable to simulation in order for the benchmark to be accessible and easy to evaluate . 3.1 TASK PROPERTIES . We describe our motivating properties for selecting tasks for the benchmark as follows : High Dimensional Spaces ( H ) High-dimensionality is a key-feature in many real-world domains where it is difficult to perform feature engineering , such as in robotics , autonomous driving , and more . In these problems , it becomes challenging to accurately estimate quantities such as the value function without the use of high-capacity models such a neural networks and large datasets with wide state coverage . Our benchmark contains complex continuous-space tasks which exercise these challenges . Long Time-Horizon ( L ) Long time horizon tasks are known to present difficult challenges for OPE algorithms . Some algorithms have difficulty doing credit assignment for these tasks . This can be made worse as the state dimension or action dimension increases . Sparse Rewards ( R ) Sparse reward tasks increase the difficulty of credit assignment and add exploration challenges , which may interact with data coverage in the offline setting . We include a range robotics and navigation tasks which are difficult to solve due to reward sparsity . Temporally extended control ( T ) The ability to make decisions hierarchically is major challenge in many reinforcement learning applications . We include two navigation tasks which require high-level planning in addition to low-level control in order to simulate the difficulty in such problems . 3.2 EVALUATION PROTOCOL The goal of DOPE to provide metrics for policy ranking , evaluation and selection . Many existing OPE methods have only been evaluated on point estimates of value such as MSE , but policy selection is an important , practical use-case of OPE . In order to explicitly measure the quality of using OPE for policy selection , we provide a set of policies with varying value , and devise two metrics that measure how well OPE methods can rank policies . For each task we include a dataset of logged experiencesD , and a set of policies { π1 , π2 , ... , πN } with varying values . For each policy , OPE algorithms must use D to produce an estimate of the policy ’ s value . For evaluation of these estimates , we provide `` ground truth values '' { V π1 , V π2 , ... , V πN } that are computed by running the policy forM ≥ 1000 episodes , where the exact value ofM is given by the number of episodes needed to lower the error bar on the ground truth values to 0.666 . The estimated values are then compared to these ground truth values using three different metrics encompassing both policy evaluation and selection ( illustrated in Figure 2 ; see Appendix A.1 for mathematical definitions ) . Absolute Error This metric measures estimate accuracy instead of its usefulness for ranking . Error is the most commonly used metric to assess performance of OPE algorithms . We opted to use absolute error instead of MSE to be robust to outliers . Regret @ k This metric measures how much worse the best policies identified by the estimates are than the best policy in the entire set . It is computed by identifying the top-k policies according to the estimated returns . Regret @ k is the difference between the actual expected return of the best policy in the entire set , and the actual value of the best policy in the top-k set . Rank correlation This metric directly measures how well estimated values rank policies , by computing the correlation between ordinal rankings according by the OPE estimates and ordinal rankings according to the ground truth values . 4 DOMAINS . DOPE contains two domains designed to provide a more comprehensive picture of how well OPE methods perform in different settings . These two domains are constructed using two benchmarks previously proposed for offline reinforcement learning : RL Unplugged ( Gulcehre et al. , 2020 ) and D4RL ( Fu et al. , 2020 ) , and reflect the challenges found within them . The DOPE RL Unplugged domain is constrained in two important ways : 1 ) the data is always generated using online RL training , ensuring there is adequate coverage of the state-action space , and 2 ) the policies are generated by applying offline RL algorithms to the same dataset we use for evaluation , ensuring that the behavior policy and evaluation policies induce similar state-action distributions . Using it , we hope to understand how OPE methods work as task complexity increases from simple Cartpole tasks to controlling a Humanoid body while controlling for ideal data . On the other hand , the DOPE D4RL domain has : 1 ) data from various sources ( including random exploration , human teleoperation , and RL-trained policies with limited exploration ) , which results in varying levels of coverage of the state-action space , and 2 ) policies that are generated using online RL algorithms , making it less likely that the behavior and evaluation policies share similar induced state-action distributions . Both of these result in distribution shift which is known to be challenging for OPE methods , even in simple tasks . So , using it we hope to measure how well OPE methods work in more practical data settings .

Summary: This article proposes a benchmark of off-policy evaluation, which provides different metrics for policy ranking, evaluation and selection. Offline metrics are provided by evaluating the value function of logged data, and then evaluating absolute error, rank correlation and regret. Verify the effectiveness of different offline evaluation methods. This article provides two evaluation scenarios, one is DOPE RL unplugged, and the other is D4RL. In the experiment, the author verified the benchmark proposed in this article in the MuJoCo environment to evaluate the effectiveness of different offline evaluation methods.