Benchmarks for Deep Off-Policy Evaluation
Authors: Justin Fu
Mohammad Norouzi
Ofir Nachum
George Tucker
ziyu wang
Alexander Novikov
Mengjiao Yang
Michael R Zhang
Yutian Chen
Aviral Kumar
Cosmin Paduraru
Sergey Levine
Thomas Paine
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
off-policy evaluation
benchmarks

Abstract: Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between papers is difficult because currently there is a lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets can be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of state-of-the-art algorithms and provide open-source access to our data and code to foster future research in this area.    

Published as a conference paper at ICLR 2021
BENCHMARKS FOR DEEP OFF-POLICY EVALUATION
Justin Fu
Mohammad 2orouzi
Ofir Nachum 22
George Tucker*2
Ziyu Wang*
Alexander Novikov
Mengjia Yang²
Michael R. Zhang
Yutian Chen
Aviral Kumar*
Cosmin Paduraru'
Sergey Levine'
Tom Le Paine"3
UC Berkeley
"Google Brain DDeppinnd
justinfu@berkeleg ledu, edu, [mnorouzi, anfirnacchm,,,ie git, tpaine) @googlg.oooooooooooooooooooooooo com
ABSTRACT
Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring on- line interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, com- paring results between papers is difficult because currently there is lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of ofte-oof-hhe-ar algorithms and provide open-source access to our data and code to foster future research in this area
INTRODUCTION
Reinforcement learning algorithms can acquire effective policies for a wide range of problems through active online interaction, such as in robotics (Kober al.. 2013), board games and video games TTsauro 1995) Mnih et al. 2013 Vinyals et al. 2019), and recommender systems (Aggarwal 2016). However, this sort of active online interaction is often impractical for real-world
problems, where active data collection can be costly (Li et al. 2010). dangerous (Hauskrecht & Fraser
2000 Kendall et al.) 2019), or time consuming (Gu et al.) 2017). Batch (or offline) reinforcement learning, has been studied extensively in domains such as healthcare (Thapa et al.) 2005 Raghu cct al. 2018), recommender systems (Dudík et al., 2014 Theocharous et al.] 2015; Swaminathan ct al. 2017), education (Mandel et al. 2014), and robotics (Kalashnikov et al. 2018). A major challenge with such methods is the off policy evaluation (OPE) problem, where one must evaluate the expected performance of policies solely from offline data. This is critical for several reasons, including providing high-con high-confidence guarantees prior to deployment (Thomas et al., 2015). and performing policy improvement and model selection (Bottou et al. 2013 Doroudi al. 2017).
The goal of this paper is to provide a standardized benchmark for evaluating OPE methods. Although considerable theoretical (Thomas & Brunskill] 2016; (Saaminathan & Joachims Diag & Li
2015
Wang et al. 2017: Yang et al., (2200) and practical progress (Gilotte et al. 20188) Nie et al.
2019
Kalashnikov et 2018) on OPE algorithms has been made in a range of different domains,
there are few broadly accepted evaluation tasks that combine complex, high nigh-dimensional problems
"Equally major contributors.
PPliiiiss and evaluation code are available at tttp:///gittubbccom
ope
See Section for links modelling code.

---

Published as a conference paper at ICLR 2021
commonly explored by modern deep reinforcement learning algorithms (Bellemare et al. 2013 Brockman et al. 2016) with standardized evaluation protocols and metrics. Our goal is to provide set of tasks with range of difficulty, excercise variety of design properties, and provide policies with different behavioral patterms in order to establish a standardized framework for comparing OPE algorithms. We put particular emphasis on large datasets, long-horizon tasks, and task complexity to facilitate the development of scalable algorithms that can solve high- gigh-dimensional problems.
Our primary contribution is the Deep Off-Policy Evaluation (DOPE) benchmark. DOPE is designed to measure the performance of OPE methods by 1) evaluating on challenging control tasks with properties known to be difficult for OPE methods, but which occur in real-world scenarios, 2) evaluating across range of policies with different values, to directly measure performance on policy evaluation, ranking and selection, and 3) evaluating ideal and adversarial settings in terms of dataset coverage and support. These factors are independent of task difficulty, but are known to have a large impact on OPE performance. To achieve 1, we selected tasks on set of design principles outlined in Section 3.1 To achieve 2, for each task we include 10 to 96 policies for evaluation and devise an evaluation protocol that measures policy evaluation, ranking, and selection as outlined in Section 3.2. To achieve
provide two domains with differing dataset coverage and support properties described in Section Finally, to enable an easy-to-use research platform, we provide the datasets, target policies, evaluation API, well the recorded results of state-of-the-art algorithms (presented in Section
5)
as open-source.
2 BACKGROUND
We briefly review the off-policy evaluation (OPE) problem setting. We consider Markov decision processes (MDPs), defined by tuple (S,A, T, R. 7). with state space S, action space A, transition distribution T(s'ss, a), initial state distribution (s).. reward function R(s, a) and dis- count factor ? E (0, 1]. In reinforcement learning, we are typically concerned with optimizing or estimating the performance of policy π (a|s))
Off-Policy Evaluation
Vm
Data
rT
Dala
The performance of policy is commonly measured by the policy value V*, defined as the expected sum of dis- counted rewards:
TT
Offline Policy Selection
2
πT*
V = E Boo Ro:so
Σ (R
(I)
we have access to state and action samples collected from policy æ, then served returns to estimate the value function above. How- ever, in off-policy evaluation estimating the value of a policy when the data is collected from a separate behavior policy (als). This setting can arise, for example, when data is being generated online from another process, or in the purely offline case when we have historical dataset.
N
use the sample mean of ob-
Figure 1: In Off-Polic Evaluation (top) the goal is to estimate the value of sin- gle policy given only data. Offline Policy Selection (bottom) is closely related problem: given set of N policies, at- tempt pick the best given only data.
are typically interested in
In this work consider the latter, purely offline setting. The typical setup for this problem formulation is that we are provided with discount 7, a dataset of trajectories collected from behavior policy D {(op, ag. ro, .1)...)), and optionally the action probabilities for the behavior policy T (a (s:).. In many practical applications, logging action propensities is not possible, for example, when the behavior policy is a mix ML and hard-coded business logic. For this reason, focus the setting without propensities to encourage future work behavior-agnostic OPE methods. For the methods that require propensities, we estimate the propensities with behavior cloning.
The objective can take multiple flavors, as shown in Fig. I A common task in OPE is to estimate the performance, or value, of policy и (which may not be the wB) that the estimated
2

---

Published as a conference paper at ICLR 2021
value is close as possible to V under metric such as MSE or absolute error. A second task is to perform policy selection, where the goal is to select the best policy or set policies out of a group of candidates. This setup corresponds to how OPE is commonly used in practice, which is to find the best performing strategy out of a pool when online evaluation too expensive to be feasible.
DOPE: DEEP OFF-POLICY EVALUATION
The goal of the Deep Off-Policy Evaluation (DOPE) benchmark is to provide tasks that are challenging and effective measures of progress for OPE methods, yet is easy to use in order to better facilitate research. Therefore, we design our benchmark around set of properties which are known to be difficult for existing OPE methods in order to gauge their shortcomings, and keep all tasks amenable simulation in order for the benchmark to be accessible and easy evaluate.
3.1 TASK PROPERTIES
We describe our motivating properties for selecting tasks for the benchmark as follows:
High Dimensional Spaces (H) figh-dimensionality is a key-feature in many real-world domains where it is difficult to perform feature engineering, such in robotics, autonomous driving, and more. In these problems, becomes challenging to accurately estimate quantities such as the value function without the use of sigh-capactty models such neural networks and large datasets with wide state coverage. Our benchmark contains complex continuous space tasks which exercise these challenges.
Long Time-Horizon (L) Long time horizon tasks are known to present difficult challenges for OPE algorithms. Some algorithms have difficulty doing credit assignment for these tasks. This can be made worse as the state dimension action dimension increases.
Sparse Rewards (R) Sparse reward tasks increase the difficulty of credit assignment and add exploration challenges, which may interact with data coverage in the offline setting. We include range robotics and navigation tasks which are difficult to solve due to reward sparsity.
Temporally extended control (T) The ability make decisions hierarchically is major challenge in many reinforcement learning applications. We include two navigation tasks which require high-level planning in addition low-level control in order simulate the difficulty in such problems.
3.2 EVALUATION PROTOCOL
The goal of DOPE to provide metrics for policy ranking, evaluation and selection. Many existing OPE methods have only been evaluated point estimates of value such as MSE, but policy selection is an important, practical use-case of OPE. In order to explicitly measure the quality of using OPE for policy selection, we provide set of policies with varying value, and devise two metrics that measure how well OPE methods can rank policies.
High correlation
regret
Estimate
Value
Value
Value
Figure 2: Error is natural measure for off-polic evaluation. However for pol- icy selection, it is sufficient to (i) rank the policies as measured by rank correla- tion, or (ii) select policy with the lowest regret.
For each task include dataset of logged experiences D, and set of policies (1. T2, TN }o  with varying values. For each policy, OPE algorithms must use D to produce an estimate of the policy's 's value. For evaluation of these estimates, we provide "ground truth values" (V", V V**} that are computed by running the policy for M > 1000 episodes, where the exact value of M is given by the number of episodes needed to lower the error bar on the ground truth values to 0.666. The estimated values are then compared to these ground truth values using three different metrics encompassing both policy evaluation and selection (illustrated in Figure?? 2 see Appendix A. for mathematical definitions).
Absolute Error This metric measures estimate accuracy instead of its usefulness for ranking. Error is the most commonly used metric to assess performance of OPE algorithms. We opted to use absolute emror instead of MSE be robust outliers.
3

---

Published as a conference paper at ICLR 2021
Regret@k This metric measures how much worse the best policies identified by the estimates than the best policy in the entire set. It is computed by identifying the top-k policies according to the estimated returns. Regret @k is the difference between the actual expected return of the best policy in the entire set, and the actual value of the best policy in the top-k set.
Rank correlation This metric directly measures how well estimated values rank policies, by comput- ing the correlation between ordinal rankings according by the OPE estimates and ordinal rankings
according to the ground truth values.
4 DOMAINS
DOPE contains two domains designed to provide more comprehensive picture of how well OPE methods perform in different settings. These two domains are constructed using two benchmarks previously proposed for offline reinforcement learning: RL Unplugged (Gulcehre et al., 2020) and D4RL (Iu et al. 2020). and reflect the challenges found within them.
The DOPE RL Unplugged domain is constrained in two important ways: 1) the data is always generated using online RL training, ensuring there is adequate coverage of the state-action space, and 2) the policies are generated by applying offline RL algorithms to the same dataset we use for evaluation, ensuring that the behavior policy and evaluation policies induce similar state-action distributions. Using it, we hope to understand how OPE methods work as task complexity increases from simple Cartpole tasks controlling a Humanoid body while controlling for ideal data.
On the other hand, the DOPE D4RL domain has: 1) data from various sources (including random exploration, human teleoperation, and RI-trained policies with limited exploration), which results in varying levels of coverage of the state-action space, and 2) policies that are generated using online RL algorithms, making it less likely that the behavior and evaluation policies share similar induced state-action distributions. Both of these result in distribution shift which is known to be challenging for OPE methods, even in simple tasks. So, using it we hope to measure how well OPE methods work in more practical data settings.
4.1 DOPE RL UNPLUGGED
DeepMind Control Suite (Tassa et al., 2018) is a set of control tasks implemented in MuJoCo (Todorov et al., 2012). We consider the subset included in RL Unplugged. This subset includes tasks that cover range of difficulties. From Cartpole swingup, simple task with a single degree of freedom, to Humanoid run which involves control of complex bodies with degrees of freedom. All tasks use the default feature representation of the system state, including proprioceptive information such as joint positions and velocity, and additional sensor information and target position where appropriate. The observation dimension ranges from 5 to 67.
Datasets and policies We train four offline RL algorithms (D4PG (Barth-Maron et al. 2018),
ABM (Siegel et al. 2020), CRR (Wang et al. 2020) and behavior cloning), varying their hyperparam- combination, we train an agent with 3 random seeds on the DM Control Suite dataset from RL Unplugged and record policy snapshots at exponentially increasing intervals (after 25k learner steps, 50k, 100K, 200K, etc). Following Gulcehre et al. (2020), consider deterministic policy for D4PG and stochastic policies for BC, ABM and CRR. The datasets are taken from the RI. Unplugged benchmark, where they were created by training multiple (online) RL agents and collecting both successful and unsuccessful episodes throughout training. All offline RL algorithms are implemented using the Acme framework (Hoffman et al. 2020).
eters. For each algorithm-taske twy--ttsrrpppppaeeeeeeeee
4.2 DOPE D4RL
Gym-MuJoCC tasks. Gym-MuJoCo consists of several continuous control tasks implemented within the MuJoCo simulator (Todorov et al.) 2012) and provided in the OpenAl Gym (Brockman et al.) 2016) benchmark for online RL. We include the HalfCheetah, Hopper, Walker2D, and Ant tasks. We include this domain primarily for comparison with past works, as array of popular RL.
4

---

Published as a conference paper at ICLR 2021
cartpole swingup
chectah run
finger turn
fish
humanoid walker walker manipulator manipulator walk
Statistics
hard
swim
run
stand
insert ball
insert peg
Dataset size
40K
300K
500K
200K
3M
200K
200K
1.5M
1.5M
State dim.
17
12
24
67
24
44
44
Action dim.
6
2
5
21
6
5
5
Properties
H. L
H.L
H,
H, L
H. L
H.
H. L,
H. L,T
Statistics
maze2d
antmaze halfcheetah
hopper walker ant hammer door
relocate
pen
Dataset size
1/2/4M
IM
1M
IM
IM
1M
11K/IM
7K/IM
10K/IM
5K/500K
datasets
1
1
5
5
5
3
State dim.
4
29
17
11
17
111
46
39
39
45
Action dim.
2
8
3
26
30
24
Properties
T
T,
H
H,
H,
H, R
H,
Table 1: Task statistics for RLUnplugged tasks (top) and D4RL tasks (bottom). Dataset size is the number of (s, a,,,,, tuples. For each dataset, we note the properties possesses: high dimensional spaces (H), long time-horizon (L), sparse rewards (R), temporally extended control (T).
cartpole swingup
fish swim
walker walk
manipulator insert ball
Retrun c2- 9991
AIM
CAR
Checkpoine
Chockpolnz
Checkpeint
Checkpointid
Figure 3: Online evaluation of policy checkpoints for 4 Offline RL algorithms with 3 random seeds. We observe a large degree variability between the behavior algorithms on different tasks. Without online evaluation. tuning the hyperparameters (c.g., choice of Offline RL algorithm and policy checkpoint) is challenging. This highlights the practical importance of Offline policy selection when online evaluation is not feasible. See Figure A.7 for additional tasks.
methods have been evaluated and developed on these tasks (Schulman al. 2015 Lillicrp et 2015 Schulman et al. [017 Fujimoto et al. 2018 Haarnoja et al.) (2018).
Gym-MuJoCC datasets and policies. For each task, in order to explore the effect of varying distributions, we include datasets originally proposed by Fu et al. (2020). 3 correspond to different performance levels of the agent "random", "medium", and "expert". We additionally include mixture of medium and expert dataset, labeled "medium-expert", and data collected from a replay buffer until the policy reaches the medium level of performance, labeled "medium "medium-replay".. For policies, we selected 1I policies collected from evenly-spaced snapshots of training Soft Actor-Critic agent (Haarnoja al. 2208). which covers range of performance between random and expert.
Maze2D and AntMaze tasks. Maze2D and AntMaze are two maze navigation tasks originally proposed in D4RL (Fu et al. 2020). The domain consists of 3 mazes ranging from easy to hard ("umaze", "medium", "large"), and two morphologies: a 2D ball in Maze2D and the "Ant" robot of the Gym benchmark in AntMaze. For Maze2D, we provide less challenging reward computed base on distance to a fixed goal. For the AntMaze environment reward is given only upon reaching the fixed goal.
Maze2D and AntMaze datasets and policies. Datasets for both morphologies consists of undirect data navigating randomly different goal locations. The datasets for Maze2D collected by using a high-level planner to command waypoints to a low-level PID controller in order to reach randomly selected goals. The dataset in AntMaze is generated using the same high-level planner, but the low-

---

Published as a conference paper at ICLR 2021
level planner is replaced with coal-conditioned policy trained to reach arbitrary waypoints. Both of these datasets are generated from -arrkvvnin policies, as the high-level controller maintains history of waypoints reached in order to construct plan to the goal. We provide policies for all environments except "antmaze-large"" by taking training snapshots obtained while running the DAPG algorithm (Rajeswaran al. 2017). Because obtaining high-performing policies for "antmaze-large" was challenging, we instead used imitation learning on a large amount of expert data to generate evaluation policies. This expert data is obtained by collecting additional trajectories that reach the goal using a high-level waypoint planner in conjunction with low-level goal- cond-condt policy (this is the same method as was used generate the dataset, Sec. 5 (Fu et al., 2020)).
Adroit tasks. The Adroit domain is a realistic simulation based on the Shadow Hand robot, first proposed by Rajeswaran et al. (2017). There are 4 tasks in this domain: opening door ("door"), ) pen twirling ("pen"), moving a ball to a target location ("relocate"), and hitting nail with hammer ("hammer"). These tasks all contain sparse rewards and are difficult to learn without demonstrations.
Adroit datasets and policies. We include 3 datasets for each task. The "human" " dataset consists of a small amount of human demonstrations performing the task. The "expert" dataset consists of data collected from expert trained via DAPG (Rajeswaran et al. 2017). Finally, the "cloned" dataset contains a mixture of human demon- strations and data collected from imitation learning algorithm trained the demonstrations, For policies, we include 11 policies collected from snapshots while running the DAPG algorithm, which range from random performance expert performance.
5 BASELINES AND RESULTS
The goal of our evaluation is two-fold. First, we wish to measure the performance of a variety of existing algorithms to provide baselines and reference numbers for future research. Second, we wish to identify shortcomings in these approaches to reveal promising directions for future research.
5.1 BASELINES
We selected six methods evaluate, which cover variety of approaches that have been explored for the OPE problem.
Fitted Q-Evaluation (FQE) As in Le et al. 2019), train a neural network estimate the value of the evaluation policy I by bootstrapping from Q(s', (s')). We tried two different implementations, one from Kostrikov & Nachum 2000 and another from Paine al. (020)) labeled FQE-L2 and FQE-D respectively to reflect different choices in loss function and parameterization.
Model-Based (MB) Similar to Paduraru (207). J. we train dynamies and reward models on transitions from the offline dataset D. Our models are deep neural networks trained to maximize the log likelihood of the next state and reward given the current state and action, similar to models from successful model-based RL algorithms (Chua et al. 2018 Janner et al., 2019). We follow the setup detailed in Zhang et al. (20TT). We include both the feed-forward and auto-regressive models labeled MB-FF and MB-AR respectively. To evaluate policy, we compute the return using simulated trajectories generated by the policy under the learned dynamics model.
Importance Sampling (IS) We perform importance sampling with learned behavior policy. We use the implementation from Kostrikov & Nachum (2020)3 which uses nelf-normalized (also known as weighted) step-wise importance sampling (Precup) 2000). Since the behavior policy is not known explicitly, we learn an estimate of it via max-likelihooe objective over the dataset D. as advocated by Xie et al. (2018): ; Hanna et al. (2019). In order to be able compute fog-brobabsllites when the target policy is deterministic, we add artificial Gaussian noise with standard deviation 0.01 for all deterministic target policies.
Coode available at master/policy_eev

---

Published as a conference paper at ICLR 2021
19-4X F202
M3-I 100AZ
DIC
1042 F2B2
FSCO
WM
Brodies
Roodines
Figure 4: DOPE RL Unplugged Mean overall performance of baselines.
Koudines
Roodines
Figure 5: DOPE D4RI, Mean overall performance of baselines.
Doubly-Robust (DR) We perform weighted doubly-robust policy evaluation Thomas & Brunskill (2016) using the implementation of Kostrikov & Nachum (2020) Specifically, this method combines the IS technique above with value estimator for variance reduction. The value estimator is learned using deep FQE with an L2 loss function. More advanced approaches that trade variance for bias exist (e.g., MAGIC (Thomas & Brunskill 2016)), but we leave implementing them future work.
DICE This method uses saddle-point objective to estimate marginalized importance weights d (s, a)=d (s, a); these weights then used to compute weighted average of reward over the offline dataset, and this serves as an estimate of the policy's 's value in the MDP. We use the implementation from Yang et al. (020)) corresponding to the algorithm BesIDIEE
Variational Power Method (VPM) This method runs a variational power iteration algorithm to estimate the importance weights d' (s, = without the knowledge the behavior policy. then estimates the target policy value using weighted average of rewards similar to the DICE method. Our implementation based on the same network and hyperparameters for OPE setting as Wen et al. (2020). We further tune the hyper-parameters including the regularization parameter X, learning rates Orp and av,, and number iterations on the Cartpole swingup task using ground-truth policy value, and then fix them for all other tasks.
5.2 RESULTS
To facilitate aggregate metrics and comparisons between tasks and between DOPE RL Unplugged and DOPE D4RL, we normalize the returns and estimated returns to range between 0 and For each set of policies
compute the worst value Vworst min (V" V .... V } and best value V*8) and normalize the returns and estimated returns according to
Vbest = mar[V=), 7T22 x' =xx (x- Vworst)) (Vhest V.
We present results averaged across DOPE RL. Unplugged in Fig. and results for DOPE D4RL in Fig. [ Overall, no evaluated algorithm attains near-oracle performance under any metric (absolute error, regret, or rank correlation). Because the dataset is finite, we do not expect that achieving oracle performance is possible. Nevertheless, based on recent progress on this benchmark (e.g., Zhang et al. (202T)), we hypothesize that the benchmark has room for improvement, making it suitable for driving further improvements on OPE methods and facilitating the development of OPE algorithms that can provide reliable estimates on the types of high- siggh-iimensonall problems that we consider.
While all algorithms achieve sub-optimal performance, some perform better than others. We find that on the DOPE RL Unplugged tasks model based (MB-AR, MB-FF) and direct value based methods (FQE-D, FQE-L2) significantly outperform importance sampling methods (VPM, DICE, IS) across all metrics. This is somewhat surprising as DICE and VPM have shown promising results in other settings. We hypothesize that this is due to the relationship between the behavior data and evaluation policies, which is different from standard OPE settings. Recall that in DOPE RL Unplugged the behavior data is collected from an online RL algorithm and the evaluation policies are learned via offline RI. from the hehavior data. In experience all methods work better when the behavior policy a noisy/perturbed version of the evaluation policy. Moreover, MB and FQE-based methods may
*Code available httptttppp htap:p:://ghtttt wtt.:ccoo/ooooeeo research/di.ce_r

---

Published as a conference paper at ICLR 2021
H5-AR
TOE-0
FQ2-12
Rank Comn Mile
Figure 6: Rank correlation for each baseline algorithm for each RL Unplugged task considered.
finger turn hard
humanoid run
manipulator insert ball FQE-D
manipulator insert peg
FQE-D
MB-AR
FQE-D
MB-AR
MB-AR
FQE-D
MB-AR
Estimate
1.5
0.5
0
0.5 0.5 0.5
0.5 1552 155
Return (d-0.995)
Figure 7: Scatter plots of estimate ground truth return for MB-AR and FQB-D selected tasks.
implicitly benefit from the architectural and optimization advancements made in policy optimization settings. which focus similar environments and where these methods are more popular than importance sampling approaches. Note that within the MB and FQE methods, design details can create significant difference in performance. For example model architecture (MB-AR vs MB-FF) and implementation differences (FQE-D vs FQE-L-2) show differing performance on certain tasks.
On DOPE D4RL, direct value based methods still do well, with FQE-L2 performing best on the Absolute Error and Regret@) I metrics. However, there are cases where other methods outperform FQE. Notably, IS and DR outperform FQE-L2 under the rank correlation metric. As expected, there is a clear performance gap between DOPE RL Unplugged and DOPE D4RI.. While both domains have challenging tasks, algorithms perform better under the more ideal conditions of DOPE RL Unplugged than under the challenging conditions of DOPE D4RL (0.69 vs 0.25 rank correlation respectively).
In Fig. A.2 we show the rank correlation for each task in DOPE RL Unplugged. Most tasks follow the overall trends, but we will highlight a few exceptions. 1) Importance sampling is among the best methods for the humanoid run task, significantly outperforming direct value-based methods. 2) while MB-AR and FQE-D are similar overall, there are a few tasks where the difference is large, for example FQE-D outperfroms MB-AR finger lurn hard, and manipulator insert ball, where as MB-AR outperforms FQE-D on cartpole swingup, fish swim, humanoid run, and manipulator insert peg. We show the scatter plots for MB-AR and FQE-D on these tasks in Fig 7 which highlights different failure modes: when MB-AR performs worse, it assigns similar values for all policies; when FQE-D performs worse, severely over-estimates the values of poor policies.
We present more detailed results, separated by task, in Appendix A.2 Note in particular how in Table A.2.2, which shows the regret metric for different D4RL tasks, the particular choice of dataset for the Gym-MuloCo, Adroit, and AntMaze domains causes a significant difference in the performance of OPE methods. This indicates the importance of evaluating multiple distinct datasets, with different data distribution properties (c.g., datasets, such as random data), as no tested method is reliably robust to the effects dataset variation.
narrow datasets, such expert data, vs. broader
High- dinen-iimensional tasks requiring temporally extended control were also challenging, highlighted by the performance on the AutMaze domain. No algorithm was able to achieve a good absolute error value on such tasks, and importance sampling was the only method able to achieve correlation consistently above zero, suggesting that these more complex tasks are a particularly important area for future methods focus
8

---

Published as a conference paper at ICLR 2021
6 RELATED WORK
Off-policy evaluation (OPE)
has been studied extensively across a range of different domains, from
healthcare (Thapa et al. 2005
Raghu et al. 2018 Nie al. 2019). to recommender systems (Li Theocharous et al. 2015), and robotics (Kalashnikov ct al. 2018).
ct al., 2010, Dudik et al. 2014
While a full survey of OPE methods outside the scope of this article, broadly speaking we can
categories OPE methods into groups based the use of importance sampling (Precupl 2200), value functions (Sutton et al. 2009 Migliavacca et al.) 2010, Sutton et al. 2016; Yang et al. [2020, and learned transition models (Paduraru 2007). though a number of methods combine two or more of these components (Jiang & 2015 Thomas & Brunskill 2016, Munos et al. 2016). A significant body of work in OPE is also concerned with providing statistical guarantees (Thmas et al. 2015). Our focus instead is on empirical evaluation while thcoretical analysis is likely to be a critical part of future OPE research, combining such analysis with empirical demonstration on broadly accepted and standardized benchmarks is likely to facilitate progress toward practically useful algorithms.
Current evaluation of OPE methods is based around several metrics, including enror in predicting the true return of the evaluated policy (Voloshin et al., 2019), correlation between the evaluation output and actual returns (Irpan al. 2019), and ranking and model selection metrics (Doroudi et al. 2017). As there is no single accepted metric used by the entire community, we provide a set of candidate metrics along with our benchmark, with a detailed justification in Section [ Our work is closely
related to (Paine et al. (020) which studies OPE in a similar setting, however in our work we present a benchmark for the community and compare a range of OPE methods. Outside of OPE, standardized benchmark suites have led to considerable standardization and progress in RL (Stone & Sutton 2001 Dutech et al. 2005| Riedmiller et al. 2007). The Arcade Learning Environment (ALE) (Bellemare) et al.] 2013) and OpenAl Gym (Brockman et al. 2016) have been widely used to compare online RL algorihhms to good effect. More recently, Gulcchre et al. (2020) Fu et al. (2020) proposed benchmark tasks for offline RL. Our benchmark is based on the tasks and environments described in these two benchmarks, which we augment with a set of standardized policies for evaluation, results for a number of existing OPE methods, and standardized evaluation metries and protocols. VVoooshin et al. (2019)
have recently proposed benchmarking for OPE methods on a variety of tasks ranging from tabular problems to image-based tasks in Atari. Our work differs in several key aspects. Volossin al. (0019)
is composed entirely of discrete action tasks, whereas out benchmark focuses on continuous action tasks. Voloshin Voloshi et al. (2019) assumes full support for the evaluation policy under the behavior policy data, whereas we designed our datasets and policies to ensure that different cases of dataset and policy distributions could be studied. Finally, all evaluations Voloshin et al. (2019) are performed using the MSE metric, and they do not provide standardized datasets. In contrast, we provide variety of policies for each problem which enables one to evaluate metrics such as ranking for policy selection, and wide range of standardized datasets for reproducbility.
CONCLUSION
We have presented the Deep Off-Policy Evaluation (DOPE) benchmark, which aims to provide a platform for studying policy evaluation and selection across a wide range of challenging tasks and datasets. In contrast to prior benchmarks, DOPE provides multiple datasets and policies, allowing researchers to study how data distributions affect performance and to evaluate wide variety of metrics, including those that are relevant for offline policy selection. In comparing existing OPE methods, we find that no existing algorithms consistently perform well across all of the tasks, which further reinforces the importance of standardized and challenging OPE benchmarks. Moreover, algorithms that perform poorly under one metric, such as absolute error, may perform better on other metrics, such correlation, which provides insight into what algorithms to use depending on the use case (e.g., policy evaluation vs. policy selection).
We believe that OPE is exciting area for future research, as it allows RL agents to learn from large and abundant datasets in domains where online RL methods are otherwise infeasible. We hope that our benchmark will enable further progress in this field, though important evaluation challenges remain. As the key benefit of OPE is the ability to utilize real-world datasets, promising direction for future evaluation efforts is to devise effective ways to use such data, where key challenge is to develop evaluation protocols that are both reproducible and accessible. This could help pave the way towards developing intelligent decision making agents that can leverage vast banks of logged information to solve important real-world problems.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Charu C Aggarwal et al. Recommender systems, volume 1. Springer, 2016.
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference on Learning Representations, 2018.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Léon Bottou, Jonas Peters, Joaquin Mauñoonero-Candlla Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. The Journal of Machine Learning Research, (4(1):3277-3260 2013.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 375444765 2018.
Shayan Doroudi, Philip S Thomas, and Emma Brunskill. Importance sampling for fair policy selection. Grantee Submission, 2017.
Miroslav Dudik, Dumitru Erhan, John Langford, Lihong Li, et al. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-511, 2014.
Alain Dutech, Timothy Edmunds, Jelle Kok, Michail Lagoudakis, Michael Littman, Martin Ried- miller, Bryan Russell, Bruno Scherrer, Richard Sutton, Stephan Timmer, et al. Reinforcement learning benchmarks and bake-offs ii. Advances in Neural Information Processing Systems (NIPS), 17:6, 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596, 2018.
Alexandre Gilotte. Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline a/b testing for recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 198-206, 2018
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off ool-cy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 338993396.. IEEE, 2017.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006.
2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv:1801.00 012900 2018.
Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with estimated behavior policy. In International Conference on Machine Learning, pp. 2605-2613, PMLR, 2019.
Milos Hauskrecht and Hamish Fraser. Planning treatment ischemic heart disease with partially observable markov decision processes. Artificial Intelligence in Medicine, 18(3):221-244, 2000.
10

---

Published as a conference paper at ICLR 2021
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: A research framework for distributed reinforcement learning, arXiv preprint S0t1::00060009777 2020.
Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off- aff-po classification. In Advances in Neural Information Processing Systems, pp. 5437-5448, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519-12530, 2019.
Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. arXiv preprint arXiv: 1.13722 2015.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint 1806::8800.11999 2018.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning drive in day. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8248 8254. IEEE, 2019.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11): 1274, 2013.
Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation, 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. arXiv preprint arXiv:1903.08733 2019.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: SEKG::::509009771 2015.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, pp. 1077- 1084, 2014.
Martino Migliavacca, Alessio Pecorino, Matteo Pirotta, Marcello Restelli, and Andrea Bonarini. Fitted policy search: Direct policy search using a batch reinforcement learning approach. In 3rd International Workshop Evolutionary and Reinforcement Learning for Autonomous Robot Systems (ERLARS 2010). pp. 35. Citeseer, 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop. 2013.
Rémi Munos, Tom Stepleton, Anna Harutyunyan. and Mare G. Bellemare. Safe and efficient off-policy reinforcement learning. arXiv preprint 1606vv/606.0266 2016.
Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to teen-tt-treat policies. arXiv preprint 195791190 2019.
Cosmin Paduraru. Planning with approximate and learned models markov decision processes. 2007.
11

---

Published as a conference paper at ICLR 2021
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXhv:2007.0005. 2020.
Doina Precup. Eligibility traces for off- policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Aniruddh Raghu, Omer Gottesman, Yao Liu, Matthieu Komorowski, Aldo Faisal, Finale Doshi- Velez, and Emma Brunskill. Behaviour policy estimation in off-policy policy evaluation: Calibration matters. arXiv preprint arXiv: 1807.01066, 2018.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
Martin Riedmiller, Jan Peters, and Stefan Schaal. Evaluation of policy gradient methods and variants the cart-pole benchmark. In 2007 IEEE International Symposium Approximate Dynamic Programming and Reinforcement Learning, pp. 254-261. IEEE, 2007.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
Peter Stone and Richard S Sutton. Scaling reinforcement learning toward robocup soccer. In Ieml, volume 1, pp. 537-544. Citeseer, 2001.
Richard S Sutton, Hamid Reza Maei, Doina Precup. Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast dradeent methods for temporal-diiffernce learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 993-1000, 2009.
Richard Sutton, Rupam Mahmood, and Martha White. An emphatic approach the problem of off-policy -ifferel-difference learning. The Journal of Machine Learning Research, 17(1): 2603-2631, 2016.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. International Conference on Machine Learning, pp. 814-823, 2015.
Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. Advances in Neural Information Processing Systems, pp.
2017.
Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3): 58 58-68,1 1995.
Devinder Thapa, In-Sung Jung, and Gi-Nam Wang. Agent based decision support system using reinforcement learning under emergency cireumstances. In International Conference on Natural Computation, pp. 388-892. Springer, 2005.
Georgios Theocharous, Philip S Thomas, and Mohammad Ghavamzadeh. Personalized ad recommen- dation systems for life-time value optimization with guarantees. In Twenty-Four Fourth International Joint Conference on Artificial Intelligence, 2015.
Philip Thomas and Emma Brunskill. Data-cfficiemt off- polico policy evaluation for reinforcement learning. In International Conference Machine Learning, pp. 2139-2148, 2016.
12

---

Published as a conference paper at ICLR 2021
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- off-policy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033,
IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(77822:350-334 2019.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off- off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.066 2019.
Yu-Xia Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and adaptive off offl-c evaluation in contextual bandits. In International Conference on Machine Learning, pp. 3589-3597, PMLR, 2017.
Ziyu Wang, Alexander Novikov, Konrad Zoina, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. arXiv preprint arXiv:2006.151333 15134, 2020.
Junfeng Wen, Bo Dai, Lihong Li, and Dale Schuurmans. Batch stationary distribution estimation. arXiv preprint M:Xii:2003000722 2020.
Yuan Xie, Boyi Liu, Qiang Liu, Zhaoran Wang, Yuan Zhou, and Jian Peng. Off-policy evaluation and learning from logged bandit feedback: Error reduction via surrogate policy. arXiv preprint arXiv:1808.0023. 2018.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. arXiv preprint arXiv:2007.00 03438, 2020.
Michael Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2021. URL https
//openreview. forss/foum?iii (tee//oorrmmii--kggggss55yy
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 METRICS
The metrics we use in our paper are defined as follows:
Absolute Error We evaluate policies using absolute in order be robust outliers. The absolute error is defined as the difference between the value and estimated value of a policy:
AbsErr = IV V
Where V is the true value the policy, and Va is the estimated value of the policy.
Regret@k @k @egret@k is the difference between the value of the best policy in the entire set, and the value of the best policy in the top-k set (where the top-k is chosen by estimated values). can be defined as:
Regret @ k
V
V
(3)
(Elp))(I:N)
Where topk(1 N) denotes the indices of the top K policies measured by estimated values 1.
Rank correlation Rank correlation (also Spearman's 's p) measures the correlation between the ordinal rankings of the value estimates and the true values. It can be written
RankCorr Cov(V ViT)
Jo
A.2 DETAILED RESULTS
Detailed results figures and tables are presented here. We show results by task in both tabular and chart form, well as scatter plots which compare the estimated returns against the ground truth returns for every policy.
A.2.1 CHART RESULTS
First we show the normalized results for each algorithm and task.
cheett.co
< Ever
Figure A.I: Absolute enror for each baseline algorithm for each RL Unplugged task considered.
14

---

Published as a conference paper at ICLR 2021
H5-AR
TOC-0
FO2-22
Rank Came Min
Figure A.2: Rank correlation for each baseline algorithm for each RL Unplugged task considered.
M5-4F
X5-FT
PO2-22
Erand pure
Regsss3
Figure A.3: Regret@1 for each baseline algorithm for each RI. Unplugged task considered.
FQE-L2
antmaao
door
Domains
hammer
hupper
meredd
pen
nelocate
welcee 2r
0.5
1.5
0.5
1.5
0.5
1.5
9.5
1.5
0.5
Absolute Error
Figure A.4: Absolute error for each baseline algorithm for each D4RL task domain considered.
animage
door
halfchoorsh
Domains
hopper
mazežd
pon
relecate
walkordd
-0.5
0.5
-0.5
0.5
-0.5
-2.
0.5
-0.5
2.5
Rank Correlation
Figure A.5: Rank correlation for each baseline algorithm for each D4RL task domain considered.
15

---

Published as a conference paper at ICLR 2021
antmaze
door
Domains
hooppr
mazete
pen
rdocate
malcar
0.25
0.50
0.75
0.25
0.50
0.75
0:25
0.50
0.75
0.25
0.50
0.75
1
0
0.25
0.50
0.75
Regretga1
Figure A.6: Regret@) I for each baseline algorithm for each D4RL task domain considered.
cheetah run
finger hard
humanoid run
ma mianuuutaoor Insert peg
1o00 COoh Lap
Burg uoee So31
CAR
AMM
AMM
Checkzore
Checksone
Crecksone
Cherepoetid
Chiddos
Figure A.7: Online evaluation policy checkpoints for Offline algorithms with random seeds. observe a large degree of variability between the behavior algorithms on different tasks.
A.2.2
TABULAR RESULTS
Next, we present the results for each task and algorithm in tabular form, with means and standard deviations reported across seeds.
Cartpole swingup
Cheetah run
Finger turn hard
Fish swim
Humanoid run
Absolute Error hiw.
OPE and ground truth
Variational power method
37.53 =3.50
61.89
24.25
46.22 23.93
31.27 20.09
35.29
1:3.03
Importance Sampling
68.75 +2.39
44.29
11.91
90.10 24.68
34.82
#1.93
27.89 #1.98
Best DICE
22.73 #1.65
23.35
11.32
33.52 23.48
59.48 =2.47
31.42 #2.04
Model based FF
6.80 20.85
13.61
20.59
35.99
=3.00
4.75 =0.28
30.12 #2.40
FQE (L2)
19.02 11.34
48.26
11.8
27.91 #1.18
19.82
56.28 4:3.2
Doubly Robust (IS, FQE)
24.38 #2.51
40.27
22.0
25.26
22.48
20.28 1.90
53.64 1:3.68
FQE (distributional)
12.63 #1.21
36.50
a 1.62
10.23
#0.93
7.76 0..95
32.36
4:2.27
Model based AR
5.32
20.54
4.64
20.48
22.93
1.77
4.31 20.22
20.95 # 1..11
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median
Absolute Error btw.
OPE and ground truth
Variational power method
96.76 23.50
87.24
24.25
79.25 20.19
21.95 =1.17
16.22
Importance Sampling
66.50 11.90
67.2
22.70
29.93 11.10
12.78 20.66
44.29
Best DICE
27.58 #3.01
47.28
23.13
103.45
22.75 23.00
31.42
Model based FF
23.34 +2.41
52.23
34.30 #2.55
121.12
+1.58
30.12
FQE (L2)
6.51 40.71
18.34
20.95
36.32 #1.0
31.12 #2.37
27.91
Doubly Robust (IS, FQE)
26.82 #2.00
24.63
1.00
13.33 +1.16
22.28 +2.34
24.63
FQB (distributional)
21.49 11.41
27.57
1154
9.75 11.1
12.66 11.39
12.66
Model based AR
19.12
11.23
5.14
20.49
17.13
1133
9.71
9.71
Table A.1: Average absolute error between OPE metrics and ground truth values at discount factor of 0.995
In each column, absolute error values that are not significantly different from the best (p >> 0.05) are bold faced. Methods are ordered by median.
16

---

Published as a conference paper at ICLR 2021
Cartpole swingup
Cheetah run
Finger
Fish
Humanoid
turn hard
swim
run
Rank Correlation btw.
OPE and ground truth
Importance Sampling
0.23 #0.11
0.01 #0.12
0.45 +0.08
0.17 #0.11
0.91 +0.02
Best DICE
0.16 #0.11
0.07
0.22 s0.11
0.44 60.09
-0.10
Variational power method
0.01 #0.11
0.01 40.12
0.25 +0.11
0.56 +0.08
0.36 +0.09
Doubly Robust (IS, FQB)
0.55 = 09
0.56:10.08
0.67 20.05
0.11 10.12
-0.03 10.12
Model based FF
0.83 20.05
0.64 1:0000
0.08 20.11
0.95 ::0 02
0.35 10010
FQE (distributional)
0.69 =0.07
0.67 -:0.00
0.94 10.01
0.59 10.10
0.74
FQE (L2)
0.70 =0.07
0.56 20.08
0.83 #0.04
0.10 10.12
0.00 1:012
Model based AR
0.91
0.74 00.07
0.57 #0.09
0.96 :0.01
0.90 +0.02
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median t
Rank Correlation buw
OPE and ground truth
Importance Sampling
0.59 20.08
0.38 10.10
0.72 #0.05
0.25 #0.08
0.17
Best DICE
00.11 10.12
0.5810..0
0.19 10.11
0.35 10.10
-0.11
Variational power method
00.35 10.10
10.10
0.61 10.08
0.41 10.09
0.01
Doubly Robust (IS, FQE)
0.88 20.03
0,85 1:0.04
0.42 10.10
00.47 10.09
0.55
Model based FF
0.82 20.04
#0.05
0.06 10
0.56 20.08
0.64
FQE (distributional)
0.87 =0.02
10.00
0.63 20.08
-0.23 20.10
0.69
FQE (I.2)
0.96
0.94 10.02
0.70 =0.07
-0.48 20.08
0.70
Model Based AR
0.96 0.001
0.98 40000
0.33 #0.09
0.47 +0.09
0.90
Table A.2: Spearman's 's rank correlation (p) coefficient (bootstrap mean + standard deviation) between different OPE metrics and ground truth values discount factor of 0.995. each column, rank correlation coefficients that are not significantly different from best (p > 0.05) are bold faced. Methods are ordered by median. Also see Table A.3 and A.I for Normalized Regret@5 and Average Absolute Error results.
Cartpole swingup
Cheetah run
Finger
Fish
Humanoid run
turn hard
swim
OPE vV. ground truth
Importance Sampling
0.73 =0.
0.40 20.21
0.64 10.05
0.12::0.05
0.31 10.09
Regret for
Best DICE
0.68 40.41
0.27 20.05
0.44 20.04
0.35 to 24
0.84 #0.22
Variational power method
0.50 10.13
0.37 20.04
0.45 20.13
0.02 +0.02
0.56 +0.08
Doubly Robust (IS, FQE)
0.28 20.05
0.09 =0.00
0.56 20.12
0.61 10.12
0.99 #0.00
FQE (L2)
0.06 00.04
0.17
0.30
0,50 40.03
0.99
Model based-
0.02 0.00
0.24 1.122
0.43 00.04
0.00 10000
0.44 4:0.02
FQE (distributional)
0.03 00.09
0.11 0.009
0.10 10.12
0.49 4:0.06
0.24 0.155
Model based AR
0.00 20.02
0.01 #0.02
0.63 20.11
0.03 2.002
0.32 100068
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median |
OPE 18. ground truth
Importance Sampling
0.51 10.11
0.54 20.23
0.83 20.05
0.22 40.0s
0.54
R5 for
Best DICE
0.24 20.07
0.55 #0.06
0.44 20.07
0.75440.0
0.44
Variational power method Doubly Robust (IS, FQE)
0.41 20.02
0.39 20.02
0.52 =0. 20
0.41
0.02 #0.01
0.05
0.30
0.73
0.30
FQE (L2)
0.04 +0.02
0.00 00.02
0.37 #0.07
0.74 40.01
0.30
Model based FF
0.18 10.10
0.03 00.05
0.83 60.00
0.74 40.00
0.24
FQE (distributional)
0.03 20.08
0.01 20.02
0.50 20.30
0.73
0.11
Model based AR
0.04 10.02
0.04 10.02
0.85 20.02
0.30 10.04
0.04
Table A.3: Normalized R (bootstrap mean + standard deviation) for OPE methods vs. ground truth values at discount factor of 0.995. In each column, normalized regret values that are not significantly different from the best (p > 0.05) are bold faced, Methods ordered by median.
17

---

Published as a conference paper at ICLR 2021
Halfcheetah
Halfcheetah
Halfcheetah medium-
Halfcheetah
Halfcheetah
expert
medium
medium-rephee
random
1404 1:22
1217 122
1400 +:46
1409 +154
1405 +100
Abs. Brror
VPM
945 +104
1374
+158
1427
1384 4:48
1411 +164
Best DICE
944
1382
+180
1078
1440
1446
Doubly Robust 1025
1222 134
1015 103
1001 1:12
949 1128
FQE (L2)
1031 1
1211 1:0
1014 101
1003 1:32
938 1:25
Antmaze large-diverss
Antmaze large-play
Antmaze medium-divee
Antmaze medium-plas
Antmaze umaze
Abs. Bror
0.62 20.01
0.85 =0.00
20.0
0.81 #0.00
0.62 20.04
VPM
0.02 +0.02
0.26 00.24
0.07 0.005
0.11 40.06
0.12 00003
Best DICE
5.55 1.036
19.62 1:128
2.42 4 56
19.47 42.15
14.97 4:1.93
Doubly Robust
0.99 40.01
1.59 +0.01
0.61 40
1.47 40.01
0.87 4:004
FQE (1.2)
0.53 10.01
0.78 =0.00
0.29 20.01
0.71 40.01
0.39 #0.03
Antmaze umaze-diverse
Door cloned
Door expert
Door
Hammer
human
cloned
Abs. Error
IS
0.14
891 1188
648
870 4178
7403
VPM
0.12 +0.08
1040 1188
879 +182
862 #163
7459 4:114
Best DICE
0.17 #0.04
697 #79
856 +184
1108 +199
4169 #839
Doubly Robust
0.11 1:002
424 173
1353 1218
379 1665
6101 1:679
FQE (L2)
0.11 10.03
438
1343
389 1600
5415
Hammer expert
Hammer human
Maze2d large
Maze2d medium
Maze2d
umaze
Abs. Enror
IS
3052
7352 =1118
45.61
100.43
61.29 17.78
50.20 19.16
VPM
7312 1117
7105 1100
44.10
110.99
60.30 48.37
62.81 +8.40
Best DICE
3963 776
5677 #936
42.46
99.66
58.97 49.37
21.95 +4.69
Doubly Robust 3485 +590
5768 7701
22.94
+6.82
23.64
4-4.96
76.93 4-4.42
FQE (L2)
2950 1728
6000
2612
24.31
16.56
35.11
16.33
79.67 14.93
Pen cloned
Pen
Pen
Relocate cloned
Relocate expert
expert
human
IS
1707 128
4547 =222
3926 112
632 4:15
2731 +147
Abs. Error
VPM
2324 129
2325 1136
1569 2216
586 1185
::14 620
Best DICE
1454
2963 =279
424
13/17 4:485
1095
Doubly Robust 1323 198
2013 1564
2846 1:200
412 1124
1193 1330
FQE (L2)
1232 100
1057 2881
2872 4:70
439 1:125
1351 1:993
Relocate human
Ant medium
Ant medium-exper
Ant medium-replay
expert
Abs. Error
638 -217
605=2 1004
594 104
604 1102
603 101
VPM
806 100
607 #108
570 +100
604 +100
612 +105
Best DICE
4526 4-474
558 =108
495 +90
471 +100
583 +110
Doubly Robust
6114
584 1114
345
326
421 +72
FQE (L2)
593 1113
583 122
345
319 167
410 179
Ant random
Hopper expert
Hopper medium
Hopper random
Walker2d
expert
Abs. Error
IS
606
106
405
412
405
VPM
570
442 :43
433
438 4:44
367 is8
Best DICE
530
259 =64
241
122 116
437 :00
Doubly Robust
106
426 #99
307 4
289 4:50
519 1179
FQE (L2)
398
282 476
283
261 4-12
453
Walker2d medium
Walker2d medium-expert
Walker2d medium-replay
Walker2d random
Median
Abs. Entor
IS
428 100
436
427 160
430 161
603.82
VPM
460
425 +G1
424
440
585.53
Best DICE
273 +31
322 :60
374 is2
419 +67
530.43
Doubly Robust
368 +74
217 #46
296 +54
347 474
411.99
FQE (L2)
350
233 442
313 173
354 1.73
398.37
18

---

Published as a conference paper at ICLR 2021
Halfcheetah expert
Halfcheetah medium-expert
Halfcheetah medium-replay
Halfcheetah random
Door
cloned
Rank Corr.
Best DICE
0.44 #0.30
0.08 +0.30
0.15 +0.41
-0.70 +0.22
0.18 #0.31
VPM
0.18 #0.36
0,47 40
0.07 +0.36
0.27 26.36
0.29 +0.36
FQE (L2)
0.78 #0.15
0,62
0.26
-0.11 40.41
0.55
0.01 20.35
0..06
0.59 10
-0.24
0.66 10.22
Doubly Robust
0.77 10.17
0.62 20 27
0.32 10.37
-0.02 1:033
0.60 10.28
Door expert
Hammer cloned
Hammer expert
Maze2d large
Maze2d
medium
Rank Con:.
Best DICE
0.06 =0.32
0.35 20.88
0.42 20.31
0.56 20.21
0.64 20.23
VPM
0.65 0.23
-0.77 40.22
0.39 10.31
0.26 0.33
0.05 00.39
FQE (L2)
0.89 0.00
-0,15 0.333
0.29 10.34
0.30 06
0.16 #0.38
IS
0.76 00.17
0.58
0.64 +0.24
0.63
0.44 +0.25
Doubly Robust
0.76
13
0.770
0.49 10.31
0.31
20.86
0.41 20.35
Pen
Relocate expert
Ant
Ant medium
Ant medium-experti
expert
expert
Rank Corr,
Best DICE
0.53
0.27 20.34
0.13 +0.37
0.36 40.28
0.33
20.33
0.39 40.31
0.442 +0.38
40.31
0..8 =0.28
FQE (L2)
0.01 =0.33
0.57 #0.28
-0.13 +0.32
0.65 +0.26
0.37 #0.36
IS
-0.45 20.31
0.52 20.23
0.14 10.41
-0.17 1:.32
-0.21 10.35
Doubly Robust
0.52
-0.40 1:0.0
10.32 -0.28
0.66
0.35 10.35
Ant medium-repla
Ant random
Hopper expert
Hopper medium
Hopper random
Rank Corr.
Best DICE
0.24 20.39
0.21 20.35
00.08 20.82
0.19 i0.as
-0.13 20.39
VPM
0.26 +0.29
0.24 +0.31
0.21 +0.32
0.13
-0.46
FQE (L2)
0.57 00.28
0.04 4..33
-0.33 +0.30
-0.29 40.33
-0.11 #0.36
0.07 00.39
0.26 +..34
0.37 40.27
-0.55 40.26
0.23 #0.34
Doubly Robust
0.45 20.32
0.01 10.33
-0.41 20.27
0.31 20.34
-0.19 20.38
Walker2d expert
Walker2d medium
Walker2d medium-expert
Walker2d
Walker2d
medium-replay
random
Rank Corr.
Best DICE
0.37 20.27
0.12 20.38
0.34 20.34
0.55 20.28
0.19 20.36
VPM
0.17 20.32
0.44 20.21
0.49 20.37
0.52 20.25
0.42 20.34
FQE (L.2)
0.35
0.09
0.25 +0.82
0.19 40.36
0.21
00.37
-0.25 10.35
0.24
0.65 0..24
-0.05 10.38
Doubly Robust
0.26 0.34
0.02 10.37
0.19 10.33
0..37 0.339
0.16 20.29
Median
Rank Corr.
Best DICE
-0.19
VPM
-0.55
FQE (L2)
0.21
IS
0.23
Doubly Robust
0.26
19

---

Published as a conference paper at ICLR 2021
Halfcheetah expert
Halfcheetah
Halfcheetah medium-expert
Halfcheetah medium-replay
Halfcheetah
medium
random
Best DICE
0.32 #0.40
0.82 40.29
0.38 +0.37
0.30 =0.07
0.81 +0.30
Regret
VPM
0.14 20.09
0.33 +0.19
0.80 +0.34
0.25 =0.09
0,12 +0007
Doubly Robust
0.11 00.08
0.37 40.15
0.14 00.07
0.33
0.31 10.10
FQB (L2)
0.12 00.07
0.38 1:
0.14 10.07
0.36 10.16
0.37 4:0
IS
0.15 00.08
0. 0.05 10.05
0.73 10.42
0.13 10.10
0.31 10.11
Antmaze large-diverse
Antmaze aage-play
Antmaze medium diverse
Antmaze medium- play
Antmaze
umaze
Best DICE
0.54 20.34
0.96 10.13
0.04 10.11
0.09 10.10
0.69 20.39
Regret
VPM
0.88 00.27
0.45 10.30
0.14 10.10
0.03 00.08
0.62 0.0.3
Doubly Robust
0.83 00.30
0.93 00.21
0.05 00.07
0.17 #0.31
0,42 0:0.3
FQE (L2)
0.93
1.00 +0.03
0.16 +0.10
0.05 +0.19
0,41 4.0.5
IS
0.39 20.20
0.71 20.20
0.14
0.18 =0.06
0.86
Antmaze umaze-diverse
Door
Door
Door
Hammer
cloned
expert
human
cloned
Best DICE
0.42
0.65
0.37 #0.27
0.10
0.67
Regret
VPM
0.63 =0.32
0.81 +0.33
0.03 +0.08
0.69 20.24
0.72 40.39
Doubly Robust
0.79 10.14
0.11 +0.08
0.05 00.07
0.05 20.09
0.78 +:038
FQE (L2)
0.64 10.37
0.11 100.66
0.03 1:0 03
0.05 10.08
0.36 1:0.39
IS
0.22
0.02 10007
0.01 1:
0.45 10.40
0.03 1:.. 15
Hammer expert
Hammer
Maze2d large
Maze2d medium
Maze2d
human
umaze
Best DICE
0.24 20.34
0.04 20.08
0.15 10.08
0.44 20.05
0.03 10.07
Regret
VPM
0.04 00.07
0.18 +0.29
0,66
0.24
0,06 40.12
Doubly Robust
0.09 #0.09
0.46 40.23
0.21 40.16
0.27 #0.14
0.03 +0007
FQE (L2)
0.05 #0.04
0.46 40.28
0.20 +0.14
0.31 #0.14
0.03 +
IS
0.01 20.04
0.19 00.30
0.16 10 23
0.15 20.15
0.02 1:0.12
Pen cloned
Pen
Pen human
Relocate cloned
Relocate
expert
expert
Best DICE
0.12 20.08
0.33 20.20
0.04 #0.09
0.96 10.18
0.97 +0.07
Regret@
VPM
0.36
0.25 10.18
0.28 10.12
0.11 20.29
0.76 20.28
Doubly Robust
0.13 00.00
0.05 00.07
0.09 20.08
0.18 20.27
0.98 40.08
FQE (L2)
0.12 0.00
10.44
0.07 0.005
0.29 10.42
1,00 1:0.06
IS
0.14 0.09
0.31
10.10
0.17 10.15
0.63 20.41
0,18 1...4
Relocate human
Ant
Ant medium
Ant medium-expert
Ant medium-replay
expert
Best DICE
0.97 20.11
0.62 10.15
0.43 10.10
0.60 10.16
0.64 10.13
Regret @
VPM
0.77 #0.18
0.88 +0.22
0.40 #0.21
0.32 00.24
0.72 +0.48
Doubly Robust
0.17 10.15
0.43 +0.22
0.12 10.18
0.37 10.13
0.05 +:0.99
FQE (L2)
0.17 10.14
0.43
+0.22
0.12 10.18
0.36
0.05 1.05+00.00
IS
0.63 00.41
0.47
10.32
0.61 10.18
0.46
10.18
0.16
Ant random
Hopper expert
Hopper medium
Hopper
Walker2d
random
expert
Best DICE
0.50
0.20
0.18 10.19
0.30
0.35
Regret@v
VPM
0.15 20.2
10.10
0.10 10.14
0.26 10.10
0.09 10.19
Doubly Robust
0.28 10.15
0.34 +0.35
0.32 20.32
0.41 10.17
0.06 20.07
FQE (L2)
0.28 00.15
0.41 4:0.20
0.32 00.32
0.36 0..22
0,06 0.0.7
IS
0.56
0.06 +0.03
0.38
0.05 +0.05
0.43
Walker2d medium
Walker2d medium-expert
Walker2d medium-replay
Walker2d random
Median
Best DICE
0.27 20.43
0.78 10.27
0.18 10.12
0.39 20.33
0.38
Regret
VPM
0.08
0.24 +0.42
0.46 40.31
0.88 +0.20
0.28
Doubly Robust
0.25 20.09
0.30 10.12
0.68 +0.23
0.15 20.20
0.25
FQE (L2)
0.31 20.10
0.22 40.14
0.24 +0.20
0.15 #0.21
0.21
IS
0.70 00.39
0.13 1.0.07
0.02 00.05
0.74 10.33
0.18
20

---

Published as a conference paper at ICLR 2021
A.2.3 SCATTER PLOTS
Finally, we present scatter plots plotting the returns of each policy against the estimated returns.
Each point on the plot represents one evaluated policy.
MB-AR
FQE-D
FQE-L2
MB-FF
DR
VPM
15
DICE
cpole swingup
1.5
cheetah run
44
hard
finger turn
0.5
12 M
fish Swim
Estimate
1.5
humanoid run
manipulator Insert
ball
0.5
manipulator insert
peg
15
walker Cann
0.5
0.5 1 152
0051
152
00.
15
0.. 1 1.5 0.5 1.5
0.5 152 0 152 0.5 1.5.
Retum (d=0.995)
Figure A.8: Scatter plots estimate vs ground truth return for each bascline on each task in DOPE RL Unplugged.
21

---

conference paper at ICLR 2021
Published as a conference
DR
FQE-L
VPM
DICE
ant expert
0.5
ant medium
ant medium expert
EStimate
1.5
Aadd
1.5
antmaze large diverse
Keto
0.5
15
2
0
15
2
0.5
1.
0.5
1
15
0.5
15
Return (d=0.995)
Figure A.9: Scatter plots of estimate ground truth return for each baseline each task in DOPE D4RL (part l).
22

---

Published as a conference paper at ICLR 2021
DR
FQE-L2
VPM
DICE
diverse
antmaze medium
0.5
antmaze medium play
antmmaze umaza
EStimate
1.5
antmaze umazee diverse
1.5
door cloned
door expert
soop
0.5
1.5
2
1.
2
1.5
2
0.5
15
0.5
15
Return (d=0.995)
Figure A.10: Scatter plots of estimate ground truth return for each baseline each task in DOPE D4RL (part 2).
23

---

Published as a conference paper at ICLR 2021
DR
FQE-L2
VPM
DICE
halfcheetah expert
0.5
halfcheetah medium
halfcheetah medium expert
EStimate
1.5
halfcheetah medium replay
1.5
halfcheetah random
hammer cloned
hammer expert
0.5
15
2
1.
2
1.5
0.5
15
0.5
15
Return (d=0.995)
Figure A.11: Scatter plots of estimate ground truth return for each baseline each task in DOPE D4RL (part 3).
24

---

Published as a conference paper at ICLR 2021
DR
FQE-L2
VPM
DICE
human
hammer
0.5
hopper expert
0.5
hopper medium
Estimate
hopper randomm
large
1.5
PaBae 22 med Reedum
1.5
azeum
Prazew
0
0.5
1.5
0
0.5
1.5
0.5
1.5
0.5
1.5
0.5
1
1.5
Retum (d=0.995)
Figure A.12: Scatter plots of estimate vs ground truth return for each baseline on each task in DOPE
D4RL (part 4).
25

---

Published as a conference paper at ICLR 2021
DR
FQE-L2
VPM
DICE
Coned
pen
0.5
pen expert
pen húman
Estimate
relbcate cloned
relocate expert
1.5
relocate human
1.5
expert
0
0.5
1.5
0
0.5
1.5
0.5
0.5
0.5
1
1.5
Retum (d=0.995)
Figure A.13: Scatter plots of estimate vs ground truth return for each baseline on each task in DOPE
D4RL (part 5).
26

---

Published as a conference paper at ICLR 2021
DR
FQE-L2
VPM
DICE
medlum
walker2d
0.5
medium expert
walker2d
istimate
Redium d00e
makkrrd
1.5
LLODUN PZ
0.5
15
2
0.5
1.5
2
0.5
1.5
0.5
1.5
0.5
1
1.5
Return (d-0.995)
Figure A.14: Scatter plots of estimate vs ground truth return for each baseline on each task in DOPE D4RL (part 6).
27


Reviews:
[{"review_id": "B_sD36S2WyC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes an off-policy evaluation dataset (DOPE) for various control tasks.\nThe authors include various baselines for off-policy evaluation and evaluate them with additional metrics to MSE, namely regret@k and rank correlation.\n\n## Writeup\n\nThe writeup is good. It is clear what the \"features\" of the proposed benchmark are, and what problems the benchmark addresses that the other benchmarks didn't address before. I don't really like that in different images form  dm-control and gym environments pop up in the paper. Since these environments as self-standing were not the contribution of the paper I would remove these images, unless they are used to show a specific point. The space should be used rather to illustrate the point of the paper, that additional metrics are needed for evaluating algorithms / better benchmark is needed.\n\n## Pros\n\nI think that using additional evaluation metrics for off-policy are indeed necessary. The authors show that not one single metric summarizes performance over all of the metrics. This is going to allow researchers to do a more thorough analysis of offline reinforcement learning algorithms.\n\n## Cons\n\nInstead of using the space for \"nice pictures\", use it to actually support the claims of the paper, maybe shifting some figures from the appendix to the main text would be more informative. \nThe paper is not making any improvement over the methods or suggesting any reason why there is such a disparity in performance between different metrics, as such, there are no deeper insights.\nA large chunk of the paper is previous work.\n\n## Comments\n\nIn general, the experimental evaluation is thorough (there are a lot of experiments in the appending with a lot of tables and figures). \n\nTo me it is a bit questionable to which extent this data might count as high-dimensional and complicated, in comparison to real-world of robotics, this data is still quite low dimensional. It would be nice to see this dataset contain image data for learning from image input.\n\nI am highly suspicious about the way the data is generated for the ant-maze environment. Specifically, you use expert data for ant-maze-large because DAPG didn't work. Please specify where does the expert data come from.\n\nI didn't try to run the code, but I looked at it. I didn't find the code for the algorithms themselves. I would expect to have the training code to reproduce the experiments. With the current code I can only evaluate saved policies?\n\nCan you comment on what kind of normalization was done on the reward function? Ideally it should be normalized across all environments in the benchmark, such that performance is comparable. \n\nMaybe also writing down the equations for regret@k and rank correlation would be helpful. It is sometimes simpler to understand the metric once one sees the actual equation. \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9EPi9APAoaY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article proposes a benchmark of off-policy evaluation, which provides different metrics for policy ranking, evaluation and selection. Offline metrics are provided by evaluating the value function of logged data, and then evaluating absolute error, rank correlation and regret. Verify the effectiveness of different offline evaluation methods. This article provides two evaluation scenarios, one is DOPE RL unplugged, and the other is D4RL. In the experiment, the author verified the benchmark proposed in this article in the MuJoCo environment to evaluate the effectiveness of different offline evaluation methods.\n\nThe paper’s key strengths:\n\n1.\tI think the research direction of this article is very meaningful, because many scenarios in the real world only have offline data, which requires a simulated environment or other offline evaluation methods for strategy evaluation and model selection. This is very crucial for model improvement and evaluation. And the author proposed absolute error and rank correlation in the benchmark to evaluate the strategy. I think this is very innovative.\n2.\tThe author has applied a large number of offline evaluation methods in the experiment to verify in different metrics, and in the appendix has conducted experiments and verifications for many MuJoCo scenarios. I think the experimental results are very rich and very confirmatory.\n\nThe paper’s key weaknesses:\n1.\tThe biggest problem with this article is that the contribution of the article is insufficient and lacks originality. This article proposes a benchmark for off-policy evaluation and verifies different OPE methods, but this article does not compare with other similar benchmarks to verify whether the benchmark proposed in this article is effective.\n2.\tIn the experimental part, this paper verifies different metrics for different OPE methods. However, in Figure 4 and Figure 5, the different methods in the two sets of benchmarks proposed in this article are quite different in different OPE methods. I hope the author can give some comments on the differences between the two sets of evaluation methods.\n3.\tThe author uses the value function in formula 1 to estimate the effect of the strategy. I doubt this method. Because of the attenuation factor here, I think it has an impact on the final value calculated by different methods. Because bellman equation is only an estimate of the value of a certain state, not an absolute strategy benefit, I hope the author will give some explanations here.", "extra_scores": {}, "extra_text": {}}, {"review_id": "29WgIty6Kny", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work describes a set of benchmarks for offline policy evaluation and selection.  The area is important and well-motivated in the introduction.  \n\nThe work builds upon existing benchmark tasks for offline RL.  However, the increment is meaningful.  By providing the triad of data, trained policies, and online evaluation results, authors have isolated the offline evaluation component, controlled for ancillary aspects, and facilitated experimentation.  In other words, experiments are both easier and more useful.\n\nThe work as presented makes little attempt to connect with representation learning which is the focus of the ICLR conference.  It could be improved in this regard, if authors make connections between estimation and representation (e.g., how are model-based estimators related to or employing representation learning?   what about model-free estimators that have to fit a propensity model to the behaviour policy?  how, if at all, do the baselines described in section 5.1 utilize representation learning, or how could they be improved via representation learning techniques?)\n\nThe name of the work is highly unfortunate.  There is nothing deep about the benchmark in the representation learning sense, and the use of the term deep contributes to the general degradation of the term into a meaningless yet popular adjective.  Unfortunately, les jeux sons faits ... do better next time!\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0A0aneBb8l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Significance\n- I view this paper has high significance level. As more and more OPE methods arise, papers rely on different datasets and metrics, which makes them hard to compare, and especially see the \"success\" and \"failure\" mode of this method. We need a benchmark dataset on this domain and this paper moves an important step towards this by providing standardized logged dataset, and various policies in different challenging, complex domains.\n\n2. Novelty\n- There is a very similar work by Voloshin et.al (2019), which gives the empirical evaluation of popular OPE methods across various domains. Though not providing the logged data, they give the logging policy to logged the data, I am concerned about the novelty of this work compared with Voloshin et.al (2019). It would be great if the author could discuss the novelty/improvement over that work.\n\n3. Quality & Clarity\n- This paper has a great structure, metrics/choice of datasets/design of policies are well motivated and clearly stated, it is a very easy to read paper.\n\n- This work gives a clear description of the logged datasets/policies/and the performance of various off-policy method. However, I feel it would be great if the authors could also think, and if possible make the following discussions along with this paper:\n\n(1). it would be great if the authors could give some statistics to measure the property of the datasets, such as a measure of the missing support of logging policy, how difference are the logging policy and target policy?\n\n(2). I know the authors discuss some properties of the environment in words, such as sparse/dense rewards, stochasticity of the environment, etc. It would be great to summarize these environment info into a single table.\n\n(3). A very important class  of estimators in OPE is these kind of interpolated estimators between value-based and weighting-based, which contains a trade-off of bias-variance, such as the work by Phillip and Emma (2016). It also shows in Voloshin et.al (2019)'s empirical evaluations and shows very promising performance. However, it is missing in this paper and it would be excited to see how it works in the D4RL tasks.\n\n(4). A further extension from (1) is that the authors did a great job in doing lots of empirical evaluations in various domains. I saw aggregated performance in main paper and individual task performance is listed in Appendix. It would be great to classify them into groups which depends on  the MDP property, the metric measuring difference of logging/target policy, and etc... This will definitely give some further directions like which method perform well in which setting. This is also shows in Voloshin et.al (2019)'s paper but kind of missing in this work.\n\n(5). One difference compared with Voloshin et.al (2019) is that it seems in the logged dataset, they know the propensity. However, I feel the dataset released along the paper does not have propensity. Will is possible to also add some datasets with propensity included?\n\nOverall, I feel this paper addressing an important problem, however, I feel there is still room for improvement to make them have more impact. I am willing to adjust my score if the authors address some of these issues. \n\nRef: \n1.  Philip S. Thomas, Emma Brunskill  Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning.\n2. Cameron Voloshin, Hoang M. Le, Nan Jiang, Yisong Yue Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes benchmark tasks for offline policy evaluation.  The proposed benchmark tasks evaluate the policy with batch data with respect to three metrics, including the standard mean squared error. The paper also evaluate several baseline offline reinforcement learning methods with the benchmark tasks, which will serve as standard baselines.  All of the reviewers are in favor of the paper.
Decision: Accept (Poster)