Offline Policy Comparison with Confidence: Benchmarks and Baselines
Authors: Anurag Koul
Mariano Phielipp
Alan Fern
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline reinforcement learning
reinforcement learning
benchmark
uncertainty
model based reinforcement learning

Abstract: Decision makers often wish to use offline historical data to compare sequential-action policies at various world states. Importantly, computational tools should produce confidence values for such offline policy comparison (OPC) to account for statistical variance and limited data coverage. Nevertheless, there is little work that directly evaluates the quality of confidence values for OPC. In this work, we address this issue by creating benchmarks for OPC with Confidence (OPCC), derived by adding sets of policy comparison queries to datasets from offline reinforcement learning. In addition, we present an empirical evaluation of the "risk versus coverage" trade-off for a class of model-based baselines. In particular, the baselines learn ensembles of dynamics models, which are used in various ways to produce simulations for answering queries with confidence values. While our results suggest advantages for certain baseline variations, there appears to be significant room for improvement in future work.

Under review as a conference paper at ICLR 2023
OFFLINE POLICY COMPARISON WITH CONFIDENCE: BENCHMARKS AND BASELINES
Anonymous authors Paper under double-blind review
ABSTRACT
Decision makers often wish to offline historical data compare sequential- action policies at various world states, Importantly, computational tools should produce confidence values for such offline policy comparison (OPC) account for statistical variance and limited data coverage. Nevertheless, there is little work that directly evaluates the quality of confidence values for OPC. In this work. we address this issue by creating benchmarks for OPC with Confidence (OPCC). derived by adding sets of policy comparison queries to datasets from offline re- inforcement learning. In addition, we present an empirical evaluation of the risk versus coverage trade-off for a class of model-based baseline methods. In partic- ular, the baselines learn ensembles of dynamics models, which are used in various ways to produce simulations for answering queries with confidence values. While our results suggest advantages for certain baseline variations, there appears be significant room for improvement in future work.
INTRODUCTION
Given historical data from a dynamic environment, how well can we make predictions about future trajectories while also quantifying the uncertainty of those predictions? Our main goal is to drive research toward a positive answer by encouraging work on specific prediction problem, offline policy comparison with confidence (OPCC).
OPCC involves using historical data to answer queries that each ask for: 1) a prediction of which of two policies is better for an initial state and horizon, where the policies, state, and horizon can be arbitrarily specified, and 2) a confidence value for the prediction. While here we use OPCC for benchmarking uncertainty quantification, it also has utility for both decision support and policy op- timization. For decision support, a farm manager may want a prediction for which of two irrigation policies will best match season-level crop goals. A careful farm manager, however, would only take the prediction seriously if comes with a meaningful measure of confidence. For policy optimiza- tion, we may want to search through policy variations to identify variations that confidently improve over others in light of historical data.
Offline reinforcement learning (ORL) (Levine al. 2020), both for policy evaluation and optimiza- tion, offers a number of techniques relevant to decision support and OPCC in particular. One of the key ORL challenges is dealing with uncertainty due statistical variance and limited coverage of historical data. This recognition has led to rapid progress in ORL, yielding different approaches for addressing uncertainty, e.g. pessimism in the face of uncertainty (Kumar al. 2020 Buckman al.
2020 Jin et al., 2021a] a. Shrestha et al. 2021) ) or regularizing policy learning toward the historical data (Kumar et al., 2019; Peng et al. 2019 Fujimoto & Gul 202T Kostrikov et al. 202T). How- ever, there has been very little work on directly evaluating the uncertainty quantification capabilities embedded in these approaches. Rather, overall ORL performance typically evaluated, which can be affected by many algorithmic choices that are not directly related to uncertainty quantification. A major motivation for our work to better measure and understand the underlying uncertainty quantification embedded in popular ORL approaches for offline policy evaluation (OPE).
Contribution. The first contribution of this paper is to develop benchmarks (Section 4 for OPCC derived from existing ORL benchmarks and to suggest metrics (Section 3.3 for the quality of un- certainty quantification. Each benchmark includes: 1) a set of trajectory data collected in an environment via different types of data collection policies, and 2) set of queries Q, where each

---

Under review as a conference paper at ICLR 2023
query asks which of two provided policies has larger expected reward with respect to speci- fied horizon and initial states. Note that our OPCC benchmarks are related to recent benchmarks for offline policy evaluation (OPE) (Fu et al., 202ID, which includes policy ranking task similar to OPCC. That work, however, does not propose evaluation metrics and protocols for measuring uncertainty quantification over policy rankings. Further, our query sets Q span much broader range of initial states than existing benchmarks, which is critical for understanding how uncertainty quantification varies across the wider state space as it relates to the trajectory data D.
Our second contribution is to present a pilot empirical evaluation (Section 5 of OPCC for class of approaches that use ensembles as the mechanism capture uncertainty, which is one of the prevalent approaches on ORL. This class uses learned ensembles of dynamics and reward models to produce Monte-Carlo simulations of each policy, which can then be compared in various ways to produce prediction and confidence value. Our results for different variations of this class provide evidence that some variations may improve aspects of uncertainty quantification. However, overall, we did not observe sizeable and consistent improvements from most of the considered variations. This suggests that there is significant room for future work aimed at consistent improvement for one more of the quantiiiatttaaniiiiaatiiinoonnnn metrics.
The benchmarks and baselines are made publicly available with the intention of supporting com- munity expansion over time.
2 BACKGROUND
We formulate our work in the framework of Markov Decision Processes (MDPs), for which we assume basic familiarity (Puterman 2014). An MDP is tuple M (S, A. R). where S is the state space, A is the action space, and P(s is, a) is the first-order Markovian transition function that gives the probability of transitioning to state given that action a is taken in state s. Finally, R(s, a) is potentially stochastic reward function, which returns the reward for taking action in state
In this work, we focus on decision problems with a finite horizon h, where action selection can depend on the time step. A non-stationary policy (s, L) is possibly stochastic function that returns an action for the specified state s and time step E {O.....tt h 1}. Given an MDP M, horizon h, and discount factor 7 [0, 1); the value of policy at state s is denoted by V (s, h) E ED (S,, As) So = 8, A, (S.: where St and A, are the state and action random variables at time t. It is important to note that we gain considerable flexibility by allowing for non-stationary policies. For example, π could be an open-loop policy even a fixed sequence of actions, which are commonly used in the context of model-predictive control (Richards] (005). Further, can implicitly represent the action value function Q" (s. a, h) for policy æ by defining a new non-stationary policy n' that takes action at = 0 and then follows x thereafter, which yields V (s, h) Qh, (s. a. h). For this reason, we will focus exclusively on comparisons in terms of value functions without loss of generality.
OFFLINE POLICY COMPARISON WITH CONFIDENCE
In this section, first introduce the concept of policy comparison queries, which are then used to define the OPCC learning problem. Finally, we discuss metrics used in our OPCC evaluations.
3.1 POLICY COMPARISON QUERIES
We consider the fundamental decision problem of predicting the relative future performance of two policies, which we formalize via policy comparison queries (PCQs). A PCQ is a tuple q = (,, 8, àt It) as, where and & are arbitrary starting states, and स policies, h is horizon. and M is a MDP. The answer to a PCQ is the truth value of V (s, h) < V h3. That is, PCQ asks whether the h-horizon value of is greater than π when started in and s respectively.
As motivated in Section I PCQs are useful for both human-decision support and automated pol- icy optimization. For example, if a farm manager wants information about which of two irriga-
Benchmark and baselines: httph:ttt////t.t. /tt::///itttt http:ttwwww.cccccccccccccaacccee
2

---

Under review as a conference paper at ICLR 2023
tion policies, and f. will result in the best future crop yield given the environment state s, then the corresponding PCQ would be (s, n. 8, h) Alternatively, the manager may be interested in whether policy π is better suited to an environmental state 8, which is captured by the PCQ (s, T, 8, π. M- In addition, PCQs can be used as the basis for the classic policy improvement step of policy iteration (Puterman 2014). We elaborate our discussion in Appendix AA.
3.2 LEARNING TO ANSWER PCQs WITH CONFIDENCE
Given an accurate generative model of the environment MDP (M), a PCQ (s, &, ht) ar can be answered via Monte Carlo trajectory sampling to estimate V (s, h) and V. (8, h). Further, the con- fidence in the answer can be arbitrarily improved by increasing the number of sampled trajectories. In this work, we do not assume an environment MDP (M), but instead are provided with an offline data set of environment trajectories produced by one or more unknown behavior policies. We will denote this dataset by D {(s,, 0g r.)] where each tuple corresponds to an observed transitions from state to state 5/ after taking action a, and receiving reward For notational simplicity, we will omit MM from PCQs and value functions; assuming the MDP is implicit through dataset D.
Given a dataset D we would like to learn model for predicting answers to PCQs from query space Q. Here, Q may assert application-specifcc restrictions on states and policies involved in PCQs. A fundamental challenge is that the coverage of D will not necessarily be representative of the dynamics and rewards relevant to answering all queries in Q. Thus, if query answers are being used to inform important decisions, then it is critical for each answer to come with a meaningful measure of confidence that accounts for data coverage and statistical variance. Dealing with this uncertainty is also core challenge for general offline RL (Levine al., (020)), which has lead to number of approaches for addressing it. However, there is little direct evaluation of the uncertainty- handling components.
The above motivates the OPCC learning problem, which provides a dataset D and desired con- straints on the query space Q. The learner should output model
(f. e) composed of: 1) query prediction function f Q + {0, 1}. which returns binary answer for any query in Q, and 2) a confidence function e: Q + [1, u] that maps queries in Q to a confidence value within a bounded interval. Given a query g. the intent is for larger values of c(q) to indicate a higher confidence in the prediction S [q). Note that we do not attach any predefined semantics to the values of c(q) to allow for flexibility of potential solutions. Rather, we focus on defining metries for directly evaluating the quality of uncertainty quantification provided by w. If desired, various methods can be used after learning to calibrate the confidence values of с to meaningful scales (c.g. (Loh) 1987, Nacini et al. 2015). Section 5 discusses possible learning approaches and the baselines evaluated in this paper.
3.3 EVALUATION METRICS
In the following, introduce metrics and brief about their intra-relation could be found in Appendix B Our metrics are designed to evaluate both the query answer (or rank) and confidence; in contrast to OPE metrics used by Tang & Wiens 2021]: et al. (2019).
Fu et al. (2021): Paine al. (2020): Irpan
Area under risk- civv-rovee curve (AURCC). In selective classification, the aim is to reduce predic- tion errors by allowing a predictor to abstain from a prediction if the confidence is below a threshold. The quality of confidence values is thus related to how well they resuli in abstaining when the pre- diction would have been incorrect. This idea is formalized via risk-coverage curves (RCCs) by El- El-Yaniv et al. (2010) and is outlined below.
Let C(g. () be loss function for predicting ğ for query g, e.g. 0/1 loss. Given a test set of queries Q = ......... .9.)....9))))) ). a model w = (f.c). c). and confidence threshold r, the coverage is the fraction of test queries with confidence at least т. The selective risk is the average loss of f over the covered queries. Formally, the coverage and selective risk are respectively define by
Σ Mc(()) 2n] and r(w, Q, = EgeQ [c((g) 2 2r (((((g)(g) 4EQ
cov(w, Q,T)=
[Q]
EnEo [c(g) 2 -|
where is the binary indicator function. Thus, each possible threshold corresponds to a risk- coverage operating point (r(w, Q, T), cov(w, Q)7)). An RCC (El-Yaniv et al.] 2010) is simply
3

---

Under review as a conference paper at ICLR 2023
the risk versus coverage curve of these operating points when sweeping through possible thresholds.
The curve starts at the point (0, 0), since the risk is 0 at zero coverage, and ends at (ry, 1), where ry is the risk of f evaluated aggregate across all thresholds to compute the Area Under the RCC (AURCC). Since lower risk is preferred, we consider lower AURCC indicate better confidence estimation.
all of Q. In order to provide single measure of the RCC quality,
Reverse Pair Proportion (RPP). Our second selective-classffcction metric is reverse pair propor- tion (RPP) from Xin et al. (2021). The main idea that the ordering of confidence values for a pair of queries should reflect the relative prediction loss for those queries. RPP measures how often the confidence value ordering conflicts with the relative losses across all pairs of queries. In particular, a conflict occurs when the loss of 91 is less than 92, but we are more confident about Q2 than The RPP(W, Q) = T Eqi.aEE (1)) < 1(2)); c(g)) < c((2)] is just the fraction of such conflicts; where l(g) = L(g, [q)) is the loss of f on
Coverage Resolution (CRR). Finally, we introduce a new metric on just the confidence functionn A practical difference between different confidence functions is the resolution of values that they output in practice. For example, given set of queries Q, one confidence function ej may result in only three distinct coverage values cor(w, Q, r) across all thresholds, while another confidence function 22 results in [Q] distinct coverage values. All else being equal C2 the preferable function, since it provides higher level of resolution with respect to abstention/ccovrrgee rates. We measure this via coverage resolution at K, denoted C C'R. To compute CRx for w (c, f) and query set Q, the coverage interval [0, 1] is partitioned into K equal bins and we return the fraction of bins which contain (ww, Q. r) for some threshold т. By increasing K we get a finer grained distinction in coverage resolution.
4 OPCC BENCHMARK CONSTRUCTION
In this section, we briefly describe our choice of environments and approach for constructing testing query sets for OPCC benchmark. An extended summary of the benchmark and construction steps can be found in Appendix A
Environments. To support easier adoption of our benchmarks, we selected seven environments and corresponding datasets that are currently used in offline RL research. We describe them as follows:
Maze2d (4 environments): They were introduced in D4RL (Fu al. 2020) and comprise of 2d mazes of different complexities: open, u-maze, medium, and large as illustrated in Figure 2 (Appendix). We use the datasets provided by D4RL, which we refer to as "IM" due to the datasets each having million state transitions.
Gym-Mujoco (3 environments): We consider three locomotion-based environments from OpenAl Gym (Brockman et al., 2016): HalfCheetah, Walker2d, and Hopper. These shown in (Ap For each environment, we use the corresponding D4RL (Fu et al. 2020) datasets that include behavior trajectories of varying qualities. This includes "random, medium, medium-replay, medium-expert, and expert".
Query Set Construction. For each environment, create set of PCQs with ground truth answers. This is done by first generating a set of diverse behavior policies. We choose to have policies for each maze2d environment and 10 policies for each gym-mujoco environment. Secondly, generate a large set of initial states by running these policies along with random policy. This gives us seed states for PCQs that go beyond initial-state distribution of environments. Thereafter, for each horizon h E {10, 20, 30, 40, 50]; we sample states from this set to create PCQs of of the form (s. I, s, ât h) where is a random initial state and π, A a random pair of the learned policies. In addition, we create a set of PCQs of the form (s, n, 8, ft, h) in the same way, except that two random initial states are used instead one.
OPCC BASELINES
In this section, describe the class of baselines that will be made available with the benchmarks and included in our pilot experiments (Section()). Recall that each baseline must provide a prediction function f and confidence function that are derived from the dataset D. Perhaps the most natural
4

---

Under review as a conference paper at ICLR 2023
approach for f to answer a PCQ (s, 3,, f,) is to estimate and then compare the relevant values using OPE. The corresponding confidence function c might then be based on the uncertainty of the value estimates.
There are at least two types of OPE approaches to consider: model-free and model-based. Model- free approaches, such as fitted Q-evaluation (Ernst et al. 2005) typically learn Q-function Q* (s, for a given policy æ that can be evaluated for any state and action. Unfortunately, each function learned by such model-free methods is valid for the single policy и and the effective horizon used during training. Thus, answering PCQs involving other policies or horizons requires costly retrain- ing. Since we are seeking an OPCC approach, which can be quickly applied to arbitrary policies, states, and horizons, we instead choose to use model-based approach for our baselines.
Our baselines are variants of model-based ensemble approaches, which are one of the most com- mon class of approaches used in model- RL for dynamics modeling and capturing uncertainty (Argenson & Dulac-Aroodd) 2020 Yu al.) 20200 Kidambi et al.] 2020). Overall, our baselines all have the following structure: 1) Learn an ensemble of models { {P:} from 'D that each predict the dynamics and reward of the environment. 2) Use each model in the ensemble to generate estimates of the relevant PCQ values, (= fi) and ($, h)). via Monte- Carlo simulation of the policies. 3) Combine the ensemble estimates to provide prediction and confidence value.
5.1 BASE MODELS
In our experiments, we consider two types of base models for forming ensembles. The first base model is the commonly use Feed-Forward (FF) Gaussian model, which, given the current state/observation and action as input, returns the mean and diagonal covariance matrix of a Gaus- sian distribution over the next state and reward. This model allows for stochastic Monte-Carlo simulations by drawing the next state from the model's Gaussian distribution at each time step. In this work, we use the same FF base-model architecture and training details as MBPO (Janner al.] 2019).
We also consider a recent base model (Zhang et al., 2021 (referred to as Auto-regressive (AR)), which was demonstrated in some cases to improve over the output architecture of FF. Instead of generating all n features of the predicted next state in a single pass, AG auto-regressively samples each feature one time using n forward passes. In particular, to sample state feature of the next state, denoted s1+1. the network receives the usual input & and a as well as the previously sampled state features s.+1 AR then returns the mean and covariance for Gaussian that is used to sample 4 The intuition is that this approach may allow for representing non- aan-Gassiin and multi-modal next-state distributions compared to the uni-modal Gaussian FF model.
5.2 ENSEMBLE LEARNING
Model- approaches to ORL have commonly used ensembles attempt quantify uncer- tainty, e.g. via measures of ensemble- two choices for generating ensembles. The first choice is the standard bootstrapping ensemble ap- proach, which simply trains each ensemble member using a different random weight initialization
disagreement (Janner et al.] 2019). We consider
and bootstrapped dataset D by sampling from DD with replacement [D] times. The intent is that the combination of classic statistical bootstrapping Eiron & Tibshirani (1994) and random initialization will produce a diverse set of ensemble models.
Often, however, is observed that the basic bootstrapping approach does not create enough diversity in an ensemble, which is counter to our motivation of representing uncertainty. For this reason, there are number of proposals for increasing the ensemble diversity, of which, we consider just one in this work. In particular, work motivated by capturing uncertainty in ORL proposed the use of randomized constant priors to increase ensemble diversity (Osband et al. 2018). For each base model, randomized constant prior is produced, which is simply network with random initial weights. The base model is trained as an additive component on top of this prior and the final output is the sum of the two. The intuition is that the constant prior should cause ensemble members to disagree more often in unrepresented parts of the state-space, which will provide a better measure of disagreement-basse uncertainty.
5

---

Under review as a conference paper at ICLR 2023
5.3 PREDICTION AND CONFIDENCE VALUES
Given a PCQ (s, m, 8, n. h) query and ensemble of size N we generate prediction and confidence by first using each ensemble member to generate, via Monte-Carlo simulation, pair of value es- timates of (=(s,h) h) and V (s, This results in a set of N value estimate pairs, denoted by V = {(Vi, Vi)..., (Vw, Vr Given the set V, we describe the three approaches we consider for producing predictions and confidences.
Ensemble Voting (EV). Following (Dietterich) 2000), EV simply returns a prediction for a query based on the majority vote across the ensemble of Vi < Vi. The confidence score is equal to the fraction of ensemble members that agree on the majority vote (in the range [0.5, but re-scaled to fall in the range [0.1].
Paired Confidence Interval (PCI). The PCI confidence value is computed by estimating the ex- pected value of V V for random run of the learning algorithm. The mean estimate is given by E. Vi Vi and the prediction is based on the sign of this estimate. The confidence value is based on computing a percentile confidence intervals on the difference, denoted by [la, Ma]- In particular, it is equal to the largest value of a such that 0 & [/a. Thus, a high confidence value reflects that there is strong evidence that the expected difference is either above or below zero (in agreement with the prediction). Confidence intervals are computed based on the distribution.
UnPaired Confidence Interval (U-PCI). This approach makes the prediction in the same way as PCI. but uses umpaired confidence intervals to compute the confidence, which should be expected to be more conservative. In particular, compute or percentile confidence intervals for the means of the V; and V. denoted respectively by [L. ua] and la. da and let the confidence be the maximum value of a for which the confidence intervals do not overlap.
EXPERIMENTS
Our pilot experiments explore the baseline methods on our benchmarks using the proposed metrics for OPCC. It is important to note that these experiments are not intended to identify top performer. Rather our primary goal for these pilot experiments is to assess the adequacy of the benchmarks and metrics for future work and to establish a basic performance bar. Secondarily, are interested to observe evidence or the lack of evidence for certain assumptions that might be drawn about the baselines from prior work.
In our experiments, unless otherwise specified the default model is an ensemble of 100 deterministic feed-forward models and uses EV for the confidence score. For brevity, the figures, tables, and analysis in the main paper are for the Gym-Mujoco environment. We note similaritiesddifferenees for Maze2d in the main paper and refer to the Appendix for figures and tables.
Too hard or too easy? We first assess the degree of difficulty posed by our OPCC benchmark for baselines. Figure I show RCCs of our default model for different data set types (averaged across the different PCQ horizons h) in gym-mujoco. Table T report their corresponding metrics i.e. AURCC, RPP, CR2,, and Loss (or risk) at complete coverage.
First, we consider risk at complete coverage and find that there is no significant difference in risk across dataset type, but varies significantly across gym-mujoco environments. This shows that some environments are more challenging than others due to their underlying complex dynamics and high dimensional observation and action sizes. Also, the risk at complete coverage for maze2d environ- ments with single dataset ( ''mm) is significantly lower than sym-mujoce This is potentially due to data collection via path-planning procedure leading to significant state-action space coverage. Further Medium and Umaze have very small risks without much room for risk improvement, while Large and Open appear to have room for improvement. Second, we consider how the risk varies across coverage values. In most cases, there are thresholds that produce points within the cover- age interval (0,0.5), which indicates a lack of sensitivity in that coverage range. There are typically multiple points between (0.5, and 1], though often just few. Ideally we would hope for a more gradual degradation in risk spanning from no coverage to complete coverage. This suggests that there is significant room to improve the coverage sensitivity, especially in the range [0,0.5].

---

Under review as a conference paper at ICLR 2023
Overall, the current set of benchmarks, with the exception of 2 Mazeld environments, not too easy and appears to offer significant room for improvement in terms of both overall risk and sen- sitivity of the RCCs across coverage values. Likewise, the observation that the risks achieved are significantly less than chance suggest that the benchmarks are not too hard.
Impact of dataset type. The different types of data sets provide different types of coverage of the system dynamics. there evidence that our baselines are able to distinguish among these types? Figure [ shows that the RCCs for different datasets are quite similar for each of the gym-mujoco environment. The AURCC and RPP values in Table | are consistent with these observations. This could be due the diverse coverage of queries across the state space that offer challenges for all datasets. The small variation in RCCs across dataset types could also be due to the models learned from different datasets providing similar types of generalization. It is also possible that differences between dataset types would become more prevalent for smaller versions of the datasets, which an interesting future extension to the benchmarks. Finally no significant patterns for CR in relation to data-set type apparent, which is not surprising since CR is expected to be more heavily influenced by the type of baseline approach.
Table Evaluation metrics for dataset-type comparison in gym-mujoco environments. This includes mean and confidence intervals estimates at 95% confidence level for metrics corresponding to (seed) dynamics trained over each dataset.
DATASET QUALITY
ENV.
AURCC(L)
RPP(.)
CRio(1)
LOSS(L)
RANDOM
0.156 + 0.008
0.045 1: 0.004
0.54 0.043
0.273 : (< 0.001)
MEDIUM
0.133 + 0.001
0.03 4: 0.001
0.4 (< 0.001)
0,26 0.002
HOPPER
EXPERT
0.152 + 0.002
0.04 0.028 + (< 0.001)
0,5 (< 0.001)
0.284
MEDIUM M- M-EXPERT
0.136 + 0.001
0.4 0.001)
0,265 - 0.001
MEDIUM MIRIUMMEEA M-REPLAY
0.128 = 0.001
0.012 # 0.001
0.3 (< 0.001)
0,258 = 0.001
RANDOM
0.206 = 0.001
0.023 + 0.001
0.3 (< 0.001)
0.378 + 0.001
0.222 + 0.001
0.048 + 0.001
0.5 (.00001)
0.374 + 0.002
HALF
0.212 + 0.002
+.000
0.++<<0m
0.361 + 0.002
CHEETAH
MEDIUM-
0.24
0.0002
0.6+
0.387 + 0.003
MEDIUM REPLAY
0.216 +
0.00
0.4
0.368 + 0.001
RANDOM
0,067 1 0.001
0.024
0.04
0.165 + 0.001
MEDIUM
0,069 + 0.001
0.007 + (< 0.001)
0.22
0,156 + 0.001 0,161 + (< 0.001)
WALKER
EXPERT
0,064 + 0.001
0.011 - (< 0.001)
2D
MEDIUM-EXPEE EXPPRT
0.068 + 0.001)
0.007 = (< 0.001)
0.24
0.153 = 0.001
MEDIUM-REPLAY
0.07 0.001
0.005 = 0.001)
0.2 (== 0.001)
0.161 = 0.001
Hopper-v2
HalfCheetah-v2
Walker2d-v2
0.15
0.10
0.2
0.05
0.0
0.0
0.00
0.5
coverage
coverage
coverage
RANDOM
MEDIUM
EXPERT
MEDIUM-EXPERI
MEDIUM REPLAY
Figure I: Selective-risk coverage curves for different gyn-mujjcco environements and dataset types (depicted by different colors). The x-axis spans from no(O) coverage to complete(1) coverage of queries and the y-axis is the risk for the corresponding query coverage. Each risk-coverage point is determined by varying the confidence threshold.
Impact of Query Horizon. Learned dynamics well known to suffer from error accumulation in multi-step rollouts. This leads to the hypothesis that OPCC performance might degrade with increasing query horizons. In Table 2 and Table 13 (Appendix) horizons h averaged across data-set types. As expected, we observe higher AURCCs for longer horizons, which provides positive evidence for the hypothesis. Interestingly, observe that in
provides metrics for various
7

---

Under review as a conference paper at ICLR 2023
Table Evaluation metrics for horizon comparison in gym-mujoco environments. These mean and confidnce (95))) estimates are over 50 samples corresponding to 5 (seed) dynamics trained over different datasets.
ENV.
HORIZON
AURCC(4)
RPP(4)
(R
LOSS(L)
10.0
0.017 = 0.002
0.001 + (< 0.001)
(.2 0.001)
0.048 + 0.002
20.0
0.078 0.002
0.016 0.003
0.003
0.336 + 0.038
0.17 0.003
HOPPER
30.0
0.146 0.004
0.000
0.003
0.284 4 0.005
40.0
0.169 0.007
0.038 0.006
0.432 0.036
0.293 + 0.004
50.0
0.196 0.009
0.047 - 0.007
0.516 0.049
0.334 - 0.006
10.0
0.077 = 0.004
0.008 + 0.001
0.288 + 0.017
0.191 + 0.006
20.0
0.217 = 0.006
0.041 = 0.005
0,416 + 0.042
0.374 + 0.005
HALF
30.0
0.215 = 0.004
0.038 + 0.005
0,404 0.038
0.377 = 0.005
CHEETAI
40.0
0.223 = 0.006
0.049 + 0.006
0.464 + 0.011
0.368 + 0.005
50.0
0.277 = 0.008
0.063 + 0.007
0.516 + 0.054
0.428 + 0.003
10.0
0.011 + (< 0.001)
0.001 + 0.001)
0.22 0.016
0.033 0.003
20.0
0.025 = 0.002
0.003 0.001
0.252 0.042
0.077 : 0.004
WALKER
30.0
0.059 =
+ 0.003
0.132 + 0.004
2D
40.0
0.093
0.017 0.004
0.384 0.049
0.209 0.004
50.0
0.131 = 0.002
0.023 0.005
0.392 + 0.06
0.259 : 0.002
Table 3: Evaluation metrics for uncertainty-type comparison in gym-mujoco environments. These mean and confidence int)rval(95%) estimates over 50 samples corresponding 5 (seed) dynam- ics trained over 5 different datasets.
UNCERTAINTY TYPE
ENV.
AURCC(L)
RPP(L)
CR1o(t)
LOSS(()
RV
0.141 0.005
0.031
0.005
0.428 = 0.034
0.268 + 0.004
HOPPER
PCI
0.135 0.002
0.004 + 0.001
0.2 : (< 0.001)
0.269 + 0.004
U-PCI
0.135 0.003
0.002 0.009
0.216 0.014
0.269 + 0.004
0.219 0.005
0.044
0.005
0.46 0.04
0.374 + 0.004
HALF
0.191 0.002
0.006
0.001
0.2 (< 0.001)
0.373 + 0.004
CHEETAH
0.196 + 0.003
0.014 0.002
0.228 0.018
0.373 + 0.004
WALKER
EV
0.068 + 0.001
0.011 + 0.003
0.3 + 0.051
0.159 + 0.002
PCI
0.078 + 0.001
0.002 + 0.001
0.2 (< 0.001)
0.16 = 0.003
U-PCI
0.076 0.001
0.004 0.001
0.22 = 0.016
0.16 + 0.003
most of the environments we have very low risk for short horizons. In general, we observe AURCCs for h 10 or h = 20 least order of magnitude smaller than for larger horizons across the benchmark. This suggests a possible threshold effect for OPCC with respect to increasing horizon due error accumulation. It also suggests our current baselines are better suited for applications like reliable policy improvement with smaller horizons.
Influence of difffrrnt confidence functions. Table 3 and Table [2]pppeddix) gives metries for our three different uncertainty functions (EV, PCI, and U-PCI) averaged over data-set types and horizons. The results for AURCC and RPP both indicate evidence that the confidence interval ap- proaches (PCI and U-PCI) have an advantage over EV. This is encouraging as it suggests considering other more sophisticated statistical testing approaches may lead to further improvement. However, the results for CR indicate that the confidence interval approaches have significantly less resolution than EV. This may lead poorer performance for probability calibration approaches applied PCI or U-PCI confidence scores. Further work is required to understand this decrease in resolution.
Impact of Dynamics Choices. There are various choices for dynamics architecture. our work, we investigate ablated impact of design choices like size of ensemble, stochasticity, auto-regressivenesss and randomized constant priors. In Appendix E we discuss each these aspects detail. As we increase the ensemble size from 10 to 100, found weak evidence of improvement in AURCC. However, RPP and Coverage resolution tend to increase significantly. With a stochastic model out- putting normal distribution over the next observation, we don't gain significantly any of
8

---

Under review as a conference paper at ICLR 2023
evaluation metrics. AR model compared to FF model marginally( not statistically signifincant) reduces AURCC in some cases as well as increases C'R₃ for gym-mujoco environments only. Fi- nally, we added randomized constant priors to our base model to encourage ensemble diversity. For maze2d environment, we observe slight improvement in AURCC for maze2d environment. whereas RPP and CRx tends to remains same. On the contrary, in the case of gym-mujoco, we generally observe a slight (but statistically insignificant) increase in AURCC, RPP, and C.K..
7 RELATED WORK
Dynamics Learning in RL. There has been much recent interest in learning deep models of dynam-
ical systems to support model-basee RL. Examples from online RL include Clavera et al. (2018); Kurutach al. (2088), which learn one-step observation- based dynamics along with extensions to ensembles Deisenroth & Rasmussen (2011); Chua et al. (2018); Janner et al. (2019): Nagabandi jet al. (020). PILCO (Deisenroth Rasmussen 201 1 Gal et al. 2016) is another model-based RL approach that learns dynamics via Gaussian Processes (Rasmussen) 2003), which are able to capture epistemic uncertainty. However, performance is primarily measured in terms of overall task performance and it is unclear how well uncertainty is actually quantified. Recent work offline reinforcement, such as MBOP(Argenson & Dulac-Arold 2020), MOPO Yu et al. 2020), and MoREL (Kidambi et al.] 2020) has also considered learning dynamics models over observations from fixed, offline data sets. These approaches incorporate uncertainty estimates in different ways (e.g. pessimistic rewards dynamics) and all use ensembles to estimate uncertainty.
Policy Ranking, DOPE (Fu et al. 2021) studies OPE and devises a protocol that measures policy evaluation, ranking, and selection. For this purpose, the approach introduces set of candidate policies along with their expected value over a distribution of initial states. Rather, in our work, we question the ability of system to rank policies from any arbitrary state for a given horizon instead of limiting to initial state distribution only. This can help provide a more comprehensive view of uncertainty estimation across the state space.
Confidence Intervals. We make use of confidence intervals over policy value estimates for an- swering queries. TTomas et al. (2015) also studies confidence interval estimation over policy value estimates using trajectories generated by a different set of policies. Their approach uses importance sampling (IS) for unbiased value estimates, which suffers from high variance leading to loose confi- dence bounds. They also introduce the problem of high confidence off-policy evaluation and produce tighter bounds on estimates using improved concentration inequalities (Massart) 2007). Metelli et al. 2020); [Kzzbrskij al. (2021); Metelli et al. (2021) further reduce variance in this problem by in-cooperating der-decision IS Preccup (2000), power-mean (Bullen) 2013). and self-normalization (Hesterherg) 1995: Owen 2013): respectively.
We extend each above related work dimension in Appendix C
8 SUMMARY
Properly quantifying uncertainty of complex models is major open problem of practical signifi- cance in machine learning. Despite this fact, only small fraction of the work in machine learning attempts address this problem. Further, in areas such as offline RL, where methods for addressing uncertainty are developed, there is very little direct evaluation of uncertainty quantification. In recent years, there has been impressive progress on where quantifying uncertainty is a core problem. This has been largely driven by the availability of benchmarks that lower the overhead for conducting research and comparing methods. Currently, The OPCC problem is rela-
detection for image classification,
there is a lack of such benchmarks for sequential decision-making tively simple problem to state, yet is rich enough to capture the essence of uncertainty quantification for sequential decision making. We hope that the OPCC benchmarks will inspire other researchers to develop new ideas for uncertainty quantification. Indeed, our pilot experiments show there is significant room to improve and that understanding of current mechanisms is incomplete. Fi- nally, we hope that this initial benchmark and baseline contribution is only the initial seed for the community at large contribute as progress is made.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008. 05556, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arKiv: 000000000444 2016.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed- dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
Peter Bullen. Handbook of means and their inequalities, volume 560. Springer Science & Business Media, 2013.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805. 12114, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Picter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pp. 617-629. PMLR, 2018
Giulio D' D'agostini. Bayesian reasoning in data analysis: A critical introduction. World Scientific, 2003.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off Off-policy confidence interval estimation, arXiv preprint arXiv:2010. 2020.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-basse and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML- /), pp- 472. Citeseer, 2011. 1.
Thomas G Dietterich.
Ensemble methods in machine learning, In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Bradley Efron. Better bootstrap confidence intervals. Journal of the American statistical Associa- tion, 82(397): 1987.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Ran EI- Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning.
Journal of Machine Learning Research, 6:503-556, 2005.
Amir-massoud Farahmand, Regularization reinforcement learning. 2011.
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 23, 2010.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072219 2020.
Justin Fu, Mohammad Norouzi, Ofrr Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off- policy evaluation. arXiv preprint arXiv:2103. /6596. 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2 (0l::1006006600 2021.
10

---

Under review as a conference paper at ICLR 2023
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data- Efficient Machine Learning workshop, ICML, volume 4, pp. 25, 2016.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 883. 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXiv:911..01603, 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565 PMLR, 2019b.
Josiah P Hanna, Peter Stone, and Scott Nickum. Bootstrapping with models: Confidence intervals for off- off-policy evaluation. In Thirty- First AAAI Conference on Artificial Intelligence, 2017.
Alexander Hans, Siegmund Duell, and Steffen Udluft. Agent self-assessment: Determining policy quality without execution. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). pp. 84-90. IEEE, 2011.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvári, and Mengdi Wang. Bootstrapping statistical inference for off-policy evaluation. arXiv preprint arXiv:2102.03607, 2021.
Tim Hesterberg. Weighted average importance sampling and defensive mixture distributions. Tech- nometrics, 37(2):185-194, 1995.
Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. Advances Neural Information Pro- cessing Systems, 32, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint arXiv:1906.0822 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021a.
Yue Jin, Yue Zhang, Tao Qin, Xudong Zhang, Jian Yuan, Houqiang Li, and Tie-Yan Yan Liu. Supervised off-policy ranking. arXiv preprint arXiv:2107.01 2021b.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.0595/ 2020.
Iyaa Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation. arXiv preprini urXiv:2007.13609 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning
with fisher divergence critic regularization. In International Conference on Machine Learning. pp. 5774-5783. PMLR, 2021.
Anurag Koul, Varun V Kumar, Alan Fern, and Somdeb Majumdar. Dream and search to control: Latent space planning for continuous control. arXiv preprint arXiv:2010.09832. 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv::906.3 00949, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 04779, 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 10592, 2018.
Ilja Kuzborskij, Claire Vernade, Andras Gyorgy, and Csaba Szepesvári. Confident off- aff-policy eval- uation and selection through self normaizzdl importance weighting. In International Conference Artificial Intelligence and Statistics, pp. 640-648. PMLR, 2021.

---

Under review as a conference paper at ICLR 2023
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint TrXiv:2005.01643 2020.
Wei- Yin Loh. Calibrating confidence coefficients. Journal of the American Statistical Association, 82(397):155-116) 1987.
Cong Lu, Philip Ball, Jack Parker-Holder, Michael Oshorne, and Stephen J Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2021.
Pascal Massart. Concentration inequalities and model selection: Ecole d' d'Elé de Probabilités de Saint-Flour XXXIII-2003, Springer, 2007.
Alberto Maria Metelli, Matteo Papini, Nico Montali, and Marcello Restelli. Importance sampling techniques for policy optimization. J. Mach. Learn. Res., 21(141):1-75, 2020.
Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. Subgaussian and differentiable impor- tance sampling for off- policyllyy evaluation and learning. Advances in Neural Information Processing Systems, 34:81 19 8132, 2021.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv:1312.5602, 2013.
Rémi Munos. Performance bounds in Lp-norm for approximate value iteration. SIAM journal on control and optimization, 46(2):541-561, 2007. 2007.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv preprint arXiv:1906.042: 04733, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligece, 2015.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMIR, 2020.
Ian Osband, John Aslanides. and Albin Cassirer. Randomized prior functions for deep reinforcement learning, arXiv preprint arXiv: 1101118360033.. 2018.
Art B Owen. Monte carlo theory, methods and examples. 2013.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna. Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 191.0:1910 2019.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offlie reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168. PMLR, 2021.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
12

---

Under review as a conference paper at ICLR 2023
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Tim GJ Rudner, Cong Lu, Michael A Osborne, Yarin Gal, and Yee Teh. On pathologies in kl- regularized reinforcement learning from expert demonstrations, Advances in Neural Information Processing Systems, 34:28376-28389, 2021.
Julian Schrittweeser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839);604--69, 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. arXiv preprint arXiv:2104.06294 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Aayam Shrestha, Stefan Lee, Prasad Tadepalli, and Alan Fern. Deepaveragers: Offline reinforce- ment learning by solving derived non-parametric mdps. In International Conference on Learning Representations, 2021.
Aaron Sonabend-W W, Junwei Lu, Leo A Celi, Tianxi Cai, and Peter Szolovits. Expert-superviseed inforcement learning for offline policy learning and evaluation. arXiv preprint 2001: 2020.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 1308)1130--133,, 2008.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Machine Learning for Heaalthaare Conference, pp. 2-35.
PMLR, 2021.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1040-1051, 2021.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci-
sion making. arXiv preprint arXiv:2102.058/5. 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.1322 13239,9...39,,.... 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint 102.::2:022.836:: 2021.
Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprini arXiv:2 104. 13877, 2021.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pp. 11194- 11203. PMLR, 2020.
13

---

Under review as a conference paper at ICLR 2023
A OPCC BENCHMARK SUMMARY
As introduced Section 4 we consider environments for OPCC benchmark. These environ- ments are relatively low-dimensional environments with non-image-based observations. This helps focus initial studies on fundamental OPCC capabilities, rather than simultaneously addressing the additional complexities that enter with lower-level perceptual observations such as images. In the following, we extend our discussion on these environments and elaborate PCQ construction steps. The outlined cenchmark-construutiee schema is generic. which can be followed by others to extend the set of available OPCC benchmarks. Tables and and Figure show a snapshot of OPCC benchmark components.
A.1 ENVIRONMENTS
Maze2d (4 environments). Figure shows different environment configurations. Each vironment has 4D observations giving the position and velocity of the ball being controlled and a 2D action space specifying the direction of movement. The goal in each environment is to control a rolling ballngreen) to reach goal(red) location. For our benchmarks, we used the dense-reward version of the environments. There are no terminal states in these environments and the episode ends after the maximum number of allowed time-steps reported in Table 4 The D4RL trajectory data sets were created by running a path-planning algorithm to navigate in the maze between different start and endpoints.
Gym-Mujoco(3 environments). Figure shows considered gym-mujoco environments. These en- vironments are qualitatively different from Maze2d in that they involve controlling periodic locomo- tion behavior based on continuous states and actions. Rather Maze2d is primarily about goal-based path planning (navigation) rather than controlling low-level locomotion.
Figure 2: Maze2d tasks: open, umaze, medium, and large lefft right).
Figure 3: Gym-Mujoco tasks: half- cheetah, hopper, walker2d (left right)
A.2 QUERY SET CONSTRUCTION.
possible starting point create of PCQs for OPCC the off-policy evaluation (OPE) extension et al. 2021) to D4RL, which includes policies for subset of the environments. In particular, one of the tasks considered is policy ranking, which similar in spirit to OPCC. However, that extension of D4RL does not capture at least two important characteristics of OPCC. First, the evaluation protocols do not explicitly address measuring the quality of uncertainty quantification. Of course, this can be addressed by just extending the evaluation protocol and metries.
Second, OPE ranking task from D4RL currently limited to just ranking policies based on their expected values over the initial state distribution of cach environment. In contrast, OPCC evalu- ations should involve sets of PCQs that cover a wide range states that are both in-distribution and aut-of-distribution relative the offline data Further, it is desirable to select the PCQs in a way that spans some notion of PCQ difficulty. In particular, the notion of difficulty we consider here for PCQ (s, ग, 8, न. h) is directly related the performance gap between the policies, i.e (s fi- V (s, h)] hs) It is expected that all clse being equal, larger gaps will reason in easier discrimination between policies. Indeed one of the initial challenges in developing the benchmarks try create query sets that were not all too easy too hard.
Based on the above considerations, we create the evaluation sets of PCQs for each environment via the following steps.
Step 1: Policy Generation. We first train (Schulman et al. 2017 /formulate multiple policies for each environment to serve the policies used for PCQ construction. For each environment, have
14

---

Under review as a conference paper at ICLR 2023
sufficiently distinct policies in terms of quality and behavior support non-trivial PCQs. In each maze2d environment, we formulated 5 policies for different goal locations. This helps in covering wider state-action space of the grid. However, in PCQs, they evaluated over the actual task. In gym-mujoco, we trained 10 policies for different stride lengths and directions( forward and backward) of motion. Performance of these policies is shared in Table 5
We chose to train new policies, rather than use policies from D4RL, to ensure they would be distinct from the behavior policies used create the D4RL datasets. For each environment, used the corresponding simulator and multiple runs of the PPO algorithm (Schulman et al., 2017) to train a set of policies of varying quality.
Step 2: Initial State Generation. For each environment we generated large set of potential initial states by running episodes of the random policy, the learned policies, and a mixture of random and learned policies. This produced a set of states that covered wide range of the environment that extended well beyond the initial state distributions.
Step 3: Candidate PCQ Generation. For each horizon h {10, 20, 30, 40, 50} we create a set of 1500 randomly constructed PCQs from the initial states and learned policies. This included explicitly creating random PCQs of the form (s, π, f.. h) with random pair of the learned policies. In addition, we create a set of PCQs of the form (8, n. f, h) in the same way, except that two random initial states are used instead of one.
random initial state and n á
Step 4: PCQ Labeling and Selection. For each generated PCQ from step we used Monte- Carlo simulation via the environment simulator to accurately estimate (* h) and V* h) and removed any PCQ having a difference of less than 10 between the value of each side of a query. The motivation is filter out PCQs that the most ambiguous and more likely to act source of noise in evaluations. Finally, for each h, we randomly selected 1500 of the PCQs to include as the benchmark query set.
In Figure 4 we show scatter plots of (V" (s) (s, h)) for the selected set of PCQs for each environment. Notice the lack of PCQs along the diagonal, which corresponds the removal of ambiguous queries. Also note that the PCQs span range of value gaps, which suggests that they span varying PCQs of varying difficult. If these plots showed bias toward only large gap queries, then additional steps would be necessary to ensure that more variation was present in the selected query sets.
Table 4: Information about OPCC Benchmark comprising of environment details, datasets, and queries.
MAX. ENV. STEPS
OBS. DIM.
ACTION DIM.
DATASET TYPE
QUERY COUNT
ENV.
MAZE2D-OPENT VO AAZEZD-MEDIUM-V
150
600
4
2
IM
1500
MAZEZD-UMAZZ--II
300
MAZE2D-LARGE-VI
800
HOPPER-V2
11
3
RANDOM, EXPERT, MEDIUM, MEDIUM-REPLAN MEDIUM- -EXPERT
HALFCHEBTAH-V2
1000
1500
17
6
WALKER2D-V2
A.3 EXTENDING PCQS
we can use PCQs for policy improvement as well. In particular, we can improve over policy и at state by identifying action a' with higher action value than chosen by æ. The corresponding PCQ for testing a' is (s, 3, n'' f,,,., h)M, M. where is the non-stationary policy
As mentioned in Section 3.1
that first takes action a' and then follows π.
In practice PCQs within an application domain need not be restricted to comparing policies via a single reward function. Rather there are often multiple quantities of interest to users. For example, a farm manager may be interested in understanding how two irrigation policies compare across multiple features of the future, such cumulative water usage, plant stress, off, etc. This can
15

---

Under review as a conference paper at ICLR 2023
Table 5: Performance of policies used in PCQs. These policies are trained using PPO (Schulman
et al. 2017) over the original environment task and hand-picked at different performance levels. We report mean( standard deviation) of policy performance over 20 episodes.
ENV.
POLICY-1
POLICY-2
POLICY-3
POLICY-4
POLICY-5
MAZE2D-OPEN-VO
122(10)
53(6)
38(3)
28(9)
25(6)
MAZE2D- UMAZE-VI
189(97)
59(59)
84(26)
33(2)
20(2)
BIZEZD-MEDIUM-VV
344(251)
41(28)
42(57)
41(53)
17(35)
MAZEED-LARGE-VI
436(304)
38(37)
31(95)
3(7)
1(3)
HALFCHEETAH-V2
6444(126)
3117(11)
1552(50)
458(4)
157(
HOPPER-V2
(296)
2045(1)
1508(2)
1238(17)
(007(4)
WALKER2D-V2
(127
1890(364)
1840(267)
1474(11)
670(26)
ENV.
POLICY-6
POLICY-7
POLICY-8
POLICY-9
POLICY-10
HALFCHEETAH-V2
1168(80)
1044(112)
785(303)
94(40)
4(8)
HOPPER-V2
840(45)
851(4)
582(4)
194(1)
82(5)
WALKER2D-V2
470(168)
70(18)
235 (72)
1330(160)
-17707730)
Maze-open
Maze-umaze
Maze-medium
40
40
40
("(s, h)
20
20
20
20
40
20
40
20
Maze-large
HalfCheetah
Hopper
40
250
200
("
20
100
-250
0
20
-500
100
200
Walker2d
(s h)
0
-200
200
vå(š, h)
Figure 4: Scatter plot of the PCQ for each benchmark environment. For each PCQ (,, we plot (*( h) vs. 8 h8,
8, h),
be facilitated by defining reward functions corresponding each feature and issuing the appropriate PCQs.
16

---

Under review as a conference paper at ICLR 2023
B OPCC EVALUATION METRICS INTRA-RELATION
A system gaining on AURCC indicates that it produces low risks at multiple coverage points. But, it doesn't It necessarily help us understand the quality of the confidence produced. Similar AURCC could be achieved by another framework with different set of coverage points. In order to under- stand this, we supplement our primary metric AURCC with C'Rk and RPP. A gain on CRKg informs us about the diversity of coverage points produced by system, in turn informing us about varying assigned confidences. This information supplements our AURCC information as a singleton view of CRgg wouldn't tell us anything about risk. Also, gain on RPP implies relatively low confidences were assigned to queries as compared to correctly answered queries. This could be achieved by bi- nary confidence indicator as well, hiding information about exhibited confidence diversity. Thereby, combination of all three metrics gives better understanding of the uncertainty estimation.
C EXTENDED RELATED WORK
This section extends related work discussed in Section 7
Dynamics Learning in RL. Going beyond observation-prediccio model; COMBO Yu et al., 2021), Muzero Unplugged (Schrittwieser et al. [202], and L.MPPO (Rafailov et al., 2021) inves- ligate learning latent-space dynamics-modee models (Ha & Schmidhuber 2018 Hafner et al. 2019b a Schrittwieser et al., 2020 Koul et al., 2020) for offline RL rather than learning in the observation space. In specific, LOMPO (Rafailov et al. [021] learns ensemble of latent space models with con- straints to have similar latent representation. Also, Yang & Nachum (202T) studies representation for OPE and primarily suggests that unsupervised learning for representation helps improve policy performance.
In all the discussed dynamics learning methods, the prime focus lies in the final task performance evaluation and it's unclear how well uncertainty is actually captured by the models. Though in our work, we restrict ourselves only observation-base models; we emphasize on better understanding of uncertainty. Also, similar to our motivation, Lu et al. (2021) recently compares various uncer- tainty heuristics in model-based based OPE and share various insights such as role of ensemble-size and imagination horizon length.
Uncertainty. Capturing uncertainty has been studied for number of quantities relevant to RL, for example, to capture the variance of Q-values (Chen et al. (021)_ learned model-dynamics Chua et al. 2018), and modeling data collection policies Rudner et al. (20TT). Rather than trying to cover all prior quantities and uncertainty metrics, in this paper, we have chosen to focus exclusively on uncertainty in policy comparisons (i.e. OPCC). This choice is based on the simplicity of OPCC combined with the immediate utility it has in decision making. Indeed, in many decision-making settings, we only need to policies (open- and/or closed-loop), rather than precisely estimate their values. Importantly, advancements made in more refined uncertainty estimation approaches, such as for Q-values, can be evaluated within the OPCC framework and yield advancements.
Policy Ranking. Hans al. 12011) is of the early works that expresses the need of policy comparison with OPE under uncertainty estimation. They estimate uncertainty over OPE using the method of Uncertainty propogation (D' agggsstinn (003) and show results only on discrete MDPs. Similarly, SSonabend-W et al. (2020) also identifies the need to measure uncertainty for policies learned with limited data. In order learn safe policies, their approach uses hypothesis testing for determining uncertainty in policy evaluation for a pair of candidate policies based on sampling from model posteriors. This helps in ranking them and selection of better performing policy over the behavior policy in safe manner. The work, however, was limited to small flat state-spaces and did not explicitly evaluate uncertainty quantification. In contrast, our work produces a benchmark to primarily focus uncertainty quantification of system using offline data, rather than evaluating in terms of overall task performance.
In similar motivation, SOPR-T (Jin et al. 2021b) also considers policy ranking from offline data and additional volicy-value supervision. This is done by learning an encoded representation of a policy using transformer based architecture and a scoring function over the representation. In order to leam the representation, they require a set of pre-defined policies, each labeled by its ground truth value with respect to initial state distribution. Our framework does not assume the availability of
17

---

Under review as a conference paper at ICLR 2023
such policy-value supervision and also puts emphasis uncertainty quantification, which is not evaluated by this work.
Irpan al. (2019) also studies ranking of policies and introduces new OPE metrics. In particular, they study a specific case of binary reward MDPs where all intermediate trajectory steps have rewards and only the last step can have binary reward 0 or 1). This boils down OPE into classi- fication of policies as feasible or catastrophic, where feasibles policies are the optimal policies with return of I and catastrophic policies are failure policies with return of O. Learning this classifier could help to rank a set of policies. They show this classification is useful as a metric for OPE; as the estimated rank correlates ranking based on ground truth values. Though it is useful as a OPE metric, it lacks measurement of uncertainty over the rankings as primarily focused by our metrics.
Confidence Intervals. Another class of approaches is based on statistical bootstrapping (Efrn 1987.. Hanna et al. (2017) bootstraps learned MDP transition models in order to estimate lower confidence bounds on policy evaluation estimates with limited data. Kostrikov & Nachum (2020) suggests that confidence intervals of these bootstrapped estimates are not guaranteed to be accurate. In practice, they are shown to overly confident especially for insufficient sample sizes and under- coverage of the data distribution. They suggest, that, in practice, this issue may be mitigated by inducing noisy rewards and regularization to learn smoother empirical transition and reward func- tions. Evaluating that claim within our OPCC framework is potential direction for future work.
CoinDICE (Dai et al. 2020), and similar methods (Strehl & Littman 2008 Nachum et al. 2019;
Zhang et al. 2020; Hao et al. 2021) progressively focus on confidence intervals for OPE based on the formulation of certain optimization problems. These iterative optimization approaches (Munos)
2007 Munos & Szepesvári 2008 Farahmand et 2010 Farahmand] (011) for estimating policy value and confidence bounds induces a computational overload. This is undesirable in our frame- work which aims to rapidly answer queries of arbitrary horizons and policies making compu- tationally unsustainable. An interesting direction for future work is to consider generalizing this optimization-bann based approach to more flexibly handle arbitrary policies.
D EXPERIMENT IMPLEMENTATION DETAIL
Two additional details are important to note for our experiments. First, as is customary in model- based RL (including ORL), we are using a pre-defined episode termination function rather than a learned one. We have found that this can significantly impact the performance of model-based RL systems and also our OPCC evaluations. Second, we clipped predicted observations and rewards to keep them within the bounds of the available data sets, which is also common practice in ORL that we found to be important.
E DYNAMICS MODEL ABLATIONS
In this section, we extend our discussion on the impact of considered dynamics architecture choices on OPCC benchmark.
Impact of Ensemble Size. We consider the impact of ensemble size for our baselines. Table the results for ensemble sizes ranging from 10 to 100. Our prior expectation was that performance would increase with significant increase in ensemble-size. In general, we do not see statistically sig- nificant differences between ensemble sized for AURCC based on our current experimental budget (i.e. confidence intervals intersect). However, based on trends in the means, there weak evidence of improved AURCC. The exception is HalfCheetah, where for AURCC, the trends is opposite of the expectation. However, the differences in means tends to be small, suggesting that ensemble size is not having a large impact even if more computational budget were devoted to support statistical significance.
For RPP and Coverage Resolution (CR2) there is typically statistically significant improvement from ensemble size 10 to 100. The exceptions are umaze and medium-maze where losses are very small for all ensemble sizes. Overall, however, differences may be due to the ensembles not being diverse enough, or the base models used to construct the ensembles are not accurate enough. These results demonstrate the value of the OPCC benchmarks
relatively small in magnitude. This
18

---

Under review as a conference paper at ICLR 2023
Evaluation metries for ensemble-size comparison in gym-mujoco and mazeld environments. We train (seed) ensemble dynamics of size 100 for each dataset and start with an ensemble of 10 models for OPCC metrics estimation. Thereafter, we incrementally increase their count and determine impact on metrics mean and confidence intervals (95%).
Table 6:
ENSEMBLE COUNT
ENV.
AURCC(L)
RPP(J)
CRio(1)
LOSS(L)
10
0.141 = 0.005
0.02 = 0.004
0.316 = 0.029
0.273 - 0.005
20
0.141 + 0.005
0.024 = 0.004
0.344 =: 0.037
0.272 + 0.004
HOPPER
40
0.141 + 0.005
0.027 0.004
0.388 #
80
0.141 + 0.005
0.03 + 0.004
0.416 0.038
0
100
0.141 = 0.005
0.031 0.005
0.428 0.034
0.268 0.004
10
0.209 + 0.004
0.028 0.004
0.32 0.029
0.378 + 0.004
0.213 = 0.004
0.034 0.004
0.36 0.04
0.376 0.004
HALF CHEETAH
40
0.215 + 0.004
0.039 0.005
0.4
0.374 0..04
80
0.219 + 0.005
0.043 0.005
0,452 0.04 0.04
0.374 0.004
100
0.219 + 0.005
0.044 0.005
0.46 0.04
0.374 0.004
10
0.072 1 0.001
0.007 + 0.002
0.256 + 0.03
0.16 1 0.002
20
0.071 + 0.001
0.008 0.002
0.268 0.038
0.16 0.002
WALKER
40
0.07 0.001
0.01 0.003
0.28 = 0.046
0.16 0.002
2D
80
0.068 = 0.001
0.01 = 0.003
0.284 0.049
100
0.068 = 0.001
0.011 0.003
0.3 = 0.051
0.159 0.002
10
0.033 + 0.004
0.009 (< 0.001)
0.48 + 0.035
0.127 + 0.015
20
0.031 + 0.003
0.01 = (< 0.001)
0.5 0.001)
+117
OPEN
40
0.03 + 0.003
0.011 = 0.001
0.5 0.001)
0.01
80
0.029 = 0.002
0.012 = 0.001
(- 0.001)
0.108
100
0.029 = 0.001
0.012 0.001
0.5 0.001)
0.000
10
0.011 = 0.002
0.002 << 0.001)
0.002 11 << 10000
(3 0.001)
0.073
20
0.009 + 0.001
0.3 0.001)
0.074 0.004 0.004
UMAZE
40
0.008 1 0.001
0.002 T <<< (.000
0.3 0.001)
0.077
80
0.008 1 0.001
0.002 1 < 0.001)
(.3 0.001)
0.075
100
0.008 = 0.001
0.002 (< 0.001)
0.3 (< 0.001)
0.075 0.003
10
0.001 + 0.001)
0.0 << (1000
0.0 (< 0.001)
0.2 0.001)
0.023 + 0.007
20
0,001 = << (.001)
0.0 (< 000(<000 10000
0.2 (< 0.001)
0.024 0.006
MEDIUM
40
0.001 = << (.001)
0.2 0.001)
0.021 0.003
80
0.001 (< 0.001)
0.0 (< (,0000
0.2 0.001)
0.022 0.001
100
0.001 + (< 0.001)
0.0 (< 0.001)
0.2 (< 0.001)
0.022 + 0.001
10
0.168 = 0.044
0.051 TT000
0.6 (4 0.001)
0.307 1 0.061
20
0.149 = 0.039
0.057 0.015
0.78 0.066
0.269 0.065
LARGE
40
0.138 = 0.081
0.056 = 0.011
0.264 0.044
80
0.146 = 0.019
0.061 0.007
0.82 0.035
0.269 0.027
100
0.14 0.015
0.062 0.004
0.82 0.035
0.251 0.029
in being able to explicitly test hypotheses about uncertainty quantification, rather than relying on downstream results that may be impacted by many possible factors.
Randomized Constant Priors. In order to encourage diversity, we introduce randomized constant priors our ensemble models. These are suggested encourage extrapolation diversity, especially on out-of-distributio state-action pairs, which could improve disagreement-boe based uncertainty es- timates. However, when we included the constant priors our model, we didn't find significant improvements in our evaluation metrics as shown in Table 7 and Figure [6] We use the same ar- chitecture as the dynamics model for prior with random weights and scale them with "prior-scale" before adding them to ensemble models. A prior-scale of 0 indicates no usage of prior. In the case of maze2d, we generally observe a slight (but statistically insignificant) reduction in AURCC, whereas RPP and CR tends to remain same. Large maze environment has significant decrease in AURCC with randomized constant prior. On the contrary, in the case of gym- sym-ujjcco, we generally observe slight (but statistically insignificant) increase in AURCC, RPP, and CRK-
19

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-v ddv2
0.15
selective-risk
0.10
0.2
0.1
0.05
0.00
coverage
coverage
coverage
Open
Umaze
Medium
Large
0.075
0.3
selective-rist
0.02
0.050
0.2
0.01
0.025
0.1
0.000
0.0
0.5
1.0
0.0
0.5
1.0
1.0
1.0
coverage
coverage
coverage
caverage
10
20
40
80
100
Figure 5: Selective-risk coverage curves ensemble-count in gym-mujoco (top-row) and maze2d (bottom-row) environments
Table 7: Evaluation metrics for prior-scale comparison in gym-mujoco environments comprising of mean and confidence (9terval(95%) over 50 samples belonging to 5 (seed) dynamics models for each of the 5 datasets. Prior scale of means no randomized constant prior is added.
PRIOR SCALE
ENV.
AURCC(L)
RPP(L)
CR
LOSS(L)
0
0.141 + 0.005
0.031 +. 0.005
0.428 1 0.034
0.268 1: 0.004
HOPPER
S
0.145 1 0.003
0.045 1 0.004
0.616 1 0.05
0.269 1: 0.004
HALF
0
0.219 + 0.005
0.014 + 0.005
0.46 + 0.04
0.374 # 0.001
CHEETAH
0.236 + 0.005
0.067 + 0.004
0.768 + 0.081
0.373 + 0.006
WALKER
0
0.068 + 0.001
0.011 + 0.003
0.3 0.051
0.159 = 0.002
2D
0.057 0.001
0.017 + 0.002
0.44 0.04
0.159 0.002
0
0.029 1: 0.001
0.012 + 0.001 0.012 (< 0.001)
0.5 (< 0.001)
0.107 1: 0.005
OPEN
5
0.032 + 0.001
0.5 (< 0.001)
0.115 = 0.008
0
0.008 + 0.001
0.002 + (< 0.001)
0.3 = 0.001)
0.075 + 0.003
UMAZE
5
0.006 + 0.001
0.002 = (< 0.001)
0.3 (< 0.001)
0.071 = 0.002
0
0.001 + (< 0.001)
0.0 0.001)
0.2 (< 0.001)
0.022 + 0.001 0.02 0.003
MEDIUM
0.0 + (< 0.001)
0.0 (< 0.001)
0.2 (< 0.001)
0
0.14 +14+00000 0.015
0.062 + 0.004
0.82 + 0.085
0.251 + 0.029
LARGE
0.104 + 0.017
0.051 + 0.012
0.82 + 0.035
0.197 0.017
Prior work by Osband al. (2018) demonstrated improvement in end-task RL performance by hav- ing an ensemble of DQN (Mnih et al., (013) models with randomized constant priors. However, explicit analysis of the uncertainty quantification was not provided. Our observations suggests that randomized constant priors do not appear to improve uncertainty quantification at least as measured through our OPCC benchmarks. Further investigation necessary to better understand the perfor- mance differences observed in Osband al. (2018). An interesting direction of future work is to consider other previously proposed mechanisms for improving ensemble diversity within the OPCC framework.
Dynamics Model Types.
We ablated our base model's (Fr) architecture with Autore- gressive(AR) architecture suggested by /Zhang al. (2021). In Table g and Figure 7 we do not
20

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-M
selective-risk
0.2
0.10
0.1
0.05
0.0
0.0
0.00
0.0
0.5
1.0
0.0
0.5 coverage
1.0
0.0
0.5 coverage
coverage
Open
Umaze
Medium
Large
0.075
selective-risk
0.02
0.050
0.2
5
0.05
0.025
0.1
0.00
0.000
0.00
0.0
1.0
0.0
0.5
1.0
0.0
0.0
0.5
coverage
coverage
coverage
coverage
Figure 6: Selective-risk coverage curves for prior-scale in gym gym-mujoco and maze environments
Table 8: Evaluation metrics for dynamics-tyie type comparison in gym-mujoco environments comprising of mean and confidence (5t))val(95%) estimates over 50 samples belonging to (seeds) dynamics models for each of the 5 Here, 'AR' and "FF" implies auto-tegressive model and feed- forward model, respectively.
DYNAMICS TYPE
ENV.
AURCC(J)
RPP(L)
CRio(1)
LOSS(L)
AR
0.139 + 0.007
0.034 + 0.005
0.48 0.042
0.268 0.007
HOPPER
0.141 0.005
0.031 + 0.005
0.428 + 0.034
0.268 0.004
HALF
0.249 0.008
0.068 0.006
0.736 0.087
0.379 0.003
CHEETAH
0.219 + 0.005
0.044 + 0.005
0.46 + 0.04
0.374 0.004
WALKER
0.065 0.002
0.013 0.002
0.356 0.046
0.158 0.002
2D
0.068 + 0.001
0.011 + 0.003
0.3 - 0.051
0.159 0.002
AR
0.034 0.003
0.014 + 0.002
0.5 + 0.001)
0.123 0.006
OPEN
0.029 0.001
0.012 + 0.001
0.5 + (< 0.001)
0.107 0.005
AR
0.007 0.001
0.002 (< 0.001)
0.3 0.001)
0.07 0.002
UMAZE
0.008 + 0.001
0.002 + (< 0.001)
0.3 (< 0.001)
0.075 0.003
AR
0.001 + (< 0.001)
0.0 0.001)
0.2 0.001)
0.031 0.001
MEDIUM
0.001 + (< 0.001)
0.0 (< 0.001)
0.2 = (< 0.001)
0.022 0.001
0.131 0.017
0.06 0.003
T (< 0.001)
0.233 0.044
LARGE
FF
0.14 0.015
0.062 - 0.004
0.82 0.035
0.251 0.029
observe significant evidence in favor of the AR model with respect to OPCC performance. There is a marginal, but no statistical significant reduction AURCC in some cases. We do see an increase in coverage resolution (C)2) for the gym-mujoco environments when using the AR model, while remains the same for the maze2d environments. This may be due to the additional uncertainty propagation that can occur during auto-regressive inference of each dimension, especially in the gigher-dimensionll gym-mujoco cnvironments. Currently our results do not suggest that the ex- tra computational cost of the AR model compared to FF is worthwhile with respect to uncertainty quantification as measured via OPCC. This may be due to the environments not needed to represent multi-modal output distribution, which is where the AR model could have distinct advantage.
Determinism. Our baseline model a deterministic version the stochastic model defined in Chua et al, (2018), trained via regression loss. A classic improvement is to induce stochasticity into
21

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-V
selective-risk
0.2
0.10
0.2
0.1
0.05
0.0
0.5
1.0
0.0
0.5 coverage
1.0
0.0
0.5 coverage
1.0
coverage
Open
Umaze
Medium
Large
0.075
0.03
Selective-risK
0.10
0.050
0.02
0.05
0.025
0.01
0.1
0.00
0.000
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
coverage
coverage
coverage
coverage
autoregressive
feed forward
Figure Selective-risk coverage curves for dynamics-type in gym-mujoco and maze environments
Table 9: Evaluation metrics for deterministic model comparison in gym-mujoco and maze2d envi- ronments. 'True' implies a deterministic model, whereas 'False' implies a stochastic model.
DETER- MINISTIC
ENV.
AURCC(L)
RPP(L)
(o)
LOSS(L)
FALSE
0.138 + 0.002
0.039 + 0.002
0.524 + 0.039
0.26 + 0.003
HOPPER
TRUE
0.141 + 0.005
0.031 + 0.005
0.428 = 0.034
0.268 =: 0.004
HALF
FALSE
0.229 + 0.007
0.054 + 0.006
0.568 + 0.057
0.377 + 0.005
CHEETAH
TRUE
0.219 0.005
0.044 + 0.005
0.46 + 0.04
0.374 #: 0.004
WALKER
FALSE
0.064 + (< 0.001)
0.012 + 0.001
0.328 + 0.024
0.16 + 0.002
2D
TRUE
0.068 1 0.001
0.011 + 0.003
0.3 0.051
0.159 1: 0.002
FALSE
0.037 + 0.001
0.01 (< 0.001)
0.4 (< 0.001)
0.143 = 0.005
OPEN
TRUE
0.029 + 0.001
0.012 + 0-001
0.5 (< 0.001)
0.107 + 0.005
FALSE
0.004 = (< 0.001)
0.001 << 0.001)
0.2 (< 0.001)
0.059 = 0.004
UMAZE
TRUE
0.008 TF 0.001
0.002 (< 0.001)
0.3 1 0.001)
0.075 = 0.003
FALSE
0.0 + 0.001)
0.0 74 0.001)
0.2 (< 0.001)
0.003 T (< 0.001)
MEDIUM
TRUE
0.001 1 0.001)
0.0 (< 0.001)
0.2 1 0.001)
0.022 1 0.001
FALSE
0.152 0.01
0.058 + 0.007
0.54 0.043
0.167 = 0.003
LARGE
TRUE
0.14 + 0.015
0.062 + 0.004
0.82 0.035
0.251 0.029
the model by learning a normal distribution over the next observation rather than point estimate. We experimented with this modification and provide results in Table 9 and Figure g Though, limitations of deterministic models are well-understood for stochastic environments, it turns out we don't gain significantly with stochastic models in pilot run. This is possibly due deterministic nature of maze environments and low stochasticity in gym-mujoco case.
Normalization of input state-space. In Table and Figure investigate the impact of learning dynamics with normalized state-space. Here, 'True' implies the dynamics was learned with normal- ized state-space and 'False' implies otherwise. There marginal performance difference between either choice for MuJoco environments. Maze2D environments show mix of results with normal- ization benefiting Umaze and hurting Large-Maze. Performance of Medium Redium-Maze Maze and Open-Maze Maze-Maze is not impacted significantly.
22

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-v2
0.15
False
True
0.2
0.1
0.05
0.0
0.0
0.00
0.0
0.5
0.0
0.5
1.0
0.0
0.5 coverage
coverage
coverage
Open
Umaze
Medium
Large
0.15
0.075
0.02
selective--tS
0.10
0.01
0.05
0.025
0.1
False
True
0.00
0.000
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
coverage
coverage
coverage
coverage
Figure 8: Selective-risk coverage curves for deterministic choice for dynamics model in mazeld and gym-mujoco environments. 'True' implies deterministic model, whereas 'False" implies a stochastic model.
Table 10: Evaluation metrics for normalize comparison in gym-mujoco environments
ENV.
NORMALIZE
AURCC(L)
RPP(1)
CRio(t)
LOSS(1)
FALSE
0.128 1 0.002
0.017 0.002
0.3 1 0.025
0.255 1: 0.003
HOPPER
TRUE
0.141 1 0.005
0.031 +: 0.005
0.428 1: 0.034
0.268 1: 0.004
HALF
FALSE
0.228 1 0.006
0.053 0.007
0.548 0.07
0.376 1 0.003
CHEETAH
TRUE
0.219 1: 0.005
0.044 1: 0.005
0.46 1: 0.04
0.374 +: 0.004
WALKER
FALSE
0.078 = 0.007
0.013 + 0. 0.005
0.316 + 0.073
0.17 + 0.009
2D
TRUE
0.068 + 0.001
0.011 + 0.003
0.3 + 0.051
0.159 + 0.002
FALSE
0.033 1: 0.001
0.014 << 0.001)
0.5 1: (< 0.001)
0.108 1 0.004
TRUE
0.029 = 0.001
0.012 + 0.001
0.5 (< 0.001)
0.107 + 0.005
FALSE
0.003 + << 0.001)
0.002 1 << 0.001)
0.3 4 (< 0.001)
0.047 1 0.004
UMAZE
TRUE
0.008 +: 0.001
0.002 + (< 0.001)
0.3 :: (< 0.001)
0.075 1 0.003
FALSE
0.0 :: (< 0.001)
0.0 (< 0.001)
0.2 (< 0.001)
0.017 + 0.002
MEDIUM
TRUE
0.001 + (< 0.001)
0.0 :: (< 0.001)
0.2 (< 0.001)
0.022 + 0.001
FALSE
0.215 + 0.027
0.067 + 0.007
0.82 + 0.035
0.402 + 0.037
LARGE
TRUE
0.14 0.015
0.062 + 0.004
0.82 + 0.035
0.251 + 0.029
23

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-v2
selectiverisk
0.15
False
0.2
0.10
0.1
0.0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
coverage
coverage
coverage
Open
Umaze
Medium
Large
0.10
0.075
0.4
selectiv-nss
0.050
0.05
0.01
0.2
0.025
0.000
0.0
coverage
coverage
coverage
coverage
Figure 9: Selective-risk coverage curves for input stale-spce normalization ablation maze and gym-mujoco environments. 'True' implies the state-space is normalized and 'False' implies other- wise.
24

---

Under review as a conference paper at ICLR 2023
F EXTENDED EXPERIMENTAL DATA
In the following sub-sections, we share OPCC metrics for various ablations discussed in our primary experiments(Section (6) regarding dataset quality, uncertainty types, and query horizons. In each table-cell, we show mean and confidence interval at 95% confidence level for corresponding metrics, estimated by evaluating dynamics runs each dataset of the corresponding environment.
F.1 DATASET QUALITY
Table 11:
Evaluation metries for dataset-types comparison in maze environments. This includes mean and confidence intervals estimates at 95% confidence level for metrics corresponding to (seed) dynamics trained over each dataset.
DATASET QUALITY
ENV.
AURCC(L)
RPP(L)
CR1o(1)
LOSS(
OPEN
TM
0.029 + 0.001
0.012 0.001 0.002 + (< 0.001)
0.5 (< 0.001)
0.107 0.005
UMAZE
IM
0.008 + 0.001
0.3 (< 0.001)
0.075 + 0.003
MEDIUM
IM
0.001 (< 0.001)
0.0 - (< 0.001)
0.2 0.001)
0.022 +
LARGE
1M
0.14 0.015
0.062 0.004
0.82 + 0.035
0.251 + 0.029
Open
Umazee
Medium
Large
0.10
0.075
selective
0.050
0.025
0.1
0.00
0.000
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 coverage
1.0
0.0
0.5 coverage
1.0
coverage
coverage
Figure 10: Selective-risk coverage curve for mIm' dataset in maze environments. navigation dataset of / million transactions.
This is the complete
F.2 UNCERTAINTY TYPES
In the following, 'EV, PCI, U-PCI' refer to Ensemble- Paired Confidence interval, and Un- paired Confidence Interval, respectively.
Table 12: Evaluation metrics for uncertainty-type commarnnn in maze environments
UNCERTAINTY TYPE
ENV.
AURCC(L)
RPP(1)
CRio(t)
LOSS(L)
EV
0.029 4: 0.001
0.012 0.001
0.5 (< 0.001)
0.107 + 0.005
OPEN
PCI
0.057 + 0.004
0.012 0.001
0.38 + 0.035
0.168 + 0.008
U-PCI
0.05 + 0.002
0.012 0.001
0.4 (< 0.001)
0.168 + 0.008
EV
0.008 + 0.001
0.002 + {< 0.001)
0.3 (< 0.001)
0.075 + 0.003
UMAZE
PCI
0.035 + 0.002
0.001 + {< 0.001)
(- 0.001)
0.084 + 0.003
U-PC1
0.017 + 0.002
0.001 + 0.001)
0.2 (< 0.001)
0.084 + 0.003
EV
0.001 + {< 0.001)
0.0 0.001)
0.2 0.001)
0.022 + 0.001
MEDIUM
0.001 <<< 0.001)
0.0 (<++<00 0.001)
0.2 0.001)
0.009 + 0.001
U-PCI
0.001 + (<] 0.001)
0.0 (< 0.001)
0.2 0.001)
0.009 + 0.001
0.14
0.062 0.004
0.02300033
0.251 + 0.029
LARGE
PCI
0.139 0.021
0.037 0.018
4.088
0.218 + 0.028
U-PCI
0.155 - 0.014
0.048 + 0.006
0.46 0.043
0.218 + 0.028
25

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfcheetah-v2
Walker2d-M
selective-risk
0.15
0.2
pci
0.10
0.1
0.0
0.0
0.00
0.0
0.5
1.0
0.0
0.5 coverage
1.0
0.0
0.5 coverage
coverage
Open
Umaze
Medium
Large
0.15
0.075
0.02
0.2
0.10
0.050
0.1
0.05
0.025
0.00
0.000
0.00
0.0
1.0
0.0
0.5
1.0
0.0
0.0
0.5
coverage
coverage
coverage
coverage
Figure 11: Selective-risk coverage curves for uncertainty-Pe in gym sym-mujoco and mazeld environ- ments.
F.3 HORIZON
Table 13: Evaluation metrics for horizon comparison in maze environments
ENV.
HORIZON
AURCC(J)
RPP(L)
CRio(t)
LOSS(L)
20.0
0.005 1: (<< 0.001)
0.001 + (< 0.001)
0.2 (< 0.001)
0.029 + 0.001
30.0
0.015 1 0.002
0.007 1 0.001
0.5 0.001)
0.104 : 0.006
0.006
OPEN
40.0
0.036 1 0.002
0.016 0.0001
0.6 0.001)
0.119 0.007
50.0
0.062 0.002
0.0001
0.6 (< 0.001)
0.148 0.006
20.0
(. 0.001)
0.0 0.001)
0.2 0.001)
0.007 0.001
30.0
0.0 0 0.001)
0.0 0.001)
0.2 (< 0.001)
0.012 0.002
UMAZE
40.0
0.006 0,000
0.002 (< 0.001)
0.2 << 0.001)
0.048 0.003
50.0
0.035 + 0.002
0.008 = 0.001
0.4 (< 0.001)
0.195 - 0.007
20.0
0.0 (< 0.001)
0.0 0.001)
0.2 0.001)
0.006 0.001
30.0
0.0 (< 0.001)
0.0 (.0010
0.2 0.001)
0.019 + 0.001
MEDIUM
40.0
0.001 + (< 0.001)
0.0 (,0000
0.2 << 0.001)
0.032 + 0.003
50.0
0.001 1 (< 0.001)
0.001 (< (.0010
0.2 << 0.001)
0.031 = (< 0.001)
20.0
0.028 +: (< 0.001)
0.021 << 0.001)
0.62 T 0.035
0.059 = (< 0.001)
30.0
0.118 1 0.013
0.047 4,000
(< 0.001)
0.293 0.037
LARGE
40.0
0,218 1 0.018
0.087 00000
0.82 1 0.035
0.321 0.027
50.0
0.16 0.018
0.07 = 0.005
0.92 = 0.035
0.247 + 0.038
26

---

Under review as a conference paper at ICLR 2023
Hopper-v2
Halfchestah-v2
Walker2d-v2
0.3
0.4
0.2
0.2
0.2
0.1
0.0
0.0
0.0
1.0
0.5
1.0
0.0
0.5 coverage
coverage
coverage
Open
Umaze
Medium
Large
0.15
0.2
0.03
0.10
0.02
0.01
1.0
coverage
20.0
30.0
40.0
50.0
Figure 12: Selective-risk coverage curves for horizon ablation in gym-mujoco (top-row) and maze2d (bottom-row) environments.
27


Reviews:
[{"review_id": "oMgyvvcdQm", "reviewer": "Reviewer_GVaQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a benchmark for offline policy comparison with confidence (OPCC). The benchmark builds on top of datasets from D4RL by specifying both sets of policy comparison queries (PCQs) that compare a variety of policies at a variety of states and a set of metrics to evaluate the performance of different evaluation algorithms. The paper also provides a set of baseline experiments using model-based policy evaluation with ensemble-based uncertainty quantification.", "summary_of_the_review": "Overall, I think that the paper takes a well-considered approach to benchmarking offline policy evaluation algorithms. I am still worried about the ability of the benchmark to clearly separate different approaches and about some minor details, so I will rate the paper as a weak accept for now. ", "strength_and_weaknesses": "### Strengths\n\n1. The introduction of PCQs that evaluate candidate policies at a variety of initial states is a nice improvement over prior work like DOPE that focuses on evaluation at initial states which may not capture the full effect of distribution shift. This seems relevant for practical applications, as described in the paper.\n\n2. The metrics take into account explicit uncertainty estimates provided by the evaluation algorithm. This is a nice improvement over prior work that evaluates point estimates of policy performance. This also seems like a practically important consideration.\n\n3. The benchmark and evaluation seem to have been rigorously tested with sufficient numbers of tasks, PCQs, seed, etc. \n\n4. The provided code seems to be fairly clean and well-documented which is important for a benchmark paper.\n\n### Weaknesses\n\n1. The main weakness in my mind is that it is unclear whether the benchmark will be able to meaningfully and clearly separate different approaches. This is seen in the experimental results where there are essentially no clear takeaways despite the rigorous experimentation. The paper argues that this is evidence of room for improvement, but I think it may, on the contrary, be evidence that the benchmark data and evaluation sets are constructed such that all reasonable algorithms perform about the same. I understand that it may just be the case that most of these algorithmic decisions do not matter very much, but it is important for a benchmark to provide clear outputs on what works where. I can see two potential solutions that the authors could pursue to partially address this issue: (1) demonstrate that there are naive (but still reasonable) baselines that perform substantially worse than the algorithms that are evaluated in the current version, and/or (2) create some more easily interpretable summary metrics that argue that the current results are actually showing significant differences between algorithms.\n\n2. I also have some worries about the low-level details of the collection of the datasets. First, I agree that it makes sense to filter out the ambiguous policy comparisons (step 4 of section A.2). However, the proposed method of just filtering out queries with difference less than 10 seems quite arbitrary given that the tasks have very different reward scales. In particular, this can be seen in Figure 4, where this filtering does not seem to have much effect in the gym tasks. It seems that it would make more sense to filter based on some more adaptive criteria, perhaps related to the variance of the distribution of returns. Second, there is not sufficient detail in appendix A.2 to understand how the policies and initial states are selected. What reward functions were the policies trained on and for how long? How were the candidate states selected from the rollouts? Third, I am not sure of the reasoning behind the choices of horizons between 10 and 50. This doesn't seem like a totally unreasonable choice, but there needs to be some rationale for using horizons much shorter than the length of the task being considered (1000 for the gym tasks).  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jFAfrc5KUEC", "reviewer": "Reviewer_J8DJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is concerned with the confidence of offline policy evaluation. The contribution is on the one hand that existing benchmarks for offline policy evaluation from (Fu et al., 2021) are extended, and on the other hand that a series of model-ensemble based offline RL methods are evaluated in experiments using the modified benchmark.", "summary_of_the_review": "Although the novelty is only incremental, I still consider the contribution important because it aims at systematic investigation of an important topic. The other points of criticism can be remedied with reasonable effort.", "strength_and_weaknesses": "**Strengths**\n* The paper addresses an important topic.\n* Further development of benchmarks can stimulate research progress.\n* The paper provides a systematic investigation of different approaches to derive uncertainty from model ensembles.\n\n**Weaknesses**\n* The contribution is incremental. It remains unclear whether it contains sufficient novelty.\n* Despite the extensive appendix, some aspects remain unclear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "irnGQInu0r", "reviewer": "Reviewer_ErjK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "First, this paper formalized the problem of \"offline policy comparison with confidence\" (OPCC), where the goal is to answer \"policy comparison queries\" (PCQs) the state-values $V$ of possibly different policies in possibly different states. The answer to PCQs are expected to include both a binary output of whether \"$π_1, s_1$'s value is less than $π_2, s_2$'s\", and a confidence level $c$. It then outlines three evaluation metrics for OPCCs with respect to a space of PCQs, including the AURCC, Reverse Pair Proportion, and Coverage Resolution. \n\nThe second contribution is benchmark construction from two sets of existing RL benchmarks (namely Maze2d and mujoco), describing how the space of PCQs are constructed. \n\nThe third contribution is a pilot experiments applying existing ensemble methods as baselines, and shows there signification future is needed to achieve good performance of OPCC. ", "summary_of_the_review": "This paper is generally clear, but the main contribution needs to be more differentiated from the existing literature, and there appear to be some missing related works. ", "strength_and_weaknesses": "**Strengths**\n- The paper is overall well written, and the problem studied is well motivated and interesting. \n\n**Weaknesses**\n- The main contribution is the new evaluation metrics and the new benchmark construction strategies. I think it needs to be make clearer how it differentiates from past benchmarks like DOPE. \n- I think there needs to be an expanded discussion on the selected metrics, especially since the Spearman rank correlation was not used, which was used in past works including the cited DOPE benchmark by Fu et al., Paine et al. [1] and Tang [2]. It seems to be related to the proposed RPP so it would be good to provide a clarification. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JDIEQpzYoz", "reviewer": "Reviewer_SRAe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new problem in reinforcement learning namely offline policy comparison with confidence (OPCC) and proposes a set of evaluation benchmarks and baseline methods. The problem is similar to the offline policy selection problem but requires the algorithm to output a confidence level as well. The authors propose different evaluation metrics such as Area under the risk-coverage curve (AURCC), Reverse Pair Proportion (RPP), and Coverage Resolution (CR) to evaluate the policy selection and confidence output results.", "summary_of_the_review": "This paper proposes several benchmarks and baseline methods for the problem named offline policy comparison with confidence. The benchmarks are very limited and may not be representative enough.", "strength_and_weaknesses": "Strength: \n1. The problem setup proposed in this paper is very meaningful and realistic. The policy comparison problem is necessary for different offline RL parts: hyper-parameter selection, cross-validation, safe policy improvement, etc. This part does not have enough discussion in previous offline RL literature and a rigorous methodology and benchmark can help the field.\n2. Using different evaluation metrics and the discussion risk versus coverage trade-off is helpful to illustrate some properties of the OPCC problem. \n\nWeakness:\nAs a general benchmark for the OPCC problem, I think the proposed evaluation tasks/datasets are in a very limited scope. \n1. It lacks diversity in the types of environment: 3 of the 7 tasks are control problems in Mujoco simulation. These problems share many common properties such as the magnitude of the horizon, types of state features, action dimensions, and reward distribution, and not all RL application domains share these. The other 4 are the same type of visual maze problem with different maps. The lack of more types of problems could be concerning for a benchmark that could guide future algorithm design.\n2. The evaluation benchmark uses D4RL datasets in the 3 Mujoco tasks, which are known to consist of a \"significant fraction of near-optimal trajectories.\" (Kostrikov et al., 2021, Offline Reinforcement Learning with Implicit Q-Learning). Some other datasets such as AntMaze in D4RL with more different patterns are not included. it is also concerning whether this special structure will prefer a particular type of algorithm.\n3. It lacks domains with discrete action space, either simple control tasks or Atari games.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject