Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation
Authors: Haruka Kiyohara
Ren Kishimoto
Kosuke Kawakami
Ken Kobayashi
Kazuhide Nakata
Yuta Saito
Venue: ICLR 2024 poster (2024)
Keywords: off-policy evaluation
offline reinforcement learning
offline policy selection
risk-return tradeoff
Primary area: datasets and benchmarks
TL;DR: We propose a new evaluation metric for OPE called SharpeRatio@k, which measures the efficiency of policy portfolios formed by an OPE estimator taking its risk-return tradeoff into consideration.

Abstract: **Off-Policy Evaluation (OPE)** aims to assess the effectiveness of counterfactual policies using offline logged data and is frequently utilized to identify the top-$k$ promising policies for deployment in online A/B tests. Existing evaluation metrics for OPE estimators primarily focus on the "accuracy" of OPE or that of downstream policy selection, neglecting risk-return tradeoff and *efficiency* in subsequent online policy deployment. To address this issue, we draw inspiration from portfolio evaluation in finance and develop a new metric, called **SharpeRatio@k**, which measures the risk-return tradeoff and efficiency of policy portfolios formed by an OPE estimator under varying online evaluation budgets ($k$). We first demonstrate, in two example scenarios, that our proposed metric can clearly distinguish between conservative and high-stakes OPE estimators and reliably identify the most *efficient* estimator capable of forming superior portfolios of candidate policies that maximize return with minimal risk during online deployment, while existing evaluation metrics produce only degenerate results. To facilitate a quick, accurate, and consistent evaluation of OPE via SharpeRatio@k, we have also implemented the proposed metric in an open-source software. Using SharpeRatio@k and the software, we conduct a benchmark experiment of various OPE estimators regarding their risk-return tradeoff, presenting several future directions for OPE research.

Published as a conference paper at ICLR 2024
TOWARDS ASSESSING AND BENCHMARKING RISK- RETURN TRADEOFF OF OFF-POLICY EVALUATION
Haruka Kiyohara"
Ren Kishimoto"
Kosuke Kawakami
Cornell University
Tokyo Institute of Technology
HAKUHODO Technologies Inc.
Ken Kobayashi
Kazuhide Nakata
Yuta Saito
Tokyo Institute of Technology
Tokyo Institute of Technology
Cornell University
ABSTRACT
Off-Policy Evaluation (OPE) aims to assess the effectiveness of counterfactual policies using only offline logged data and is often used to identify the top-k promising policies for deployment in online A/B tests. Existing evaluation metrics for OPE estimators primarily focus on the "accuracy" of OPE or that of downstream policy selection, neglecting risk-return tradeoff in the subsequent online policy deployment. To address this issue, we draw inspiration from portfolio evaluation in finance and develop a new metric, called SharpeRatio @k, which measures the risk-return tradeoff of policy portfolios formed by an OPE estimator under varying online evaluation budgets (R:). We validate our metric in two example scenarios, demonstrating its ability to effectively distinguish between low-risk and high-risk estimators and to accurately identify the most efficient one. Efficiency of an estimator is characterized by its capability to form the most advantageous policy portfolios, maximizing returns while minimizing risks during online deployment, a nuance that existing metrics typically overlook. To facilitate a quick, accurate, and consistent evaluation of OPE via SharpeRatio metric into an open-source software, SCOPE-RI Employing SharpeRatio@@ and SCOPE-RL, we conduct comprehensive benchmarking experiments on various estimators and RL tasks, focusing on their risk-return tradcoff. These experiments offer several interesting directions and suggestions for future OPE research.
have also integrated this
INTRODUCTION
Reinforcement Learning (RI. has achieved considerable success in a variety of applications requiring sequential recision-making. Nonetheless, its online learning approach often seen as problematic due to the need for active interaction with the environment, which can be risky, time-consuming, and unethical (Fu et al., 202T; Matsushima et al., 2021) To mitigate these issues, learning new policies offline from existing historical data, known Offline RL (Levine et al. [2020), is becoming increasingly popular for real-world applications (Qin et al. 2021). Typically, in the offline RL lifecycle, promising candidate policies are initially screened through Off-Policy Evaluation (OPE) (Fu jt al. 2020). followed by the selection of the final production policy from the shortlisted candidates using more dependable online A/B tests (Kurenkov & Kolesnikov 2022), shown in Figure II
When evaluating the efficacy of OPE methods, research has largely concentrated on "accuracy" metrics like mean-squared error (MSE) (Uchara et al., 2022 Voloshin et al. 2019), rank correlation (rankcorr) (Fu et al. 2021] Paine et al. 2020). and regret in subsequent policy selection (Doroudi et al.. 2017, & Wiens (022)) I). However, these existing metrics do not adequately assess the balance between risk and return experienced by an estimator during the online deployment of selected policies. Crucially, MSE and rankcorr fall short in distinguishing whether an estimator is underevaluating near- optimal policies or overevaluating poor porr-erformiing ones, which influence the risk-return dynamics in OPE and policy selection different ways. The regret metric solely emphasizes the optimal outcome
"This work was done during their internship at negocia, Inc. Correspondence to: hk844 ccomell.ccu http:///hhhkkaommaaaaahccceeeeee

---

Published as a conference paper at ICLR 2024
cand palicies
candidate policies
top-k policy portpollo
production policy
production policy
offline policy selection
offline policy selection
online A/B
222
I22
π
7T
7mm
Figure 1: (Left) Conventional (off- ooffiy selection directly chooses the production policy via OPE, (Right) Practical workflow of policy evaluation and selection involves OPE as a screening process where an OPE estimator (J) chooses top-k candidate policies that are to be tested in online A/B tests, where & is a pre-defined online evaluation budget. A policy that is identified as the best policy based on the evaluation process will be chosen as the production policy (f*) ))
among the top-k policies selected by OPE, neglecting the potential adverse effects on the environment when implementing suboptimal policies in online A/B testing scenarios.
Contributions. Aiming for a more informative cvaluation-o--OPE, we propose a new evaluation- of-OPE metric called SharpeRatio@k, This metric evaluates both the potential risk and return when deploying the resulting top-k policies in the A/B testing phase. Drawing inspiration from financial portfolio management, we view the shortlisted candidate policies as a policy portfolio formulated by an OPE estimator. This approach allows us to assess the risk, return, and efficiency of an estimator based on the statistics of its policy portfolio. Specifically, we use the Sharpe ratio (Sharpe) (998) to measure OPE efficiency. A policy portfolio is deemed efficiení if it includes policies that substantially enhance the performance of the behavior (data collection) policy (high return), while avoiding inclusion of underperforming policies that could detrimentally affect rewards during A/B testing (low risk). As an initial validation, we showcase in two example scenarios how SharpeRatio@N can discern between high- and low-risk OPE estimators, distinction not captured by any existing metric.
In addition to developing SharpeRatio@k, k, we have also implemented this metric in an open-source software named SCOPE-RL (Kiyohara et al. 2023). SCOPE-RL serves as versatile tool and testing platform for OPE estimators, providing user-friendly APIs and comprehensive insights into the risk-return tradeoff of OPE estimators. Using SharpeRatio @ k and SCOPE-RI, we conduct extensive benchmark experiments with various OPE estimators and RL environments, evaluating their risk-return tradeoffs across different online evaluation budgets (i.e., varying k values).
Our contributions can be summarized as follows:
New Evaluation-of-OPE Metric:
We have introduced SharpeRatio@k, a novel evaluation metric specifically designed to assess the risk, return, and efficiency of OPE estimators, offering a more comprehensive analysis regarding the evaluation of OPE.
Benchmarks and Future Directions: Through example scenarios and a range of RL tasks, we demonstrate that SharpeRatio@1 is capable of measuring the risk, return, and efficiency of estimators whereas existing metrics fail to do so. We also suggest several future directions for OPE research based on the results that SharpeRatio @k produces.
Open-Source Implementation: We have developed and integrated SharpeRatio @a @k into an open-source software named SCOPE-RI. This software is tailored to enable quick, accurate, and insightful evaluation of OPE, enhancing the accessibility and utility of our metric.
2 OFF-POLICY EVALUATION IN RL
We consider a general RL setup, formalized by a Markov Decision Process (MDP) defined by the tuple
(S, A., T. Pr, 7). Here, S represents the state space and A denotes the action space, which can either be discrete or continuous. Let T S x A + (S) be the state transition probability, where T(s'ls |s, a) the probability of observing state s' after taking action a in state P, S x A x IR + [0, 1] represents the probability distribution of the immediate reward, and R(s, a) := P. [r] is the
expected immediate reward when taking action a in state s. π + P(AA) denotes policy, where (als) is the probability of taking action in state 8.
2

---

Published as a conference paper at ICLR 2024
Off-Policy Evaluation (OPE) aims evaluate the expected reward under an evaluation (new) policy (called the policy value of an evaluation policy), using only a fixed logged dataset. More precisely, the policy value is defined the following expected trajectory-wis. reward obtained by deploying an evaluation policy n.
J(T) = E, Barpa (r)
E
Σ 1==
where E (0. 1] is discount factor and p=(r) = p(su) Ii (a, st) ( (relss,,
the probability of observing trajectory under evaluation policy и. The logged dataset D available for OPE is generated by a behavior policy To (which is different from T) as
D ((8 at S1+1 8111, ri) p(0))
II
Te (s (r). t=
Given the logged dataset D, the typical goal of OPE is to develop estimator J(T; 'D) that can accurately estimate the true value of т, i.c., J(r). The complexity of OPE arises from the fact that only have access to logged feedback for actions selected by the behavior policy. This necessitates tackling challenges like counterfactual estimation and distribution shift, both of which significantly contribute high bias and variance in estimations (Liu et al.) ( Thomas et al., (213) (Appendix [A.]] provides detailed overview of some notable OPE estimators developed to address these specific challenges.).
Practical workflow of policy evaluation and selection. Due to the presence of bias-variance issues, we cannot exclusively depend on OPE results for selecting a policy for production. Instead, a more practical approach often involves a combination of OPE results and online A/B testing, offering a balance of speed and reliability in policy evaluation and selection (Kurenkov Kolesnikov (202). Specifically, demonstrated in Figure I (Right), this practical approach starts with the climination of harmful policies using OPE results, followed by A/B testing of the top-k shortlisted policies for a more dependable online assessment. In the following sections, we focus on this practical workflow, aiming to evaluate the effectiveness of OPE estimators in choosing the top-k candidate policies for further assessment in online A/B testing.
Existing evaluation-of-OPE metries and their issues. In existing literature, the following three metrics are often used to evaluate and compare the "accuracy" of OPE estimators:
Mean Squared Error (MSE) (Voloshin et
2019: This metric measures the estimation accuracy of estimator J among of policies II as (1/ITI) Exen
Rank Correlation (Rankcorr) (Fu al. 2021 Paine al., 2020 This metric measures how well the ranking of candidate policies is preserved in the OPE results and is defined as the spearman's 's rank correlation between (J(z)) ren and {J D))=)))
Regret@ @ (Doroudi et al. 2017): :): This metric measures how well the best policy among the top-k candidate policies selected by an estimator performs. particular, Regret measures the performance difference between the true best policy " and the best policy
estimated by the estimator as J (xx*) J( where A" arg max (x;
MSE measures the accuracy of OPE estimation whereas the latter two metrics assess the accuracy of policy selection. By combining these metrics, we can quantify how likely an OPE estimator can choose a near-optimal policy in policy selection (Figure (Left). However, a significant limitation of these metrics their failure to assess the potential risk associated with implementing harmful policies, a frequent occurrence in more practical two-stage selection workflow having online A/B tests as a final process (Figure [ (Right)). For example, in the scenario depicted in Figure all three metrics yield identical evaluations for both estimators X and Y. Yet. with estimator X underestimating the value of near-optimal policies and estimator overestimating that of detrimental policies, there is marked difference in their risk-retumn tradeoff, which existing metrics overlook by design. This discrepancy underscores the need for a new evaluation-on -ffoai metric that can accurately quantify an estimator's return tradeoff, going beyond mere accuracy assessment.
3

---

Published as a conference paper at ICLR 2024
estimator
estimator Y
estimated policy value
estimated policy value
12- 10-
12-
10-
estimator X
estimator Y
MSE
11.3
11.3
Rankcorr
0.413
0.413
Regret
0.0
0.0
true policy value
true policy value
Figure A toy example illustrating the situation where existing metrics (MSE, Rankcorr, and Regret) fail to evaluate the risk in the off-polic selection (OPS) task. Both estimators X and Y conduct OPS on the same set of candidate policies. underestimates the values the policies indicated by the black dots while Y overestimates them. The shaded regions show the top-3 policies (policy portfolio) selected by each estimator, indicating that Y is riskier than since Y includes worse policies in its policy portfolio. Nonetheless, existing metrics give completely identical evaluations for X and Y.
SharpeRatio@k (f(m)) = 8.0)
1.8
numerator (best@k ((m)))
denominator (std@k)
SharpeRatic
4,2
be J(moa)
0.8
sto@k
3.9
6.0
estimator
3.8
estimator
estimator
8 policies deployed (K)
# of policies deployed (k)
of policies deployed (k)
Figure 3: Evaluating estimators X and Y in the toy example of Figure 2 with SharpeRatio@k. @k.
3 EVALUATING THE KISK-RETURE TRADEOFF IN OPE VIA SHARPERATIO @
Driven by the inadequacy in assessing risk-return tradeoff in existing literature, we introduce a novel evaluation metric for OPE estimators, which we call SharpeRatio@* k. This evaluation-of-OP0 metric conceptualizes the top-k candidate policies selected by an estimator as its "policy portfolio" inspired by financial risk-return assessments (Connor et al. 2010 Sharpe 1998).
SharpeRatio k is the proposed metric formulated to assess the risk-return tradeoff of an OPE estimator, considering a predefined online evaluation budget It defined as follows:
SharpeRatio@k(J) best@ l: J(mb)
std@k(j)
where best@k(J) represents the value of the highest-performing policy within the top-k policies under the estimator J (the policy portfolio of J), and std@k(] refers the standard deviation of policy values among these top-k policies selected by the estimator. These are defined more precisely as:
Σ J(m) )(m
best Bast@kl (ost@k(J)== :- max J(T), std@k(j) = TE)T(())
Σ (E)L(())
(((T)
xEII(
where (J) denotes the set of top-k policies as determined by the estimated policy values under estimator J (i.e., the policy portfolio formed by j).
Note that we include the behavior policy n, as one of the candidate policies when computing SharpeRatio@kk, @k, and thus it is always non-negative and behaves differently given different ma-
4

---

Published as a conference paper at ICLR 2024
estimnatd policy value
20-
estimator
estinnaii policy vvalue
-2
estimator
estimator W
estimator Z
MSE Rankcorr Regret@1
60.1
58.6
0.079
0.023
9.0
9,0
is true pollcy value
10 true palicy value
SharpeRatio@k ([mm) - 5.0)
SharpeRatio@k (((m)) 10.0)
SharpeRatio@k (f(m)) 15.0)
SharpeRatio
6-
estimator W
estimetor
SharpeRatio
1.5
SharpeRatio
0.6
7.0
4-
0.4
0.5
0.0
0.0
20
15
20
of policies deployed (k)
of policies deployed (k)
of policies deployed (k)
Figure 4: A toy example illustrating the case of evaluating a conservative OPE (estimator W) and uniform random selection (estimator Z) with conventional evaluation-of-O0 metries (the right top table) and SharpeRatio@& k (the bottom figures).
Our SharpeRatio @k calculates the return (best @ kt) relative to a risk-free bascline (J(m))), while incorporating risk (std@k) k) in its denominator Reporting SharpeRatio@t k under varying online evaluation budgets, i.e., different values of k is particularly useful to evaluate and understand the risk-return tradeoff' of OPE estimators. Below, we showcase how SharpeRatio @k provides valuable insights for comparing OPE estimators in two practical scenarios while the existing metrics fall short.
Scenario 1: Overestimation Underestimation. The first example is the one illustrated in Figure 2 where current evaluation-of-OPe metries fail to assess the risk of overestimating the effectiveness of underperforming policies, producing identical values for estimators X and Y. In this context, SharpeRatio@ k offers more valuable insights. As evident in Figure 3 it assigns a higher score to estimator X than to Y. To delve into the mechanisms of SharpeRatio @ K, we have separately charted its numerator (return) and denominator (risk) in Figure 3 This breakdown of SharpeRatio@e k reveals that the return (best@ l: () J(x)) is identical for both and Y, but the risk (std@) k- (J) is substantially greater for estimator Y. This is because Y overestimates the value of underperforming policies, thereby increasing the risk of implementing these harmful policies in future online A/B tests. Consequently, in this particular instance, estimator X a better choice than Y, a distinction that existing metrics utterly fail to recognize.
Conservative vs. Random. In our next example, we compare a conservative OPE estimator (estimator W, which consistently underestimates) with a uniform random estimator (es- timator Z), as depicted Figure 4 Similarly to the prior example, the table in Figure 4 indicates that conventional metrics yield nearly the same values for W and Z, despite their distinetly different behaviors. This obscures the selection of the more suitable estimator. SharpeRatio k excels in identifying more effective estimator, considering the risk-retum tradeoff and the specific problem instance. Specifically, Figure Sngureeddemonseee demonstrates how SharpeRatio @k evaluates estimators W and Z under three different behavior policies, each with varying effectiveness (J(mp) 5.0, 10, 15, where higher values signify more effective behavior policy TT). The results show that when is less effective (TL) = 5.0). SharpeRatio@@* @k favors estimator W for its lower risk. Conversely, with a moderately effective т6 (J(,,) = 10), SharpeRatio k shows no preference between the two estimators, suggesting their similar efficiencies in this context. Lastly, scenarios where M, highly effective (J(mT) = 15), estimator Z is preferred by SharpeRatio @k k due to its higher potential for selecting superior policies compared to n.. Thus, SharpeRatio @ k proves to be a more informative and practical metric than conventional accuracy metrics, providing crucial insights for selecting the most efficient estimator in relation to their risk-retumm return tradeoff.
Scenario 2:
By subtracting the baseline in the numerator, we can avoid undesired edge case where estimator that chooses only policies that worse than the behavior policy but accidentally has small std evaluated highly.
5

---

Published as a conference paper at ICLR 2024
Open-Source Implementation. In addition to developing SharpeRatio@k, to facilitate its concise and precise usage in future research and practice, we provide its implementation and examples in SCOPE-RL, an spen-source python software for offline RL and OPE (Kiyohara et al. 2023 Our experiments in the following section are also implemented on top of this software.
4 BENCHMARKING OPE ESTIMATORS WITH SHARPERATIO @
This section evaluates typical OPE estimators with SharpeRatio@k and SCOPE-RL, which results in several valuable future directions and recommendations for OPE research
4.1 EXPERIMENT SETTING
RL tasks. We use a total of 7 tasks including Reacher, InvertedPendulum. Hopper, Swimmer (continuous control) from Gym-Mujuco (Brockman et al. 2016 Todorov al. (012) and CartPole, MountainCar, Acrobot (discrete control) from Gym-Claasii Control (Brockman et al. 2016). Below, we describe the experiment setting for continuous control tasks in detail. A similar procedure is applicable to the discrete control tasks as described in Appendix A.2
Data Generation. We initially train a behavior policy using SAC (Haarnoja et al. 2018 in an online environment and then introduce Gaussian noise to induce stochasticity, creating a logged dataset D(tr) for the offline training of candidate policies. To establish diverse of candidate policies (II, we train CQL (Kumar et al.) (020) and IQL (Kostrikov et al.) 2022). each implemented in d3rlpy (Seno & Imail 2021b. using three distinct neural networks for each method. By adding Gaussian noise with four different standard deviations to each of these trained policies, we generate total of 24 policies. From this assortment, we randomly select 10 policies for each experiment to form our of candidate policies (II). We conduct OPE simulations based on logged dataset D(te) with n = 10. 000 trajectories generated under the behavior policy and 10 different random seeds. Note that we report MSE and Regret normalized by the average and maximum policy value among candidate policies, respectively as nMSE and nRegret@11
4.2 COMPARED ESTIMATORS
We evaluate the following estimators using SharpeRatio @k and conventional accuracy metrics. Appendix A. provides rigorous definitions and detailed descriptions of these estimators.
Direct Method (DM) (Le et al. 2019): This estimator approximates the Q-function (state-action value) Q(+,, a.) = R(s:, az) +
E/= 7" "resse, 4,, π| using the logged data and then estimates the policy value based on this estimated Q-function. This estimator can produce high bias when Q is inaccurate, while its variance is often smaller than other methods.
Per-Decision Importance Sampling (PDIS) (Precup al. 2000): Leveraging the sequential structure of the MDP, this estimator corrects the distribution shift between policies by multiplying the slagle,) This estimator is importance weights of previous actions each time step t, i.c., II molaylaz)) unbiased, but it suffers from high variance, especially when T is large.
Doubly Robust (DR) (Jiang & Li 2016; Thomas & Brunskill, 2016): This estimator combines model-based and importance sampling-base approaches. Specifically, uses DM a baseline estimation and applies importance sampling only to the residual of the discounted reward. In this way, DR often reduces variance comapred PDIS while remaining unbiased.
Marginal Importance Sampling (MIS) (Liu al.
2018 Uchara et al. 2020): This estimator relies
on the marginalized importance weight [s.a]
instead of the per-decision importance weights used in PDIS. MIS reduces variance compared to PDIS because the marginal importance weights do
(8.0)
not depend on the trajectory length T. Marginal Doubly Robust (MDR) is its DR variant.

---

Published as a conference paper at ICLR 2024
Table Spearman's rank correlation in estimator ranking and disagreement in best estimator selection between SharpeRatio 5 and conventional metrics.
metric
Reacher
Inv. Pww.Pendulm
Hopper
Swimmer
CartPole
MountainCar
Acrobot
RankCorr
0.81 (7/10)
0.18 (5/10)
0.70 (0/10)
0.79 (3/10)
(10/10)
0.57 (1/10)
0.38 (10/10)
nRegret
0.33 (9/10)
0.02 (9/10)
0.45 (3/10)
0.45 (10/10)
0.57 (9/10)
-0.77 (10/10)
-0.10 (9/10)
nMSE
0.76 (9/10)
-0.11 (8/10)
0.83 (0/10)
0.06 (4/10)
0.45 (1/10)
-0.20 (10/10)
-0.08 (10/10)
Note: The value outside and inside the parentheses represent the mean of Spearman's rank correlation regarding the ranking of estimators, and the number of trials in which SharpeRatio and other metrics disagree regarding best estimator selection, respectively, calculated over random seeds. The blue font indicates instances where SharpeRatio@S demonstrates high correlation, characterized by the condition (mean std where std is the standard deviation of rank correlation. Conversely, the red font signifies the opposite scenario, where the condition (mean std < 0) applies.
PDIS
MDR
SharpeRatio@k
nMSE
RankCorr
nRegret@)
0.8
SharpeRatio
0.6
0.3-
2-
0.4
0.2
0.0
# of policies deployed (k)
Figure 5: Evaluation-of- SPPE results based SharpeRatio@k (the left figure) and conventional metrics including uMSE, RankCorr, and nRegret@1 1 (the right three figures) in MountainCar. lower value of nMSE and nRegret and higher value RankCorr and SharpeRatio @k mean a better estimator. The stars (*) indicate the best estimator under each metric.
best Inumerator]
std (denominator)
k-th best policy's performance
best pollcy value
policy values
policy''s
-160
20
-160-
-160
10
-180
-700
-200
10
10
of policies deployed (K]
policies deployed (K)
policies deployed
Figure 6: Reference statistics of the top-k policy portfolio formed by each estimator in Moun- tainCar. "best" is used as the numerator of SharpeRatio@k, while "std" is used as its denominator. higher value is better for "best" and "k-th best policy's performance", while lower value is better for "std". The dark red lines show (TT), which is the risk-free baseline of SharpeRatio @ k.
Note that we use self-normalized importance weights for PDIS, DR, MIS, and MDR to stabilize their estimations and estimate the marginal importance weights by BestDICE (Yang et 2020). For DM, DR, and MDR, we use FQE (Le et al. (219)) with a neural network to estimate Q 0z).
4.3 RESULTS
This section discusses key observations in our benchmark experiments. Appendix A.3 provides more comprehensive results and discussion with more RL environments and metrics.
How does SharpeRatio@k differ from conventional metrics? Table presents the Spearman's rank correlation between SharpeRatio@5 and conventional metrics (RankCorr, nRegret, and nMSE). The findings reveal that SharpeRatio@5 @ correlates with conventional metries in most of the tasks
7

---

Published as a conference paper at ICLR 2024
tested, notably Reacher, Hopper, Swimmer, and CartPole. However, better estimators under conventional metrics (higher RankCorr and lower nRegret and nMSE) do not necessarily equate to higher SharpeRatio@5 values in other environments (i.e., InvertedPendullum and MountainCar). A significant deviation of SharpeRatio@3 5 from other metrics is particularly observed with nMSE and nRegret. This divergence is understandable, as nMSE focuses exclusively on the estimation accuracy of OPE without considering the effectiveness of policy selection, while nRegret measures only the performance of the best policy, neglecting the risks of other selected policies. In contrast, SharpeRatio@5 @ exhibits some correlation with RankCorr, which assesses the alignment of policy performances. However, the optimal estimator identified by SharpeRatio@ @5 often differs from that chosen by RankCorr, due to the fact that it can overevaluate the effectiveness of estimators that may choose harmful policies in the top-k policy selection. fact, SharpeRatio@@ @5 and RankCorr do not agree on the best estimator in more than 50% of trials (36 trials out of 70), followed by 42 trials (out of 70) for nMSE and 59 trials (out of 70) for nRegret, across 7 different tasks and 10 random seeds as shown in the parentheses in Table T The following section will investigate specific environment, MountainCar, where SharpeRatio @k and conventional metrics diverge substantially.
SharpeRatio@k is a more appropriate metric than conventional ones. This section delves into a qualitative analysis using the MountainCar example, where SharpeRatio demonstrates relatively weak correlation to RankCorr and displays an inverse conclusion with nMSE and nRegret. Figure S contrasts the benchmark results obtained with SharpeRatio@@ @k under various online evaluation budgets k against those derived from conventional metrics. Figure provides reference statistics for the top-k policy portfolios generated by each estimator. Importantly, the performance of the "k-th best policy" showcases how well the policy, ranked k-th by each estimator, performs. These findings illustrate that the preferred OPE estimator varies significantly depending on the evaluation metrics used. For example, MSE and Regret identify MIS as the best estimator, whereas RankCorr and SharpeRatio 7 favor DM, and SharpeRatio 4 selects PDIS. A closer examination of these three estimators, through the reference statistics in Figure 6 reveals that conventional metrics tend to ignore the risks associated with OPE estimators that mistakenly include suboptimal policies in their portfolios. Specifically, nMSE and nRegret overlook the risk of MIS being worst-case estimator for 3. Moreover, RankCorr fails to recognize the risk of PDIS being a near worst-case estimator for k 2 5, and it incorrectly ranks PDIS higher than MDR, which avoids deploying suboptimal policy until A = 8, 9. In contrast, SharpeRatio@ k adeptly discerns the varied characteristics of policy portfolios and identifies a safe and efficient estimator that adjusts to the specific budget (k) and problem instance (J(T))) Overall, the benchmark results suggest that SharpeRatio k provides a more pragmatically meaningful comparison of OPE estimators than existing accuracy metrics.
SharpeRatio @k and benchmark results suggest future research directions. The benchmarking results with SharpeRatio @k k provide the following directions and suggestions for future OPE research.
Future research should perform evaluation-of-OPe based on SharpeRatio @k: The findings from the previous section suggest that SharpeRatio@ @ k provides more actionable insights compared to traditional accuracy metrics. The benchmark results using SharpeR- atio@k k sometimes significantly differ from those obtained with conventional accuracy metrics. This highlights the importance of integrating SharpeRatio@e k into future research to provide more insightful evaluation-of-OPP regarding the risk-return tradeoff and efficiency.
2. A new estimator that explicitly optimizes the risk-return tradcoff: While DR and MDR are generally regarded as advanced in existing literature, they do not consistently outperform DM, PDIS, and MIS according to SharpeRatio@) This is attributable their lack of specific design for optimizing the risk-return tradeoff and efficiency. Consequently. a promising research avenue would be to create new estimator that explicitly focuses more on optimizing this risk-return tradeoff than existing methods.
3. A new estimator selection method: significantly across different environments, underscoring the need for adaptively selecting the most suitable estimator for reliable OPE. Given that existing estimator selection methods predominantly focus on "accuracy" metrics like MSE and Regret [Lee et al., 2022 Su et al., 2020; Udagawa et al. 2023 Xie & Jiang 2021 Zhang & Jiang (021). there intriguing opportunity for future research to develop a novel estimator selection method that considers risks and efficiency.
The results show that the most efficient estimator varies
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Evaluation-of-O0 Metries The pursuit of accurately evaluating counterfactual policies has tradi- tionally been a comnestonn of OPE research (Uchara et al. 2022 Saito et al. 20221b) Consequently, the Mlean-qquared Error (MSE) has become the go-to "accuracy" metric, with the comprehensive study by Voloshin et al. (2019) exploring how the MSE of OPE estimators varies with trajectory lengths and degrees of distribution shift. In a different approach, Doroudi al. (2017) highlights instances of biased policy selection caused by OPE and advocates for evaluating estimators based on their capability to identify the most effective policy, aligning with the concept of the regret metric. Subsequent benchmarks (Fu al., 2021 Tang & Wiens (021)) have compared estimators on their regret and rank correlation in downstream policy selection, but these metrics remain focused solely on the "accuracy" of policy selection, neglecting the risk-return tradeoff in subsequent A/B tests.
In contrast, SharpeRatio @k brings a novel perspective by emphasizing the risk-return tradcoff in OPE, drawing upon principles from portfolio management (Sharpe) 1998). We demonstrate that SharpeRatio can identify the most efficient estimator under various behavior policies, offering valuable insights for more effective estimator evaluation and selection in both academic and practical settings. Our research aligns somewhat with the findings of Kurenkov & Kolesnikoy (2022), who show that preferred offline RL algorithms vary depending on the number of hyperparameterss tested online. However, their work only assesses the expected performance of the best policy in the top-k deployment, overlooking potential risks encountered during online testing. Additionally, their focus is on comparing offline RL methods, whereas our study centers on the evaluation of OPE.
Risk-Return Tradeoff Metries in Statistics and Finance Risk assessment plays a pivotal role in high-stakes fields such as finance, autonomous driving, and healthcare, ensuring safety and informed decision-making Commonly used risk assessment metries include quartile measures like the a- quartile range, Value at Risk (VaR), and Conditional VaR (CVaR). For instance, CVaR calculates the average outcome in the worst a% of cases. These metrics are recognized for effectively gauging the risk in decisions based their worst-case scenarios (Rockafellar al., 2000), yet they fall short in evaluating the risk-return tradeoff. The Sharpe Ratio, a concept widely applied in various contexts, particularly addresses this by evaluating the return of a decision relative to its associated risk. Originally conceptualized in finance (Sharpe) 1998). it calculates the return on investment (the asset's end-period price minus its purchase price) adjusted for the asset price's 's volatility (standard deviation during the period), thereby facilitating discussions on investment efficiency.
Our work draws inspiration from the Sharpe Ratio as defined in portfolio management (Sharpe 1998) and adapts it for the first time as metric to assess OPE estimators. The core idea is to view the lop-k policies chosen by an OPE estimator as a "policy portfolio". In this context, SharpeRatio @K k measures the "efficiency" of this policy portfolio by balancing the "return" (performance improvement) against the "risk" of implementing suboptimal policies during the A/B testing phase. Our experimental results demonstrate that this application of SharpeRatio @k provides more insightful and relevant evaluation of OPE estimators compared to conventional accuracy metrics.
CONCLUSION
This work proposed the concept of "efficiency" in Off-Policy Evaluation (OPE) and introduced a novel evaluation metric, SharpeRatio@k, Illustrative examples revealed that while existing metrics focus solely on the "accuracy" of OPE and Off-Policy Selection (OPS), they fail to capture the crucial differences in the risk-return tradeoff of various OPE estimators. We further show that SharpeRatio @k effectively assesses the efficiency of OPE estimators, taking into account specific problem instances, such as the performance of the behavior policy and online evaluation budget. The insights gained from our benchmark studies, using both SharpeRatio @k and SCOPE-RL, offer valuable guidance and recommendations for future research in OPE.
ACKNOWLEDGMENTS
We would like to thank Daniel Cao and Romain Deffayet for their valuable feedback on the manuscript. We would also like to thank anonymous reviewers for their helpful feedhack.

---

Published as a conference paper at ICLR 2024
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.cm 01500, 2016.
Gregory Connor, Lisa R Goldberg, and Robert A Korajezyk. Portfolio risk analysis. Princeton University Press, 2010.
Shayan Doroudi, Philip S Thomas, and Emma Brunskill. Importance sampling for fair policy selection. Grantee Submission, 2017.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML' 11, pp. 1097-1104, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072193 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang. Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2002-205266 PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Nan Jiang and Lihong Li. Doubly robust off- aff-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pp. 652-661. PMLR, 2016.
Nathan Kallus and Masatoshi Uehara. Intrinsically efficient, stable, and bounded off off-policy evaluation for reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
Nathan Kallus and Angela Zhou, Policy evaluation and optimization with continuous treatments. In Proceedings of the 21 International Conference on Artificial Intelligence and Statistics, volume 84, pp. 1243-1251. PMLR, 2018
Haruka Kiyohara, Kosuke Kawakami, and Yuta Saito. Accelerating offline reinforcement learning application in real-time bidding and recommendation: Potential use of simulation. arXiv preprint arXiv:2109.08331, 2021.
Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. Doubly robust off- off-policy evaluation for ranking policies under the cascade behavior model. Proceedings of the 15th ACM International Conference on Web Search and Data Mining, pp. 487-497, 2022.
Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi. Kazuhide Nakata, and Yuta Saito. Scope-rl: A python library for offline reinforcement learning and off- aff-policy evaluation. arXiv preprint arXiv:2311.18206 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit g-learning In International Conference on Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp- 1179-1191, 2020.
Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. In Proceedings of the 39th International Conference on Machine Learning, pp. 11729-1121 1752. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints, In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97, pp. 3703-3712. PMLR, 2019.
Jonathan N Lee, George Tucker, Ofir Nachum, Bo Dai, and Emma Brunskill. Oracle inequalities for model selection in offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35. pp. 1949922277, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005. 0000000000000,,,.. 2020.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2021.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang. and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055. 2020.
Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the 17th International Conference on Machine Learning, pp. 759-766, 2000.
Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. arXiv preprint arXiv:2102.000) 2021.
Tyrrell Rockafellar, Stanislav Uryasev, al. Optimization of conditional value-at-risk. Journal of risk, 2:21 42, 2000.
Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. arXiv preprint arXiv:2202.06317, 2022.
Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. Open bandit dataset and pipeline: Towards realistic and reproducible off off-policy evaluation. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021a.
Yuta Saito, Takuma Udagawa, Haruka Kiyohara, Kazuki Mogi, Yusuke Narita, and Kei Tateno. Evaluating the robustness of off-policy evaluation. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 14-1233, 2021b.
Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. arXiv preprint arXiv:2305.. 08062, 2023.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. arXiv preprint Sttv::111.037//0 2021.
William F Sharpe. The sharpe ratio. Streetwise-the Best of the Journal of Portfolio Management, 3: 185, 1998.
Yi Su. Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In Prooceedings of the 38th International Conference on Machine Learning, pp. 9196-9205. PMLR, 2020.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. Machine Learning for Healthcare Conference, pp. 2-35. PMLR, 1021.
Philip Thomas and Emma Brunskill. Data-efficient off policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning. volume 48, pp. 2139-2148. PMLR, 2016.
11

---

Published as a conference paper at ICLR 2024
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-Confidence Off- Policy Evaluation. AAAI, pp. 3000-3006, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno. Policy-adaptiie estimator selection for off- off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2023.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pp. 9659-9668. PMLR, 2020.
Masatoshi Uchara, Chengehun Shi, and Nathan Kallus. review of off-policy evaluation rein- forcement learning. arXiv preprint arXiv:2212.0635. 2022.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, pp. 2094 2104,,100, 2016.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off aff-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: 1.165944, 2019.
Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In Proceedings of the 38th International Conference on Machine Learning, pp. 14404-11413, PMLR, 2021.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Prucessing Systems, 33:6551-65611 2020.
Siyuan Zhang and Nan Jiang. Towards hyperparameter-ffeee policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 12864- 2021.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENT DETAILS AND ADDITIONAL RESULTS
A. DEFINITION OF COMPARED ESTIMATORS
Direct Method (DM) DM is model- based approach, which uses the initial state value estimated by Fitted Q Evaluation (FQE) (Le et al. 2019 It first learns the Q-function from the logged data via temporal-difference (TD) learning and then utilizes the estimated Q-function for OPE as follows.
(TM D)
(2) T )Q(() .a).
(1) .a).
where Q(1), az) is estimated state-action value. DM has lower variance compared other estimators, but often produces large bias due to approximation errors of the Q-function (Jiang & Li 2016 Thomas & Brunskill] 2016).
(Self-Nor )off-Normalized) Per-Decision Importance Sampling (SN))DIS) PDIS applies importance sampling to correct the distribution shift between Tr and т. PDIS also leverages the sequential nature of the MDP reduce the variance of trajectory-wise importance sampling. Specifically, since St only depends on the states and actions observed previously (i.c., So...., St-1 and ao. ........ at-1) and is independent of those observed in future time steps (i.e., $t+1:-.., 8T and d1+1,.... ar), PDIS considers only the importance weights related to past interactions for each time step as follows (Precup al. 2000).
T-1
JPDIS D)
(1) (8) Wo
72
where Wost == IT oo (a ///)/Tb(ay st)) represents the importance weight with respect to the previous action choices for time step t. PDIS is unbiased but often introduces substantial variance (Jiang & Lij 2016)] Thomas & Brunskill 2016). To alleviate the variance issue, the following self-normalized version (Kallus & Uehara 2019) of PDIS in our experiments.
JSNPDIS(T:D) =
Σ 1=1 t=0
Wot
where wo:t/ [>==] Wo:e) is the self-normalized importance weight. The self- oelf-normalized estimator is no longer unbiased, but it retains the consistency of PDIS. Despite their reduced variance compared trajectory- wise variants, (SN)PDIS might still suffer from high variance when T is large.
) Doubly Robust ((NN)RR) DR is hybrid of model- estimation and importance sampling (Dudik et al. 2011). It introduces Q as baseline estimation in the recursive form of PDIS and applies importance weighting only to its residual (Jiang & Li 2016 Thomas &
Brunskill 2016).
(T; D)
M Σ rd
w ( -(sss (.)))) worl Σ T(a [ 1(= .1)
n
where Q works as control variate. DR unbiased and reduces the variance of PDIS when Q(-) reasonably accurate. However, it can still have high variance when the trajectory length 7 is large Fu ct al. 2021) or the action space [A] large (Saito & Joachims 2022 Saito et al.) 2023 is large. Again, we use the following self-normalize version of DR in our experiment.
(ST D)
T-1
(i) Woat
( Σr=
(r
Q
Σ (Q)))
Wit
W 0:1
DEA
*Our open source software, SCOPE-RI, uses the implementation of FQF provided by d3rlpy Seno & Imai
2021
13

---

Published as a conference paper at ICLR 2024
Marginalized Importance Sampling estimators When the trajectory length (T) is large, the variance of PDIS and DR can be very high. This issue is often referred to as "the curse of horizon" " in OPE (Liu et al. 2018). To alleviate this variance issue of estimators that rely importance sampling with respect to the policies, several estimators utilize state marginal or state-action marginal importance weights, which are defined follows (Liu al. 2018 Uehara al. 2020):
p(s, a) == (= aa/dd)dd//s (s, (8,a);
p= = d= (s)
where d= (s) and d= (s) are the marginal visitation probability of the policy (s, a) or s, respectively. We use the state-action marginal importance weight in our experiment, resulting in the following Marginal Importance Sampling (MIS) and Marginal Doubly Robust (MDR) estimators.
(8)
JSNMIS (SNMIS(T;7)) (I;
p
(2)
(2)
1= x==1
( 44
(SNMDR(T;:D)
T(als0 (2)
(30 a))
(1)
AEA
T-1
Σ a(a|s} ( [ -Q(3 acA
+7
E pls
Note that, as given in the above definitions, we use self-normalized variants of these estimators in our experiment.
Extension to the continuous action space When the action space is continuous, the original version of importance weight W, (aalss))mm (z)
[s) a,) ends up rejecting all actions, as I[(a = a,) filters only the action observed in the logged data. To address this issue, continuous OPE estimators (PDIS and DR) apply the kernel density estimation technique to smooth the rejection sampling part as follows (Kallus & Zhou 2018).
T(alse) JaEA (o(a))):)
( a ae da,
Wt
where K--) denotes a kernel function and we use the Gaussian kernel K(x) in our experiments. h is the bandwidth hyperparameter; a large value of h leads to a high-bias but low-variance estimator, while a small value of results low-bias but high- estimator.
V2n exp(-x²110
A.2 OMITTED DETAILS OF THE BENCHMARK EXPERIMENTS
Experiment setting for discrete control tasks The experimental procedures of discrete control tasks are similar to those of continuous control tasks. First, we train DDQN (Van Hasselt et al. 2016)
in an online environment to learn base Q-function (5( 0z) s E -0 7
""'"""/ll (st, az). Then, obtain stochastic behavior policy my, apply the softmax function the estimated Q-function
as (T (52) = Ea'e (x where 7 controls the stochasticity of To (i.c., a large absolute value of leads to unif-unifomm behavior policy, while small absolute value of leads to near- deterministic one). Next, we use the behavior policy R to generate a logged dataset D(tr), which is used to train set of candidate policies (LI). To define candidate policies, we train CQL (Kumar et al. 2020) and BCQ (Fujimoto et al. 2019) with 3 different neural networks to estimate the Q-function ((5t, ae). Then, we define epsilon- greedy style policies as π (a))))) == (1 - c)I {az = a" + </ |A), where a" := max {( a') and controls the degree of exploration. We use different values of e for each of the base Q-functions, thus the total number of candidate policies is 24. From this assortment, we randomly select 10 policies to form candidate policies (IT) each experiment.
exp(dn(e,.a)))
Baseline accuracy metrics In our experiments, we report the value of MSE and Regret normalized by the best performance among the candidate policies in IT. More formally, the following defines the
14

---

Published as a conference paper at ICLR 2024
Table 2: Statistics of the environments used in the experiments
env.
state dim.
action type
# of actions action dim.
max episode steps
Reacher
11
continuous
30 (50)
InvertedPendulum
4
continuous
100) (1000)
Hopper
11
continuous
30 (1000)
Swimmer
continuous
100 (1000)
CartPole
4
discrete
200 (500)
MountainCar
2
discrete
200 (200)
Acrobot
6
discrete
500 (500)
Note: column "max episode steps". the values in parenthesis show the default value set by
gym (Brockman et 2011) which means that use smaller maximum episode steps than default. This avoid extremely large importance weights of PDIS. We also use the latest version of the environments provided by gym==0.26.2.
normalized version of MSE and Regret as used in our experiment.
[)en(((T;))) - ((x))
nMSE(J) [1I]] max (x)) (max (TT) ( min (T))²²
max (TT) max (J) J(x))
nRegret@k(() = max(max ((T), max J(T) min
Note that max ((x) is used when the policy performance is always positive, while max ((T) ein (TT) used when the policy performance can be a negative value.
Environments The following describes the state, action, and reward settings of the environments used in our experiments. Table 2 summarizes the key statistics of each environment.
Gym-Mujoco provides several continuous control tasks that simulate the physical movements of animals or objects. Among them, Reacher a task to control two-jointed robot arms. Provided with target position, the goal of Reacher is to move the fingertip of the robot arm close the target. Thus, the action is 2-dimensional, corresponding to the torque applied to the joints of the arms, while the reward is the distance between the target and the fingertip. There are also penalties applied the reward for taking large actions. The state observation consists of 1 --dimensional features, including similar task, but its goal is push
the positions and velocities of the two arms. Swimmer two-joined robot forward. Thus, the reward is defined by the difference in the robot's 's current and previous positions, which is penalized by the norm of actions. It also observes 8-dimensional states and 2-dimensional actions corresponding to the torque applied joints. Hopper is another task that involves moving the hopper object forward; thus, the reward system is similar to that Swimmer. Its state is I I-dimensional, and its action is 3-dimensional. Finally, InvertedPendulum is a task that involves controlling a cart order to keep the pendulum on upright. The state is 4-dimensional, corresponding to the position, velocity, and angle of the cart and pendulum, and the action is the force applied to the cart, which is I-dimensional. When the pendulum is upright, a positive reward (+1) is given. If the pendulum falls, the reward becomes zero thereafter.
Gym-Classic Control provides several discrete and continuous control tasks. CartPole a task that involves controlling cart to keep the pole on it upright, similar to InvertedPendulum. While the state and reward settings of CartPole are the same as those of InvertedPendulum, the action space for CartPole discrete. Specifically, CartPole has only two actions, which correspond to pushing the cart right or left. Mountain Car is task where the goal is to control a car in order to climb a mountain. The agent receives a reward of until the car reaches the top of the mountain. The state is 2- 2-dimensional, corresponding to the position and velocity of the car, while there are 3 discrete actions: accelerating to the left, accelerating to the right, or no acceleration. Finally, Acrobot is task that involves swinging two-jointed chain upwards. The reward is -1 until the chain is swung up. and discrete actions correspond to torques of +1, -1, or O applied to the chain. The states are 6-dimensional, corresponding to the position and velocity of the two chains.
Difference between our benchmark design and those of (Fu et al. 2021: Voloshin et al. 2019) Previously, DOPE (Fu et al. 2021) and COBS (Voloshin al. 2019) empirically evaluated OPE
15

---

Published as a conference paper at ICLR 2024
estimators in continuous control tasks, including Gym-Mujoco, and in discrete control task (Moun- tainCar), respectively. While we share the choice of environments with these existing benchmarks, experiment design differs from theirs, as we emphasize the evaluation of risks, returns, and efficiency of OPE estimators beyond their accuracy.
The most significant difference between DOPE (Fu et al. 2021) and our work is use of more diverse and stochastic candidate policies. Specifically, while DOPE obtains evaluation policies from different training phases of single SAC (Haarnoja et al. 2018) policy without adding explicit noise to actions, we first train two different offline RL algorithms with three different parameters and then add some stochastic noise to the action with four different noise levels. By doing this, we can obtain a candidate set of policies with varying policy values and create a more complex and diverse distributional shift between behavior and evaluation policies. Another difference the derivation of importance weight. While we use the true (self-normalized) importance weight to define PDIS and DR, DOPE estimates the behavior policy (p(ag]ss) from the logged data. This results in more accurate estimation for PDIS and DR in our experiments compared to those in the DOPE experiments S It worth noting that precise behavior policy estimations are often accessible in applications such e-commerce music streaming, behavior policy probabilities can easily be logged in these systems (Saito et al. 2021a] Kiyohara et al. | 2022). Although COBS (Voloshin et al., (019)) also uses precise importance weights to define PDIS and DR, these estimators provide less accurate estimation than DM because COBS uses smaller datasets, e.g., 256, 512, and 1024 (s, O, 8', rttupess, Finally, while COBS investigates how the MSE of each estimator changes with various configurations, including the degree of distributional shift, data sizes, and the length of trajectories, COBS does not study policy selection as we and DOPE do.
while ours and DOPE use more than 100K tuples [
Models and parameters While our experimental code is publicly available" also describe the implementation details of our experiments. Tables 3 undd[summarii the models and parameters used for learning behavior policies, and Tables 5 II describe those for learning candidate (evaluation) policies. Figures 28 34 also show the (on-policy) policy performance of the behavior and candidate policies used in our benchmark experiments. Lastly, [2 reports the bandwidth hyperparameter (of continuous OPE) and the networks and parameters of FQE and DICE, which are used for estimating the Q-function and marginal importance weights. Please note that we did not conduct extensive hyperparameter tuning for FQE and DICE in line with previous works (Fu et al. 2020) Qin et al. 202T; Voloshin et al., 2019) because hyperparameter tuning of OPE estimators using only logged data is non-trivial task and is of independent interest.
Computation We conducted benchmark experiments for discrete control tasks on an MI MacBook Pro. The computational time greatly varied among benchmark environments, taking about hours for CartPole, 4 hours for MountainCar, and 8 hours for Acrobot. For experiments on continuous control tasks, we used Google Colaboratory utilizing the V100 GPU. These experiments took about hours for Reacher, 10 hours for InvertedPendullmm, 6 hours for Hopper, and 10 hours for Swimmer.
A.3 ADDITIONAL RESULTS AND DISCUSSIONS
Figures 7 3 illustrate SharpeRatio@u @k and its decomposition into returns over risk-free baseline
(i.e., numerator) and risks (i.e., denominator) for varying online evaluation budgets k. As references,
Figures 14 20 show the statistics of the top-k policy portfolio formed by each OPE estimator.
Figures
21 27 compare the results using conventional metrics, SharpeRatio@4 and SharpeRatio@7.
Are low-variance estimators always evaluated as safe and efficient by SharpeRatio@k In the MountainCar example from Section 4 we noted that a low-variance estimator (MDR or DM) was significantly more efficient than the most "accurate" " (in terms of MSE) and the highest-return (Regret) but high-variance estimator, i.e MIS. Although SharpeRatio @1 has been shown to yield reasonable results in the main text, one might wonder; Does SharpeRatio k invariably favor a low-variance estimator over a high-variance one? Our benchmark results definitively indicate that this is not the case, with the Hopper environment serving as a counterexample. In Figure 9 we
'We use small " "max episode steps" setting avoid extreme importance weights that may produce "NaNs". 'The number of data tuples is calculated as "number of trajectorics" x "max episode steps" in our henchmark.
16

---

Published as a conference paper at ICLR 2024
observe that the standard deviation l @s) for DM and MIS among the lowest compared to other estimators and are nearly identical. Yet, SharpeRatio@ k assesses these two estimators quite differently MIS ranks the best among the compared estimators, whereas DM ranks the worst. This discrepancy arises because MIS consistently selects well werl-performing policies, while DM tends to overestimate poor-performing policies, as depicted in Figure [8] SharpeRatio @ k effectively discerns this critical difference by considering both returns (best@k in the numerator) and risks (std@) k in the denominator). These findings underscore that the trade-off between risks and returns crucial, and the denominator (std @ k)) alone is insufficient for determining safe and efficient OPE estimator.
How does evaluation-of-OPP results on SharpeRatio@k change with varying online evaluation budgets (k)? Next, we examine how SharpeRatio @ k varies among small values of k (c.g.. k = 2, 3), medium values (k = 4, 5, 6), and large values (A: 2 7) as illustrated in Figures 27 Overall, we notice that SharpeRatio @ & delivers consistent evaluations for &e 2 4 in most benchmark environments, except for CartPole and MountainCar. This consistency is primarily because the policy portfolio chosen by each estimator tends to converge similar portfolio k increases, resulting in less divergence in the values of SharpeRatio k among OPE estimators. As highlighted in the main text, MountainCar (Figure 13) is outlier, indicating that efficient OPE estimator sometimes vary with the online evaluation budget (k). Furthermore, we observe that SharpeRatio@ k for k S 3 can drastically differ from those for top-k policy selection often involves considerable uncertainty when k S 3. Consequently, we would recommend implementing least policies in online A/B tests as a prudent strategy to achieve stable policy selection results and reliable evaluations of OPE estimator efficiency.
due to sigmificant fluctuations in std This suggests that
How should we interpret the results when SharpeRatio and other metrics behave differently? As highlighted in the main text, the estimator selection results under SharpeRatio @ k sometimes deviate from those determined by other metrics. This divergence occurs because SharpeRatio considers risk, while other metries overlook this factor. Specifically, current metries primarily focus on the accuracy of OPE (MSE), the alignment of resulting policies (RankCorr), or the top-1 policy selection (Regret). In contrast, SharpeRatio evaluates the enhancement in the performance of the production policy, discounting it by the risk incurred during online A/B tests. Ideally, an estimator would excel in all metrics. However, when an estimator is favored by conventional metrics but assessed differently under SharpeRatio, it is indicative of the estimator being high-return yet high-risk. Consequently, in critical scenarios, such as medical fields, SharpeRatio should be given precedence. This because SharpeRatio is the sole metric that acknowledges safety and can adeptly measure the risk-return tradeoff in the two-stage policy selection process, which involves online A/B tests.
17

---

Published as a conference paper at ICLR 2024
DM
PDIS
MIS
MDR
SharpeRatio@k
(best@kk [(n))
denominator (std@i)
SharpeRatio
best@gk J(mg]
20
20-
# palicies deployed (k)
of policies deployed (K)
of palicies deployed (k)
Figure 7: Estimators' performance comparison with SharpeRatio@k @k k in Reacher. A higher value is better for SharpeRatio @k and the numerator, while lower value is better for the denominator.
SharpeRatio@k
numerator (best@k ((ms))
denominator (std@kk)
SharpeRatio
best@gk J[mm))
sta@k
15-
10
policies deployed
policies deployed (R)
policies deployed
Figure 8: Estimators' performance comparison with SharpeRatio @k in InvertedPendulum. A higher value is better for SharpeRatio @ k and the numerator, while a lower value is better for the denominator.
SharpeRatio@k
2.5
numerator (best@k- -Kn))
denominator (std@k))
SharpeRatio
1.5
05
10
# of policies deployed (k)
# of policies deployed (k)
of policies deployed (k)
Figure 9: Estimators' performance comparison with SharpeRatio@k Hopper. A higher value better SharpeRatio@1 numerator, while lower value better for denominator.
SharpeRatio@k
2.0
numerator (best@k j(ma))
denominator (std@kk)
SharpeRatio
1.5
1.0
0.5
of policies deployed (K)
of policies deployed (K)
of policies deployed (k)
Figure 10: Estimators' performance comparison with SharpeRatio@2 @ in Swimmer. A higher value is better for SharpeRatio@" @k and the numerator, while lower value better for the denominator.
18

---

Published as a conference paper at ICLR 2024
DM
PDIS
DR
MIS
MDR
SharpeRatio@k
2.5
numerator (best@k -((aa)
denominator (std@kk)
SharpeRatio
1.5
best@kk :[tm))
100
80
25
sta@k
60-
0.5
25
10
6
10
# policies deployed (k)
of policies deployed
of policies deployed (k)
Figure 11; Estimators' performance comparison with SharpeRatio@k in CartPole. A higher value is better for SharpeRatio @k @k and the numerator, while a lower value better for the denominator.
SharpeRatio@k
numerator (best@k (rrst@k-((n))) J(m))
denominator (std@k)
SharpeRatio
best@k Jimo)
15
30
20-
85
30-
10
10
# policies deployed (K)
of policies deployed (K)
of policies deployed (k)
Figure 12: Estimators' performance comparison with SharpeRatio@k in MountainCar. A higher value is better for SharpeRatio@ k and the numerator, while a lower value better for the denominator.
SharpeRatio@k
(best@k ((ma))
denominator (std@k))
SharpeRatio
bestgik J(mna))
80
60
std@@
60
20
10
# policies deployed (k)
of policies deployed (K)
of policies deployed (k)
Figure 13: Estimators' performance comparison with SharpeRatio@n k in Acrobot. higher value is better for SharpeRatio @k and the numerator, while lower value is better for the denominator.
19

---

Published as a conference paper at ICLR 2024
DM
PDIS
MIS
MDR
mean
std
mean polcyy volue
30
30
30-
-00
3
policies deployed
police coployed
policies deployed (M)
policies deployed
Figure 14: Statisties of the top-k policy portfolio of each estimator in Reacher. A lower value is better for std, while a higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of which is the risk-free baseline of SharpeRatio@k k.
best
worst
mean
std
palcy
23
of policies deployed 181
of polictes deployed 18]
of policies degloyed (k)
of policies deployed (K)
Figure 15: Statistics of the top-k policy portfolio of each estimator in InvertedPendulum. A lower value is better for std, while higher value is betler for best, worst, and mean policy performances. The dark red line shows the performance of Tg. which is the risk-free baseline of
SharpeRatio@kk.
best
worst
mean
std
polcy value
Sajed mnoa
atd of
of policies deployed (1)
of policies deployed (0)
policies deployed 00
of policies deployed (k)
Figure 16: Statistics the top-k policy portfolio of cach estimator in Hopper. lower value is better for std, while higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of n. which the risk-free baseline of SharpeRatio@k
best
beltoollyyvee volue
worze policy valus
580 of policy vol
23.0
27.
27.
25
of policles deployed (3)
policies decloyed
of policies deployed (1)
polcies deployed
Figure 17: Statistics of the top-k policy portfolio of each estimator in Swimmer. A lower value is better for std, while higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of Te which the risk-free baseline of SharpeRatio@k.
20

---

Published as a conference paper at ICLR 2024
DM
PDIS
DR
MIS
MDR
mean
mean polcy value
S000 police
sac policy
policies deployed
of policles depicyed
polictes daployed (k)
policies deployed
Figure 18: Statistics of the top-k policy portfolio of each estimator in CartPole. A lower value is better for std, while higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of Wr. which the risk-free baseline of SharpeRatio@k. k.
best
worst
mean
std
mean policy valuee
380
1300
100
100
2nd
200
of policies daployed [k]
policies deployed (k)
of policies deployed (0)
of polcies deployed (A)
Figure 19: Statistics of the top-k policy portfolio each estimator MountainCar. lower value is better for std, while higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of mb. which is the risk-free baseline of SharpeRatio
palicy vate
mean polccy value
360
-200
222 policy
-000
policies deployed
policies deoloyed
policies deployed
of policies deployed
Figure 20: Statistics of the top-k policy portfolio of each estimator in Acrobot. A lower value is better for std, while higher value is better for best, worst, and mean policy performances. The dark red line shows the performance of Ty. which the risk-free baseline of SharpeRatio@k.
21

---

Published as a conference paper at ICLR 2024
DM
PDIS
MIS
MDR
nMSE
RankCorr
nRegret:11
SharpeRatiog4
Sharpefatio@@
0.-
0.3
1.0
0.6
02
04
0.0
02
00
0.0
Figure 21: Estimators' performance comparison with conventional metrics, SharpeRatio@4 and SharpeRatio@* 7 in Reacher. A lower value is better for nMSE and nRegret I, while a higher value better for RankCorr and SharpeRatio. The star (x) indicates the best estimator(s).
nRagret@11
SharpeRatiogu
SharpeRatiog?
2.0
2.0
1.2
1.8
0.
0.5
0.-
6.4
0.4
033
04
0.0
0.7
05
0.5
02
0.1
00
0.0
0.0
0.0
Figure 22: Estimators' performance comparison with conventional metries, SharpeRatio@4, and SharpeRatio@ @7 in InvertedPendulum. A lower value is better for nMSE and nRegret@) 1, while higher value is better for RankCorr and SharpeRatio. The star (+) indicates the best estimator(s).
nMSE
RankCorr
nRegret@11
SharpeRatio@4
SharpeRatio@?
25-
0.3
0.5
1.
0.2
1.0
1.0
10
0.0
05
-03
Figure 23: Estimators' performance comparison with conventional metrics, SharpeRatio@44 and SharpeRatio in Hopper. A lower value is better for nMSE and nRegret@ while a higher value better for RankCorr and SharpeRatio. The star (*) indicates the best estimator(s).
nMSE
RankCor
nRegrett31
SharpeRatio@u
SharpeRttiogg
10:
85
0.3
0.0
0.0
Figure 24: Estimators' performance comparison with conventional metrics, SharpeRatio@4 and SharpeRatio@ 7 in Swimmer. A lower value is better for nMSE and nRegret@ 1, while a higher value better for RankCorr and SharpeRatio. The star (x) indicates the best estimator(s).
22

---

Published as a conference paper at ICLR 2024
DM
PDIS
DR
MIS
MDR
IMSE
RankCorr
nRegreo81 1
SharpeRatio@4
SherpeRetio@)
0.0
28
2.0
0.15
-0.2
0.5
0.05
-0.6
0.5
85
Figure 25: Estimators' performance comparison with conventional metrics, SharpeRatio@ @4 4, and SharpeRatio @7 in CartPole. A lower value is better for nMSE and nRegret I, while higher value better for RankCorr and SharpeRatio. The star (+) indicates the best estimator(s).
nMSE
RankCarr
nRegretg11
SharpeRatiog4
SharpeRatio@/
0.8
20
0.6
15
0.4
02-
05
0.2
0.1
0.
0.0
0.0
Figure 26: Estimators' performance comparison with conventional metrics, SharpeRatio@4, and SharpeRatio@ @7 in Mountain Car. A lower value is better for nMSE and nRegret@ 1, while higher value is better for RankCorr and SharpeRatio. The star (x) indicates the hest estimator(s).
RankCorr
nRegret@1
SharpeRatio@11
SharpeRatio@7
0.50
12.
1.
6.25
88
0.
0.
3050
0,4-
-075
62-
00
Figure 27: Estimators' performance comparison with conventional metrics, SharpeRatio@4 and SharpeRatio@y Acrobot. A lower value better for nMSE and nRegret@ while a higher value is better for RankCorr and SharpeRatio. The star (+) indicates the best estimator(s).
23

---

Published as a conference paper at ICLR 2024
Figure 28: On-policy policy value of the behavior policy (leftmost) and the candidate policies (others) Reacher
Figure 29: On-policy policy value of the behavior policy (leftmost) and the candidate policies (others) in InveriedPendullum
Figure 30: On-policy policy value of the behavior policy (lefimost) and the candidate policies (others) in Hopper
Figure 31: On-policy policy value of the behavior policy (lefimost) and the candidate policies (others) in Swimmer
Figure 32: On-policy policy value of the behavior policy (leftmost) and the candidate policies (others) in CarPPole
Figure 33: On-policy policy value of the behavior policy (leftmost) and the candidate policies (others) in MountainCar
Figure 34: On-policy policy value of the behavior policy (leftmost) and the candidate policies (others) Acrobot
24

---

Published as a conference paper at ICLR 2024
Table 3: Models and parameters used to train the behavior policies continuous control tasks
env.
model
parameter
value
Reacher
SAC (aarrooj el 2018
actor_entee eetor_encuder _enccder__acttrryy critic enwodeenneeeeeee factory"" [ffne_ffatiin actor_ cetlo_leamming_tiee
VectorEnncoeeraactttythiblbo _nnits=[1000)
VectorEnncodr/aattoy/bbddee (trprrrtiiiddeeuuiii11100000 units=[[000)
Mean(FunncinnFactoryy)
le-3
critic_ Jearneearnnnnneeeeeeeeeee rate
le-3
temp temp_leamnn Jearning rate batch size"
le-3
n_steps"
10000
socdatesstartsteee (randomization) class
1000
GaussianHeal
(randomization) sigma
0.7
InvertedPendulum
SAC (Haarnoja 2018
actor_ actor_encoder ___endddeerrrr ___eedder_fatorryy factory" Pntic_encoderr_cco trs__ecccde_fcttrrrre
VectorFncoddrFaccttryhhidd -nnils=(00n)))(0))))
VettofFncoderPeccrryhhhhue (_ns
g_func_faee factory" actor citor_carminn__.. "etaming_rate" rate" critic Jearningg raae" temp Jearning rate batch size" n_steps* update_staip step" (randomization) class
Mcan(FunnciioFFactory))
le-3
le-3
1c-3
32
10000
1000
GaussianHead
(randomization) sigma
2.0
Hopper
SAC (Haarnoja 2018
actor_ iitor_encodrr_com factory" critic ciceeeencderrrc ctccoder_fcctoryyy factory" _ffuneffactrr actor actor__camaat learning rate" critic Jearning rate" temp_ Jeariiennnngg..mm rate" batch size" stepss"
VectorEncoddrFFattrrybbbdde (100)=
VectorancodeFaaccoyyhhddd (otpuuunit(tOO))
Mean(Functioon//atttry
le-3
le-3
32
10000
gypdate_start_ste
1000
(randomization) class
GaussianHeaa
(randomization) sigma
1.5
Swimmer
SAC [aaarnoja el al. 2018
encuude_facctor aitii_eeccddrr__.. "rrroddrffactry"
VestorEnccweeriacttyyttie (nni
VectorEaccoderFactttyybbiii (
9 q_fune_factory"
MeanFFunction
actor_ @ilo__eerming_nar rate"
le-3
critic Jea_einannnnaat rate"
le-3
temp Jeampleanning..o rate" batch_size size" n_steps"
le-3
32
10000
update_ upplate_star_______ step" (randomization) class
1000
Geas
(randomization) sigma
1.0
Note: use implementations provided by d3rlpy (Seno Imai] 2021). * and ** indicate the parameters classes d3rlpy, while and Q indicate those our soltware. the other parameters that not specified the table, use the default values set by d3rlpy.
25

---

Published as a conference paper at ICLR 2024
Table 4: Models and parameters used to train the behavior policies in discrete control tasks
env.
model
parameter
value
CartPole
DDQN
Van Hasselt et al. 2016
encoder factorr
VectorEinoodrFaatooyyyddeee units=:(000)
q fune factory"
Mcun(FuuncionFFactory
target update_ interval" batch size" leaming rate TL_teps"
100
32
1c-3
8000
update start step"
1000
(replay buffer) maxlen" (explorer) class**
10000
Linear DeseaeDDeeyysssloGGeeee
(explorer) start_epsiin "arttepssilon (explorer) end end__eeddlee epsilon" (explurer) duration" (randomizaaionn) class (randomization) tau
1.0
0.01
8000
Seld
1.0
MountainCar
DDQN (Van Hasselt et al. 2016
encoder "acttrrr _.func_factory"
VectorEnnwdlerFaattoytcieei (unorerrffuttoyccccceeuuusss(o))
MeunQFunctiionFacoooyy actort)
target_ corect_ppatte___ "uce__update_nterral" interval"
100
batch_size"
32
leaming_rate
1e-3
n_steps"
50000
sandatesstart_see
1000
(replay buffer) maxlen"
10000
(explorer) class**
LineaDDaaayppssooooGnned
start_epsilon (explorer) start, Lepsiloo"
1.0
(explorer) end_epsill ood_epsilon" (explorer) duration" (randomization) class 96
0.01
20000
SoftmaxHead
(randomization) tau
3.0
Acrobot
DDQN Van Hasselt et al. 2016
encoder factory" q_fune_ factory" target_updateiingg innterval target_updatc_ interval'
VestorFncodoFFaccoryyhhdeee units=((0int))
MeanoFFunctionFacctryy)
100
100
hatch_size"
32
learning rate"
5c-5
n s_stpss"
20000
update_ start, step" (replay buffer) maxten" (explorer) class" (explorcr) start, cpsilln" (explorer) end "osseeo" (explorer) duration" (randomization) class (randomization) tau
1000
10000
LinearDeaayEpsilorGGnee
1.0
0.1
10000
SoftmaxHead
5.0
Note: We use the implementations provided by d3ripy (Seno Imai] [02]] * and indicate the parameters and classes of d3ripy, while & and O indicate those of software. For the other parameters that not specified in the table, use the default values set by d3rlpy.
26

---

Published as a conference paper at ICLR 2024
Table 5: Models and parameters used to train the candidate policies in Reacher
model
parameter
value
CQL Kumar al. 2020
actor encoder factory"
VectoEEncodcraaccoyyyhddde (nnr
aectorEnncoofFFaattttyhddda unnits=[30, 30))
VecoorEncoderr' factory sacieenn_niitt000.1 10))
eactorr
aestoodncodcrFaattyyyyhoo units=[30. 30)
VectorEncodde''ace (eroorrrddenuuniissstesseeeeeeee 10)
g_func _afuuc_factor" (randomization) class (randomization) sigma
MeantFuunctionFacttoyy GaussianHead
(0.3, 0.5. 0.7. 0.9)
IQL Kootrikov 2022
encoder_factonn
VectorEnnoodrFFaatryyhbbeee (ttprr/dddideeeuiisssiieee0 (100)
Vector inncudeeFactryyhhidleee sutiless3_nttt 30D
10))
critic_ catie_encoee_ffattrry
Vector Factory uniis=[330, 30)
VectorEncoder Factolerractttyhhhddeeee (uttinss__iiiiO0,. 10))
(randomizatinn) class 9
GaussianIlead (0.3, 0.5, 0.7, 0.9)
(randomization) sigma
Note: We use the implementations provided by d3rlpy (Seno Imai 2021 and classes of d3rlpy, while O and 2 indicate those specified in the table, we use the default values set by d3rlpy. Note that since we use three different encoders and four different values of sigma, we obtain 12 different policies for both CQL and IQL.
and indicate the parameters soltware. For the other parameters that not
Table 6: Models and parameters used to train the candidate policies in InvertedPendulum
model
parameter
value
CQL Kumar al. 2020
actor actor_encoder, factory
VettorEncoderfucctoyybddee
(ttprr/ccctrcccdddeeuuuustttu)) uniis=(100))
SectorEncooorfFfccoooyhhddo _a___uniitt..... 30))
Encooerrn aecloryhhddden_mm Satuuddennniiss(50,,1100 10))
aertic_encder__faee
Vector Factoudraaattrryhhbdeenn
(ectury(hiden_uuiiss([0]]] (otpuunnt==(10))))
units=|30. 30))
EncolerFacttoorg (hiddenndeeue [50,snttiisS, (hidden_unis==500, 10)
g_ffne_factory (randomization) class (randomization) sigma
MeennnFuctiionFaactoo
GaussianHeea
[1.75. 2.0. 2.25. 2.5]
Kostrikuv 2022
Billo_eencode__aaee factory"
VectorFncoderf EnncodeFFactoryhhidde (hidddddunnniniis333.... 30)
VectorFncnderFF hatpolerrFaatthhhhhheennnn
10)
latiic_encuder_ffaee factory
VectorEncudeed (acturryhhileeenuiiiii
EncodeeFactooryyytoeoee units=[30. 30)
FastorrFnneeFaattryyhhoo units=[50, 10)
(randomization) class
GaussianHeal [1.75, 2.0, 2.25, 2.5)
(randomization) sigma
Note: We use the implementations provided by d3rlpy (Seno Imai 2021 and indicate the parameters and classes of d3ripy, while & and O indicate those of our sottware. For the other parameters that not specified in the table, we use the default values set by d3rlpy. Note that since we use three different encoders and four different values of sigma, we obtain 12 different policies for both CQL and IQI..
27

---

Published as a conference paper at ICLR 2024
Table 7: Models and parameters used to train the candidate policies in Hopper
model
parameter
value
CQL
Kumar al. 2020
actor encoder factory"
VectorEncooerFacctryyhhhee unites=
aarorEEnncorraactttyyhdddee units=[330 30))
VecoorEncoderr' factory units=[50, 10))
vctorrchidden_uuiiis550000
eactorr
CecoooEnnoocFaaattyyylloo units=[30. 30)
VectorEncoddr''acee (hiiddy
(n0 (hidden__uiis=(550
10)
g_func _afuuc_factor"" (randomization) class (randomization) sigma
McanFFunncioonaattooyy GaussianHead
(0.5. 1.0. 2.03
IQL Kootrikov 2022
encoder_factenn
VectorEnnoodrFFaatttyhhbeee http:ttorrrribbdeeeiiin--00000 -ntpuuunntt==0i
Vector EncuderFactorryhcdllee __ntim=3nniini 30D
10))
critic_ bitie_encoee_ffattrry
Vector Factory units=[330 30)
VectorEncoder Factolerraattttyhhccdleee u50 10))
(randomizatinn) class 9
GaussianHlead
(randomization) sigma
(0.5, 1.0, 2.0)
Note: We use the implementations provided by d3rlpy (Seno Imai 2021
and indicate the parameters
and classes of d3rlpy, while O and 2 indicate those specified in the table, we use the default values set by d3rlpy. Note that since we use three different encoders and four different values of sigma, we obtain 12 different policies for both CQL and IQL.
soltware. For the other parameters that not
Table 8: Models and parameters used to train the candidate policies in Swimmer
model
parameter
value
CQL
Kumar al. 2020
actor actor_encoder. factory
VectorEnnouer/lucttyybddee (ttrrrrdddddeeuiiss11100000 (100)=
aestoodinnooccFaaaoooyhhdde units=|330 30))
EncoderFaccioo (ecooryhiidee___n Soanldlennniiss(50,,0000 10))
criticeencdder factory"
VectorEncoder/ Fancoodeeaatttrrhhhheeeen (otrr//hiideu_uuuss(00000]1 (mmis==(000)
www.e/eoooddraattyyyiiiee units=|30. 30D
EncoderFactooor (hiddenn (actorrhhodeeeeeeennno..
(50 (hidden_unis==50, 10)
g_ffne_factory (randomization) class (randomization) sigma
GaussianHHead
(1.0. 1.5. 2.0. 2.5)
Kostrikuv 2022
Billo_eencodr__aaee factory"
VectorFncoderr EhccodeFFactrr(hhddee (hirrr/hhhddeeeeiis3e3..,. ____unniit30,. 30)
VectorEncnden aactaarraaatthhhhneennuuu
10)
latiic_encuder_ffaee factory
Ewwoooonooooaooryyttte units=[30. 30)
FantoorFnnneaaaattyyyhhho units=[50, 10)
(randomization) class
GaussianHead [1.0, 1.5, 2.0, 2.5)
(randomization) sigma
Note: We use the implementations provided by d3rlpy (Seno Imai 2021 and indicate the parameters and classes of d3ripy, while & and O indicate those of our sottware. For the other parameters that not specified in the table, we use the default values set by d3rlpy. Note that since we use three different encoders and four different values of sigma, we obtain 12 different policies for both CQL and IQI..
28

---

Published as a conference paper at ICLR 2024
Table 9: Models and parameters used train the candidate policies in CartPole
model
parameter
value
CQL Kumar al. 2020
encoder factory"
VectoräincodrFaatteyyllddde units=[[100]
aentornnnnceeeaauuuutttuutuuu (trilden_uiiss[3300 30)
VectorEncoder/ (hidden (50 10))
f_func f_fuc_ffcttory" targeL largeL_pdaate intervall* batch_siee
MeaunFFunctionFacttuyy
32
leurning_rat rute" (randomization) class (randomization)) epsilon
6.25e-5 FpsilonCrredyyHadd
(0.1, 0.3, 0.5, 0.71
BCQ (Fujimoto al. 2019
encoder factory
VettorFaccodeFFactotyyhdddeee units=(1000)
VectorFncnneeFaattryyiildi (tiiddeeenniiii(03..1.1100 30))
WecorrEnoweerFhcuuryyttllee Stn 10))
f__fun__faae factory* target update intervall* batch_size
McanFFuunctionaaatttry
1000
learning rate" action action_flexibility beta (randomization) class
lc-3
0.3
0.1
EpsilonGreedyIeel
(randomization) epsilon
{0.1. 0.3, 0.5, 0.7]
Note: We use the implementations provided by d3rrpy (Seno & Imai] 2021). * and ** indicate the parameters and classes d3rlpy, while O and &6 indicate those of our soltware. For the other parameters that are not specified the table, we use default values set by d3rlpy. Note that since we use three different encoders and four different values of epsilon, we obtain different policies both CQL and BCQ.
Table 10: Models and parameters used to train the candidate policies in MountainCar
model
parameter
value
CQL Kumar et al. 2020
encoder factory'
VectorEnccodeFasttoyyhhdde units=(100))
Vectorfnncoder//ctttrryyt (hiddee units=[[30, 30))
VectorEncoodrrFattttylhdddd units=[50. 10))
fune f_func_factory" pngget_uddat__itervye intervall batch size" learning rate" alpha* (randomization) 0 (randomization) epsilon o
Mcun(FunnctioFFacttoryy)
100
32
1c-3
EpsilonGrredyHHeu
(0.0. 0.1. 0.2, 0.3)
BCQ (Fujimoto et al. 2019
encoder ffatorr_
VectorEnncoderraattryy (hinddeerrydddnnnnn. units=(100)
VectorEnnooder/atttory (hidden units=[300 30))
VectorEnnoddcrFactoryyti units=[500. 10))
MeuntFFuncioonaacttury))
target_ ungeeLuppate_jjterra' interval* batch saaich_size" learning_rate rate" action flxxbbiliyy beta (randomization) 0
100
32
le-3
0.01
EpsilonGreedyHeall
(randornization) epsilon &
(0.0, 0.1, 0.2, 0.31
Note: We use the implementations provided by d3ripy (Seno & Iaai 2021) and ** indicate the parameters and classes of d3rlpy, while & and & indicate those of our sottware. Forr the other parameters that are not specified in the table. we use the default values set by d3rlpy. Note that since we use three different encoders and four different values of epsilon, we obtain 12 different policies for hoth CQL and BCQ.
29

---

Published as a conference paper at ICLR 2024
Table 11: Models and parameters used to train the candidate policies in Acrobot
model
parameter
value
CQL Kumar el al. 2020
encoder_ encoder_factory"
VeetorrinncdeeePPatttythdddee aar units=[100))
CactorrEncoder/atttrryttye (hiddenn,rnnnnnnnn soteennnnnttt000 30))
FectorFaeedderactoo (hidreenhhndnnnn units=[50, 10)
q_fune_factor larget largeLupdate upddate_iintervaa inteevaal* hatch size"
100
32
learning_rai rate"
6.25e-5
(randomization) class (randornization) epsilon
EpsilonGreedyyTeeu
(0.2, 0.3, 0.5.
BCQ (Fujimoto et al. 2019
encoder encoder_factory"
FectooFincoderraattttyti (hidden (hddiuuunnddddennss((s((0))))
CooorEnnodddFacttryyldddee units=[30, 30))
VicoofEnooddFFaccttyyiiidden units=[50, 10))
[_func_factor factory* target update interval" batch_ size" learning rate* action Rexibility beta (randomization) c (randomization) epsilon
MrantFFnnccionaccttry)
1000
32
5e-5
0.1
0.01
ErsilonGrreddycadd
[0.05, 0.10, 0.15, 0.20]
Note: use implementations provided by d3rlpy (Seno Imai 2021). and classes d33ipy, while & and ( indicate those specified in the table, we the default values set d3rlpy. Note that since three different encoders and four different values of epsilon, we obtain 12 different policies for both CQL and BCQ.
indicate the parameters software. For the other parameters that not
Table 12: Models and parameters used in OPE
model etc.
parameter
value
FQB
[e
2019
Serouderfacttryy [_fuuneffctoryy learning_rate rate"
VesturEnnealeratttoryhhddeee (1d0
(nni))=(100))
le-3
DICE
Yang al. 2020
(all model parameters) & sigma
(default values)
0.5
continuous OPE
sigma
0.5
Note: We use the FQE implementations provided by d3rlpy (Seno & Imail 2021 and DICE implementation
available our software. indicate parameters d3rlpp. while < indicate those our software. sigma is the bandwidth hyperparameter of kernel function used DICE and continuous OPE, Note that, for both FQE and DICE, disable state scaler Reacher enable the other enviroments to stabilize the training process.
30


Reviews:
[{"review_id": "4xa41YpNTR", "reviewer": "Reviewer_wVqE", "summary": "This paper is concerned with defining an appropriate measure for off-policy evaluation. The authors argue that commonly used metrics such as MSE and rank correlation are insufficient because they fail to properly account for risk/return tradeoffs. To remedy this it’s proposed that ideas from portfolio optimization be used. In particular, the authors propose the use of the Sharpe ratio to measure efficiency. A number of empirical results are provided which demonstrate the properties of the Sharpe ratio @ k metric, and compare it to other metrics such as rank correlation and regret.", "questions": "It would be useful if the authors could give a description of the settings in which one would prefer Sharpe@k generally. Also, would be useful if the authors could characterize the necessary assumptions on the reward distributions.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper tackles an important problem–off policy evaluation is a critical aspect of deployment of RL systems in many real world contexts. The authors' proposal to use ideas from portfolio optimization is an interesting one, and the proposal to use Sharpe @ k is both intuitive and simple. The authors do a nice job of evaluating their work empirically and demonstrating the properties of the proposal.", "weaknesses": "My biggest issue with this paper is that the work is very limited in scope which limits the benefit to the larger community. While the proposal to use ideas from portfolio theory is interesting, the authors focus on a fairly simple definition and don't describe the properties and behavior of the proposed approach theoretically. It would be useful if the authors described the proposal in slightly more generality. For example, are there rank-based analogs of the current approach? It would also be useful if there was a full discussion of the necessary assumptions/conditions for Sharpe@k to be applicable in a real world setting. It would also be useful if the authors highlighted cases where current metrics could be preferable. In general, it would seem that the appropriateness of a given evaluation metric is entirely task dependent, a discussion of this could be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "anNIgx81Xa", "reviewer": "Reviewer_S1vJ", "summary": "This paper studies the risk-return tradeoff between different OPE evaluations. Existing methods evaluate the superiority of an OPE estimator via various “accuracy” measures. However, the paper argues that merely looking at the accuracy may not be sufficient as two OPEs with similar accuracies may have different risk implications in practice. \n\nTo fix this issue, the paper proposes to use concepts from portfolio evaluation in finance and develops a new metric called SharpeRatio@k. This metric helps distinguish between conservative and high-stakes OPE estimators. The key idea behind this is to regard the set of top-k candidate policies selected by an OPE estimator as its “policy portfolio”. The paper constructs a policy portfolio that is “efficient”, i,e, it contains policies that improve the performance of behavior policy without including poorly performing policies. Finally, the paper evaluates typical OPE estimators using the proposed metric using a number of continuous control benchmarks.", "questions": "– In description of Fig 2, I found the sentence “X underestimates the performance… Y overestimates ” confusing – I was wondering if it should be the other way around. If the current sentence has to be true, then my understanding of the interpretation is that black policies suffer an underestiation (i.e. even though x-axis value is high, y-axis value is lower than ground truth) and so the top-k ends up picking next best policies. Is this correct or am I missing something? \n\n– top-k policies are considered in several resource allocation setups where there are limited resources and some index is computed so that resource can be allocated to the top k indexes \n(For e.g. [1] “Restless Bandits: Activity Allocation in Changing World”, P Whittle; [2] https://arxiv.org/abs/2110.02128). Can the method be using for designing robust top-k selection/resource allocation policies in this context? I also wonder how it compares to existing robustness in bandits work such as [3] https://arxiv.org/abs/2107.01689 and whether those ideas can be applied to the OPE evaluation setting discussed in the current paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "– I think the main strength of the paper lies in identifying how the portfolio evaluation concepts in finance can be applied to evaluation of OPEs. I’m not an expert in finance, and can’t speak to the novelty of the idea, but assuming it is novel, this certainly sounds interesting to me.\n\n– The paper is well-organized, is a pleasure to read and explains difficult concepts well enough. \n\n– I also really liked the use of toy examples throughout the paper to drive home the key concepts. \n\n– Experiments section is thorough; it compares several of commonly used OPE estimators and also identifies several directions for future OPE research.", "weaknesses": "– I felt the figures and toy examples could use more explanation. For example, it wasn’t clear in Fig 2 what the red dots are, black dots are and how to interpret the axes. \n\n– Related work: In the section on “Risk Measures and risk-Return Tradeoff in Statistics and Finance”, the paper discusses the Sharpe,1998 paper in length (relevant for the Sharpe ratio used in current paper). However, there are only two other related works identified in this entire section. I’m not an expert in this domain but I suspect there has to be more prior research done in this space – and if that is true, the existing related work section seems a little thin. \n\n– Other minor suggestions: \n\n1.  In the abstract, the sentence “We first demonstrate, …” is a bit too long and unwieldy and hard to understand easily. \n\n2.  Contributions paragraph talks about “top-k policy deployment” without any prior context on what that means. Difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOe63WnaYg", "reviewer": "Reviewer_qCGC", "summary": "The paper proposes a new Off-line policy evaluation technique that account for the risk-return tradeoff as usually done in the financial literature when evaluating a portfolio. \nNamely the papers proposes a new evaluation metric (SharpeRatio@k) that measure the risk/return tradeoff of OPEs.\nThe paper is complemented by empirical analysis to evaluate the effectiveness of the method.", "questions": "1) Way is it important to consider the std of the estimators in assessing OPEs? In finance, big swings might prompt the activation of risk measures, and thus one usually prefers loosing some performance points in favour of more stable results. Way is this important in OPE?\n2) Is there any provable theoretical advantages of using your method compared to others?\n3) Seems like $\\hat J(\\pi; D)$ was never defined explicitly.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper studies the important problem of offline policy evaluation and propose a risk-aware method for selecting OPE estimators. The idea of incorporating the variance of the estimators seems novel and worth investigating. Moreover the paper is fairly well structured and clearly written.", "weaknesses": "My biggest concern is about the technical contribution of the work. While the idea seems novel it also sound quite natural and simple and bears concern about the actual interest that would spark in the community. Moreover the introduction of Sharpe ratio like measure in OPE it seems poorly motivated by the authors who should try to sell better the motivations for the idea", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owEI3V3KHf", "reviewer": "Reviewer_n49S", "summary": "This paper proposed a new metric for off-policy evaluation, names SharpeRatio@K. This metric measures the risk-return tradeoff and efficiency of policy portfolios formed by an OPE estimator under varying online evaluation budgets (i.e. top-K policies selected by the estimator). Via examples, the paper demonstrates that existing metrics (MSE, regret@1, rank correlation) fail to differentiate policies with different risk-return tradeoffs while the proposed new metric does. The authors evaluated SharpeRatio@K via benchmark experiments using various existing OPE estimators regarding their risk-return tradeoff. The authors also developed open-source software for using the proposed metric.", "questions": "- The classic sharpe ratio uses the mean return of the portfolio as the numerator instead of the best policy's return. Is there a particular reason that the best return is used in the proposed metric?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall the paper has a good clarity and is well-organized. I found the examples on over/under-estimation, conservative / high-stakes estimation to be helpful for understanding the benefit of the proposed metric.\n- Given that in practice, OPE is more often used as a screening process for selecting top-K policies to deploy in A/B tests, the risk-to-return ratio can be a useful and meaningful metric for comparing OPE estimators. The proposed metric based on Sharpe Ratio to be natural and easy-to-evaluate.\n- The benchmark experiments / open-source software provide good evidence that SharpeRatio@K is capable of measuring the risk-to-return efficiency of various OPE estimators, and facilitate the usage of such metric in future research.", "weaknesses": "The benchmark results show that SharpeRatio@k can sometime diverge significantly from the conventional metrics in terms of estimator selection. Additional discussion should be added on how practitioners may consolidate the insights given by the different metrics evaluated under certain scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)