OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
Authors: Allen Nie
Yash Chandak
Christina J. Yuan
Anirudhan Badrinath
Yannis Flet-Berliac
Emma Brunskill
Venue: NeurIPS 2024 poster (2024)
Keywords: offline reinforcement learning
off policy evaluation
statistics
ensemble method
Primary area: reinforcement_learning
TL;DR: We propose an estimator-agnostic meta-algorithm to combine multiple OPE estimates that achieves lower MSE in Offline RL tasks.

Abstract: Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

OPERA: Automatic Offlin Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
Allen Nie!
Yash Chandak' Yannis Flet-Berliac Emma Brunskill**
Christina Yuan?
Anirudhan Badrinath
Computer Science, Stanford University
CComutter Science, University of Texas, Austin *Cohere
Abstract
Offline policy evaluation (OPE) allows us to evaluate and estimate sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating new policy online without confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, espe- cially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfor- tunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, to alternative approaches, our estimator can be used to select higher-performing policies healthcare and roboties. Our work contributes improving ease use for general-purposs. estimator-agnostic off- off-policy evaluation framework for offline RL.
demonstrate that when compared
1 Introduction
Offline reinforcement learning (RI) involves learning better sequential decision policies from logged historical data, such as learning personalized policy for math education software (Mandel et al., 2014; Ruan et al., 2024), providing treatment recommendations in the ICU (Komorowski et al., 2018; Luo et al., 2024) or learning
controllers for robotics(Kumar et al., 2020; Yu et al., 2020). Offline policy evaluation (OPE), in which the performance J ( of a new evaluation policy nu is estimated given historical data, is common subroutine in offline RL for policy selection, and be particularly important when deciding whether to deploy a new decision policy that might be unsafe costly. Offline policy evaluation methods estimate the performance of an evaluation policy me given data collected by behavior policy ma There are many existing OPE algorithms, including those that create importance sampling-based estimators (IS) (Precup, 2000), value-based estimators (FQE) (I.e et al., 2019), model-based estimators (Paduraru, 2013: 13 Liu et al., 2018b; Dong et al., 2023), doubly robust estimators Jiang and Li (2016); Thomas and Brunskill (2016), and minimax-style estimators (Liu et al., 201 8a; Nachum et al., 2019; Yang et al., 2020).
This raises an important practical question: given set of different OPE methods, each producing particular value estimate for an evaluation policy, what value estimate should be returned? A simple approach is to avoid the problem and pick only one OPE algorithm look at the direction of a set of OPE algorithms' scores as a coarse agreement measure. Voloshin et al. (2021) offered heuristics based
Emails: (anie.ychanddak.cbun) htw.:cccunnnnt.tttfooaalaa.l.aaa Dts.stanfor..cdu. "Wrk done while at Stanford.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

on high-level domain structure (e.g., horizon length, stochasticity, or partial observability), but this does not account for spseifc- information related the offline dataset adn policies.
In this paper we seek to aggregate the results of a set of multiple off- policy RL estimators to produce a new estimand with low mean square error. This work is related to several streams of prior work: (1) multi-armed bandit and RL algorithms that combine two estimands to yield a more accurate estimate; (2) multi- bandit and RL algorithms that select a single estimand out of set of estimands, and (3) stacked generalization / meta meta-learning / super learning methods in machine learning.
Research in (1) builds on doubly robust (DR) estimation in statistics to produce an estimand that combines important sampling and model-based methods (Jiang and Li, 2016; Gottesman et al., 2019; Farajtabar al., 2018). Similarly, accounting for multiple steps, the MAGIC estimator blends between IS-based and value- based estimators within a trajectory (Thomas and Brunskill, 2016). The second line of work (2) does not combine scores but instead introduces an automatic estimator selection subroutine in the algorithm. However, such methods typically assume strong structural requirements on the input estimators. For example, Su al. (2020); Tucker and Lee (2021) assume as input a nested set of OPE estimators, where the bias is known to strictly decrease across the set. Zhang and Jiang (2021) leveraged set of Q-functions trained with fitted Q-evaluation (FQE) and cross-compare them in a tournament style until one Q-function emerged. None of these methods allow mix-and-match of different kinds of OPE estimators.
Our work is closest to a third line of more distant work, that of stacked generalization (Wolpert, 1992) / meta-learning and super learning across ensembles. There is long history in statistics and supervised learning of combining multiple input classification or regression functions to produce a better meta- function. Perhaps surprizingly, there is little exploration of this idea to our knowledge in the context of RL or multi arded bandits. The one exception we are aware of was for heterogeneous treatment effect estimation a 2-action contextual bandit problem, where Nie and Wager (2021) utilized linear stacking to build consensus treatment effect estimate using two input estimatands.
In this paper we introduce the meta- alga--aloor OPERA (Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators). Inspired by linear weighted stack, OPERA combines multiple generic OPE estimates for RI. in an ensemble to produce an aggregate estimate. Unlike in supervised learning where ground truth labels are available, in our setting a key choice is how to estimate the mean squared error of the resulting weighted ensemble. Under certain conditions, bootstrapping (Efron, 1992) can approximate finite sample bias and variance. We use bootstrapping to compute estimates of the mean squared error of different weightings of the underlying input estimators, which can then be optimized as a constrained convex problem. OPERA can be used with any input OPE estimands. We prove under mild conditions that OPERA produces an estimate that is consistent, and will be at least as accurate as any input estimand. We show several common benchmark tasks that OPERA achieves more accurate offline policy evaluation than prior approaches, and we also provide a more detailed analysis of the accuracy of OPERA as a function of choices made for the meta-algorithm.
Related Work
Offline policy evaluation Most commonly used offline policy estimators can be divided into a few categories depending on the algorithm. An important family of estimators focuses on using importance sampling (IS) and weighted importance sampling (WIS) to reweigh the reward from the behavior policy (Precup. 2000). These estimators are known to produce an unbiased estimate but have a high variance when the dataset size is small. For a fully observed Markov Decision Process (MDP), model-free estimator, such fitted Q evaluation (FQE), is proposed by Le et al. (2019), and one can also learn a model given the data to produce a model-based (MB) estimate (Päduraru, 2007: Fu et al., 2021; Gao et al., 2023). When the behavior policy's 's probability distribution over action is unknown, a minimax style optimization estimator (DualDICE) can jointly estimate the distribution ratio and the policy performance (Nachum et al., 2019). For a partial observable MDP (POMDP), many of these methods have been extended to account for unobserved confounding, such as minimax style estimation (Shi et al., 2022), value-basee estimation (Tennenholtz et al., 2020; Nair and Jiang, 2021), uses sensitivity analysis to bound policy value (Kallus and Zhou, 2020; Namkoong et al., 2020; Zhang and Bareinboim, 2021), learns useful representation over latent space (Chang et al., 2022).
OPE with multiple estimators Choosing the right estimators has become an issue when there are many proposals even under the same task setup and assumptions. Voloshin et al. (2021) proposed

---

an empirical guide on estimator selection. One line of work tries to combine multiple estimators to produce a better estimate by leveraging the strengths of the underlying estimators, for example. (weighted) doubly robust (DR) method (Jiang and Li, 2016). For contextual bandit, Wang et al. (2017) proposed switch estimator that interpolates between DM and DR estimates with an explicitly set hyperparameter. For sequential problems, MAGIC blends a model-based estimator and guided importance sampling estimator to produce a single score (Thomas and Brunskill, 2016). Another line of work tackles the many-estimator problem by reformulating multiple estimators one estimator. Yang et al. (2020) reformulated a set of minimax estimators as a single estimator with different hyperparameter configurations. Yuan et al. (2021) constructed spectrum of estimators where the endpoints are an IS estimator and a minimax estimator and proposed a hyperparameter to control the new estimator. This line of approaches does not leverage multiple estimators solve the OPE selection problem because they recast the OPE selection problem as a hyperparameter selection problem. The last line of work provides automatic selection algorithm that chooses estimator from many, relying on an ordering of estimators (Tucker and Lee, 2021) or being able to compare the output (such Q-values) directly Zhang and Jiang (2021).
Bootstrapping for model selection Using bootstrap to estimate the mean-squared error for model selection was initially proposed by Hall (1990), for the application of kernel density estimation. The idea was subsequently used by others for density estimation (Delaigle and Gijbels, 2004), selecting sample fractions for tail index estimation (Daniclsson et al., 2001), time-series forecasting (dos Santos and Franco, 2019) and other econometric applications (Marchetti et al., 2012). Similar ideas have been explored by Thomas et al. (2015) to construct confidence interval for the estimator. We extend this idea to use bootstrapping to combine multiple OPE estimators to produce a single score.
3 Notation and Problem Setup
We define stochastic Decision Process M = AS, 7)), where S is set of states; A is set
of actions; T is the transition dynamics; 7 is the reward function; and y E (0, 1) is the discount factor. Let D,, {T. Y2= (s,, ai, 84, r.) be the trajectories sampled from on M. We denote the true performance of a policy π as its expected discounted return J(x) = Ey-p, [G(T)] where
G(r) = =rr)= 7'1, and Pr is the distribution of under policy л. In off-policy policy evaluation problem, we take a dataset Dn, which can be collected by one or a group of policies which we refer as the behavior policy ттh on the decision process M. An OPE estimator takes in a policy Tee and dataset D, and returns an estimate of its performance, where we mark it as V II x D - R. We focus on estimating the performance of a single policy x. We define the true performance of the policy V = [[T), and multiple OPE estimates of its performance as V.2T (Da)= = Vi Dx) for the i-th OPE's estimate.
4 OPERA
In this section, we consider combining results from multiple estimators (V,T), )/= to obtain better estimate for V Towards this goal, given {(/T)/ we propose estimating set of weights of e R such that - = Ex= a V E R has the lowest mean squared error (MSE) towards estimating V*. Formally, let 1 E k**1 be vector whose elements correspond to values from different estimators, and let E xx1 correspond to a vector where each element is the same and corresponds to V= Let E R be vector with values of all a 's and let a E R*x1 be estimate of a", For any estimator 17", the mean-squared error is denoted by,
(Vm) E
MSE ( = ED..
where we make V.2T explicitly depend on D, to indicate that the expectation is over the random variables V." which depend on the sampled data D,. With this formulation, estimating a* can be elicited as a solution to the following constrained optimization problem.
Remark 1. Let E 04 = 1, then
a² E araggii min a Aa where A TF (v (D-v)( (1- acRtx1
-v)7 R
(2)
3

---

Using the fact that E/=1 aq 1.
V vm) | Σ as
27
MV) = EE
Σ a V
E
Vi-
V
(3)
Now re- re-writing the equation above equation 3 in vector form,
MSE(V") E (v- - (D-v)
(D-
(4)
Finally, simplifying equation 4 further
MSE(V=) a: "E ( a= a Aa.
(5)
Therefore, a that minimizes MSE(VT) (V*) is equivalent to that minimizes a Aa.
is worth highlighting that the optimization problem in Remark 1 is convex in with linear constraint and thus can be solved by any off-the-shelf solvers (Diamond and Boyd, 2016).
An advantage of Remark is that it provides the objective for estimating a". Unfortunately, this objective depends on A, and thus on V*, which is not available. Further, observe that A can be decomposed
Estimating A:
A= E (E-
{ -E|D- 1 v $17
(6)
where the first term corresponds to co-variance between the estimators {V,- and the second term corresponds to the outer product between their biases. One potential approach for approximating A could be to ignore biases. While this could resolve the issue of not requiring access to V, ignoring bias can result in severe underestimation of A, especially in finite-sample settings or when function approximation is used, Further, even if we ignore the biases, it is not immediate how to compute the covariance of various OPE estimators, e.g., FQE.
We propose overcoming these challenges by constructing À R estimate of A R using a statistical bootstrapping procedure (Efrrn and Tibshirani, 1994). Subsequently, we will use the A as plug-in replacement for A to search for the values of as discussed in Remark 1. There is a rich literature on using bootstrap to estimate bias (Efron, 1990; Efron and Tibshirani, 1994; Hong, 1999; Shi, 2012; Mikusheva, 2013) and variance (Chen, 2017b; 7b; Gamero et al., 1998; Shao, 1990; Ghosh et al., 1984; Li and Maddala, 1999) of an estimator that can be leveraged to estimate the terms in equation 6. Instead of estimating the bias and variance individually, we directly use the bootstrap MSE estimate (Chen, 20 Williams, 2010; Cao, 1993; Hall, 1990) to approximate A.
For bootstrap estimation to work, two key challenges need to be resolved. Even if provided with V*, the regular bootstrap is not guaranteed to yield an MSE estimate which is asymptotic the true MSE if the distribution has heavy tails (Ghosh et al.. 1984). Furthermore, V is unknown in the first place. To address these challenges we follow the work by Hall (1990), where the first issue is resolved by using sub-sampling based bootstrap resamples of size nı n. where mn is of a smaller order than n. Therefore, we draw data D;, (Ti....,T"), T from D,, {TI.... Tn with replacement. To resolve the second issue, we leverage the MSE estimate by Hall (1990), and approximate equation 1 using
MSE(V?) = ED;, [VTT(D..) - -p7))
Dn E
(7)
Building upon this direction, we propose using the following estimator A for A, (VV))
| [..].e R
A = A=
(8)
and we substitute à for A in equation 2 to obtain the weights for combining estimates (V,3} Jimi
S Σ
á à E where,
à E arg a Aa E BAx1
(9)
acRex1

---

There are two key advantages of the proposed procedure: (1) Bias: it does not require access to the ground truth performance estimates 6*, In the supervised learning setting, held-out/validation set can provide a way to infer approximation error, However, for the OPE setting there is no such held-out dataset that can be used to obtain reliable estimates of the ground truth performance. (2) Variance: Depending on the choice of the estimator (e.g.. FQE), it might not be possible to have a closed-form estimate the variance, especially when using rich function approximators. Using statistical bootstrapping, OPERA mitigates both these issues and thus is particularly suitable for off-policy evaluation.
Estimating a": We consider how emror in estimating the optimal weight coefficient a* affects the MSE of the resulting estimator PTT Without loss of generality, we consider |!:"| I 1, since can trivially normalize each estimator's output by | We now prove that under the mild assumption that the error in the estimated á can be bounded as some function of the dataset size, that can bound the mean squared error of the resulting value estimate:
Theorem 1 (Finite Sample Analysis). Assume given n samples in dataset D, and let A = Ep..
there exists a À such that
Vi, ED., 1á;; a,1l S nn
(10)
1.2
MS v n2 + +
(11)
The error of OPERA is divided into two terms. First note that A. is the approximation error: the difference between the true estimate of the policy performance V and the best estimand OPERA can yield when using the optimal (unknown) a If V can be expressed as a linear combination of the input OPE estimands 6, then there is zero approximation error and A, = 0. The second term in the bound comes from the estimation error due to estimating a=" this arises from the bootstrapping process used for estimating A in equation 8. For this second term we compute an upper bound using a Cauchy-Schward inequality. This term decreases as the dataset size n increases. The resulting error depends on the rate at which the estimated & converges to the true as function of the dataset size. For example, if A = 0.5, (a n -5 rate), the MSE will converge at a n rate in the first term, and if À 0.25 (a 225 rate) the MSE will converge at -0.5 rate in the first term. We provide the full proof in Appendix A.4.
We show full practical implementation of OPERA in Algorithm 1, where we demonstrate how to efficiently construct Å and compute á.
4.1 Properties of OPERA
For Á obtained from the bootstrap procedure in equation 8 to be asymptotically accurate estimate of A, (a) consistent estimator of V is required, and (b) the estimators 1' need to be smooth. We discuss these points in more detail in Appendix A.3. In the following, we theoretically establish the properties of OPERA on performance improvement and consistency. We also demonstrate how OPERA allows us to interpret each estimator's 's quality. Further, in Section 6, we empirically study the effectiveness of OPERA even when we do not have any consistent base estimators, or V.-" is constructed using deep neural networks.
Performance Improvement It would be ideal that the combined estimator V does not perform worse than any of the base estimators (V:T) 1- As OPERA optimizes for the MSE, we can directly obtain the following desired result.
Theorem 2 (Performance improvement). ffá= = a", Vi (1, M < MSE((.")
However, observe that due to bootstrap approximation, A may not be equal to A, and thus â may not be equal to a* Nonetheless, as we will illustrate Section 6, even the non-idealized setting OPERA can often achieve MSE better than any of the base estimators (V 1n
Consistency Some prior works that deal with multiple OPE estimators assume that there at least one known consistent estimator (Thomas and Brunskill, 2016). Under similar assumption that EV. V.- _, J(T), OPERA can be made to fall back to the consistent estimator after a large n, such that v is also consistent, i.c., v P, ((T). Naturally, Vx is weighted combination
5

---

of the base estimators [V;")= Ji=1: if all the base estimators provide unreliable estimates, even in the limit of infinite data, then there is not much that can be achieved by weighted combinations of these unreliable estimators.
Interpretability With a linear weighted for- mulation for V., OPERA allows for the inspec-
Algorithm 1: OPERA with Bootstrap
tion of the assigned weights to which give fur-
Input: offline RL data Da: evaluation policy m: set of OPE estimators
ther insights into the procedure. In Figure I we provide a synthetic example to illustrate the im- pact of bias and variance of the input estimators on the values a.
[OPE1, OPE2, ... OPEx I: number of
bootstrap B; a subsample coefficient
n [0, 1].
Output: estimated x performance SOPERA
Consider a case where there are two OPE es- timators (VT and V/2T) and two corresponding weights and (:2). Let the true unknown quan-
+ 1...K do 8 = -OPE
0
tity be V O. As we can see below, when both estimators have low bias, but one has higher variance, OPERA assigns a higher magnitude of o; for V with a lower variance (Figurel, left). When both estimators have similar vari- ance, and their biases have opposite signs with similar magnitude, then ag ss a1 (Figurel, mid- dle left).
for 11...Bdo do
m1 [DI"
D; < Bootstrap (D., m1)
& = s, (OPE((D)
end
end
M <-
[51, 82, ..., 8k] E RX8
M t [si, 82 ... 8/1 E R Rx
t [(3) 81, 82 82. 3y - st] E Rx
Interestingly, unlike related prior work (Thomas
and Brunskill, 2016), our optimization proce- dure in equation 2 does not require a; 2 Therefore the resulting estimator V may as-
A B "1 E E ERk
arg min, aAa
stt. aa=
SOPERA = a M
sign negative weights for some the estimators.
return SOPERA
This can be observed for the case when the sign of the (V{T( and Bias (IT) are the same. such case, using a positive and negative weight can help cancel out the biases of the base estimators, as observed in Figurel I (middle right). When one estimator has no bias and the other has no variance, a values are inversely proportional their contributions towards the MSE (Figure right).
oy- 0.24
ay- 0.59
Og- 1.28
ay- 0.35
0,76
0.41
-0.28
0.65
Figure 1: Interpreting weights for different estimators. X-axi shows the value of V and Y-axis shows the value of V.
5
Experiment
We now evaluate OPERA on number of domains commonly used for offline policy evaluation. Experimental details, when omitted, are presented in the appendix.
5.1 Task/Domains
Contextual Bandit. We validate the performance of OPERA on the synthetic bandit domain with a 10-dimensional feature space proposed in SLOPE (Su et al., 2020). This domain illustrates how OPERA compares to estimator-selectionn algorithm (SLOPE) that assumes special structure between the estimators. The true reward is a non-linear neural network function. The reward estimators are parametrized by kernels and the bandwidths the main hyperparameters. As in their paper, we ran 180 configurations this simulated environment with different parametrization of the environment. Each configuration is replicated 30 times.
Sepsis. This domain is based on a simulator that allows us model learning treatment options for sepsis patients in ICU (Oberst and Sontag, 2019). There are 8 actions and + +1/-1/0 reward at the

---

Learning Caree
CDP of normaliod MBB
10
38-
655
1834
6/2
12
482
605
3.0
366
1838
3000
5003
10003
0.0
022
5:4
0.5 1.0 330 300
306
530
500
600 300 3930
(b)
Figure 2: Left: Results for contextual handits. (a) MSE of estimators when the dataset size grows. (b) CDF of normalized MSE across 180 conditions by the worst MSE of that condition. Better methods lie in the top-left quadrant. Right: (c) For an MDP domain (Sepsis), we show that as dataset sizes increase, our bootstrap estimation of MSE approaches true MSE for each OPE estimator.
episode end. We experiment with two settings: Sepsis-MDP and Sepsis-POMID. where some crucial states have been masked. We evaluate 7 different policies: optimal policy and 6 noised suboptimal policies, which we obtain by adding uniform noise to the optimal policy.
Graph. Voloshin et al. (2019) introduced a ToyGraph environment with a horizon length T and an absorbing state Tabs 27. Rewards can be deterministic stochastic, with +1 for odd states, -1 for even states plus one based on the penultimate state. We evaluate the considered methods on a short horizon H=4, varying stochasticity of the reward and transitions, and MDP/POMDP settings.
D4RL-Gymm D4RL (Fu et al., 2020) is offline RL standardized benchmark designed and commonly used to evaluate the progress of offline RI algorithms. We use 6 datasets (200k samples each) from three Gym environments: Hopper, HalfCheetah, and Walker2d. We use two datasets from each: the medium-replay dataset, which consists of samples from the experience replay buffer, and the medium dataset, which consists of samples collected by the medium-quality policy. We use conservative Q-learning (CQL) (Kumar et al., 2020), implicit Q-learning (IQL) (Kostrikov et al., 2021), and TD3 (Fujimoto et al., 2018). We train 6 policies from these three algorithms with 2 different hyperparameters for the neural network. We selected FQE hyperparameters for each task and picked checkpoints (one early, one late) to obtain 4 estimators build the OPE ensemble.
5.2 Baseline Ensemble OPE Methods
We compare to using single OPE estimators as well two new baseline algorithms that combine OPE estimates together. AvgOPE: We can compute a simple average estimator that just outputs the average of all underlying OPE estimates. If an estimator the ensemble outputs an arbitrarily bad value, this estimator has no implicit mechanism to ignore such an adversarial estimator. BestOPE: We select the OPE estimator that has the smallest estimated MSE. This estimator can be better than AvgOPE as it can ignore bad estimators. In addition, in different domains, we compare to other OPE strategies such as BVFT (Batch Value Funetion Tournament): making pariwise comparisons between different Q-function estimators with the BVFT-loss (Xie and Jiang, 2021; Zhang and Jiang, 2021). SLOPE: an estimator selection method that based on Lepski's ss method, assuming the estimators forming an order of decreasing variance and increasing bias (Yuan et al., 2021). DR (Doubly Robust); a semi camimerraeetric estimator that combines the IS estimator and FQE estimator to have an unbiased
Table 1: We report the Mean-Squared Error (MSE) for the Sepsis domain. Each number is averaged across 20 trials. We underscore the estimator that has the lowest MSE in the ensemble.
Sepsis
N
OPERA
IS
WIS
FQE
MDP
200
0.2205
0.2753
0.2998
0.2448
MDP
1000
0.1705
0.1720
0.2948
0.2995
POMDP
200
0.2750
0.2804
0.2850
0.3931 0.4078
POMDP
1000
0.2749
0.2799
0.3092
7

---

low variance estimator (Jiang and Li, 2016; Gottesman et al., 2019; Farajtabar et al., 2018). All of these methods place explicit constraints on the type of OPE estimator to include.
Table 2: Root lean-Squared Error (RMSE) of different OPE algorithms across D4RI. tasks.
Multi-OPE Estimator
Single OPE Estimator
Env/Dataset
OPERA
BestOPE
AvgOPE
BVFT
DR
Dual-DICE
MB
Hopper
medium-replay
13.0
15.5
60.7
61.2
112.7
1565.2 368.58
298.7
medium
8.5
12.5
120.8
16.4
16.5
269.7
HalfCheetah
medium-replay
46.0 100.5
65.0
218.6
140.2
119.5
567.9
750,9
medium
111.8
262.1
166.6
145.2
3450.0
589.9
Walker2d
medium-replay 138.3
167.4 .4
187.2
221.5
155.3
2124.3
316.8
medium 149.0
183.8
859.4
264.1
232.1
1756.4
1269.3
5.3 Results
Contextual Bandit We report the result in Figure 2. Figure 2a shows that the dataset size grows, the bootstrapping procedure employed by OPERA can quickly estimate the performance each estimator and compute weighted score that is better than single estimator. In the ultra-small data regime, OPERA worse than singlc-estimator selection style algorithms, mainly because OPERA does not explicitly reject estimators. We can add an additional procedure to reject bad estimators and then combine the rest with OPERA, using a rejection algorithm by Lee et al. (2022).
Sepsis We report the results in Table 1. In this domain, OPERA able produce estimate, on average, across many policies with different degrees of optimality. that matches and exceeds the best estimator the ensemble. Even though in three of the best estimator in the ensemble, in the MDP (N=200) setup. OPERA is able to get significantly lower MSE than any of the estimators in the ensemble, suggesting a future direction of carefully choosing set of weak estimators to put in the ensemble to obtain strong estimator.
of four tasks, OPERA MSE close to the MSE
Graph We report the graph domain result in Appendix A.9 and in Table We find a similar result to the Sepsis domain. OPERA is able to outperform AvgOPE and BestOPE in different setups.
We choose this domain because, in continuous control tasks, the horizon is often very long. Many OPE estimators that rely on short-horizon or discrete actions will not be able to extend to this domain. A popular OPE choice is FQE with function approximation, but it is difficult determine hyperparameters like early stopping, network architecture, and learning rate. We can see that even though FQE used in D4RL is not consistent estimator and does not satisfy OPERA's theoretical assumption, we are still able combine the estimations reach an aggregate estimate with lower MSF.
D4RL We report the results in Table 2.
6 Discussion: Different MSE Estimation Strategies
6.1 Estimating MSE with MAGIC
Part of our algorithm implicitly involves estimating the MSE of each OPE estimator. In our algorithm do this using bootstrapping but other alternatives are possible. For example, prior work by (Thomas and Brunskill, 2016) provided a way to estimate the bias and variance of an OPE estimator are computed through per-trajectory OPE scores and used this as part of their MAGIC estimator. However, this method cannot estimate the MSE of self-normalizing estimators (such as WIS) or minimax-style estimators (such as any estimator in the DICE family (Yang al., 2020)). We denote this estimator as MASEM (V*) and now explore how our approach using boostrapping compares to this method in illustrative setting.
In particular, we consider estimating the MSE of the FQE and IS estimands on the Sepsis-POMII and Sepsis-MDP domains. MAGIC estimates the bias of OPE as the distance between the OPE

---

Table 3: We compare styles of MSE estimations and how well they estimate the true MSE of each estimator. We report averaged results over 10 trials, with N=200.
Sepsis-POMDP
Sepsis-MDP
MSE(V")
MSEMAGIC(VV (V*)
MSE(V")
MSE(V*)
MSEMAGIC (V*)
MSE(V V*)
IS
0.0161
0.0281
0.0088
0.3445
0.0485
0.0056
FQE
0.0979
0.4953
0.0163
0.0077
0.0771
0.0011
Table 4: We report the Mean-Squared Error (MSE) for the Sepsis domain. We additionally present two variants of OPERA where experimented with different MSE estimation strategies.
Sepsis
N
OPERA
OPERA-IS
OPERA-MAGIC
IS
WIS
FQE
MDP
200
0.2205
0.2181
0.2657
0.2753
0.2998
0.2448
MDP
1000
0.1705
0.1779
0.1848
0.1720
0.2948
0.2995
POMDP
200
0.2750
0.2768
0.2827
0.2804
0.2850
0.3931
POMDP
1000
0.2749
0.2720
0.2802
0.2799
0.3092
0.4078
value and the closest upper or lower bound a weighted importance sampling (WIS) policy estimate. We use percentile bootstrap to construct 50% confidence interval CI around WIS.
Our bootstrap (VT) procedure is able to provide a consistently more accurate estimate of the true MSE of the FQE estimate compared to MSEMAGIC( (V) and a comparable or better one for the IS estimate (see Table 3). We suspect that this is due to MAGIC's unique way of computing bias. Specifically, MAGIC computes bias by comparing two estimates (in this case, FQE and the uppper/lower bounds on WIS) which may significantly misestimate the bias in some situations.
6.2 Variants of OPERA with Different Strategies
We now explore two alternative strategies to estimate the MSE of each estimator. The first strategy is, instead of using the estimator's own score as the centering variable V7, use consistent and unbiased estimator's score as D. We call this OPERA-IS. Another strategy is use the idea from (Thomas and Brunskill, 2016)'s MAGIC algorithm, where the bias estimate of each estimator compares the estimand to the upper or lower confidence bound weighted importance sampling estimator, as above. We call this OPERA-MAGIC. These are two new variants of our OPERA A algorithm that will may lead to learning different weights and producing different linearly stacked estimates. We use these two new methods, and compute the true MSE of the resulting stacked estimate, compared to OPERA and other baseline estimates. We use the Sepsis domains to illustrate the results and use as input IS, WIS and FQE OPE estimates.
The true MSE
resulting estimates are presented Table While using an unbiased consistent estimator as the centering variable can help further improve OPERA's estimate, sometimes it also hurts the performance (MDP N=1000 setting). OPERA-MAGIC however almost always performs worse than the best estimator in the ensemble. This suggests that when combining OPE scores this bound on the bias, which will provide a distorted estimate of the estimator bias especially in low data regimes, can lead to learning less effective weightings of the input OPE estimands, OPERA remains a solid option across all settings presented the table.
7 Conclusion
We propose novel offline policy evaluation algorithm, OPERA, that leverages ideas from stack generalization to combine many OPE estimators produce single estimate that achieves lower MSE. Though such stacked generalization / meta-learning has been frequently used to create better estimates from ensembles of input methods in supervised learning, to our knowledge this is the first time it has been explored in offline reinforcement learning. One challenge is that unlike in supervised learning, we do not have ground truth labels for offline policy learning. OPERA uses bootstrapping to estimate the MSE for each OPE estimator in order to find set of weights to blend each OPE's estimate. We provide finite sample analysis of OPERA's performance under mild assumptions, and demonstrate that OPERA provides notably more accurate offline policy evaluation

---

estimates compared to prior methods in benchmark bandit tasks and offline RL tasks, including a Sepsis simulator and the D4RL settings. There are many interesting directions for future work, including using more complicated meta-aggregators.
Acknowledgments and Disclosure of Funding
The research reported in this paper was supported in part by Stanford HAI Hoffman-Yer grant and an NSF #21 12926 grant. thank Sanath Kumar Krishnamurthy, Omer Gottesman, Yi Su, and Adith Swaminathan for the discussions.
References
Cao, R. (1993). Bootstrapping the mean integrated squared error. Journal of Multivariate Analysis, 45(1):137-160.
Chang, J., Wang, K., Kallus, N., and Sun, W. (2022). Learning bellman complete representations for offline policy evaluation. In International Conference on Machine Learning, pages 2938-2971. PMLR.
Chen, Y.-C. (2017a). Introduction to resampling methods. lecture 5: Bootstrap. https://faculty. .df..
washington.edu/yeeee edt//eenchicc1//p4443////
Chen, Y.-C. (2017b). Introduction to resampling methods. lecture 9: Introduction to the bootstrap theory. https://facully: http://faaaulll....asninnntoo http://acceccccccc///..///////// Lttpe//rr_ttthtrrreeeeeeeeee pdf.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2016). Double/debiased machine learning for treatment and causal parameters. arXiv preprint arXiv: 100.000000000000
Danielsson, J., de Haan, L., Peng, L., and de Vries, C. (2001). Using bootstrap method choose the sample fraction in tail index estimation. Journal of Multivariate analysis, 76(2):226-248.
Delaigle, A. and Gijbels. I. (2004). Bootstrap bandwidth selection in kernel density estimation from contaminated sample. Annals of the Institute of Statistical Mathematics, 56(1):19-47.
Diamond, S. and Boyd, (2016). Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research, ((1):2999-29111
Dong, K., Flet-Berliac, Y., Nie, A., and Brunskill, E. (2023). Model-based offline reinforcement learn- ing with local misspecification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7423-7431.
dos Santos, T. R. and Franco, G. C. (2019). Bootstrap for correcting the mean square eπтоr of prediction and smoothed estimates in structural models. Brazilian Journal of Probability and Statistics.
Efron, B. (1990). More efficient bootstrap computations. Journal of the American Statistical Association, 85(409):79-89
Efron, B. (1992). Bootstrap methods: another look the jackknife. Springer.
Efron, B. and Tibshirani, R. J. (1994). An introduction to the bootstrap. CRC press.
Farajtabar, M., Chow, Y., and Ghavamzadeh, M. (2018). More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456. PMLR.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722
Fu, J., Norouzi, M., Nachum, O., Tucker, G., Wang. Z., Novikov, A., Yang, M., Zhang, M. R., Chen, Y., Kumar, A., et al. (2021). Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103. 16596.
Fujimoto, S., Hoof, H., and Meger, (2018). Addressing function approximation error actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR.
10

---

Gamero, M. J., García, J. M., and Reyes, M. (1998). Bootstrapping statistical functionals. Statisties & probability letters, 39(3):229-22 2366
Gao, Q.. Gao, G., Chi, M., and Pajic, M. (2023). Variational latent branching model for off- off-policy evaluation. arXiv preprint 2rx1:22301./2066
Ghosh, M., Parr, W. C., Singh, K., and Babu, G. (1984). A note on bootstrapping the sample median. The Annals of Statisties, 12(3):1 133-)1133--135.
Gottesman, O.. Liu, Y. Sussex, S., Brunskill, E., and Doshi-Velez. F. (2019). Combining parametric and nonparametric models for off-policy evaluation. In International Conference on Machine Learning, pages 2366-2666 2375. PMLR.
Hall, (1990). Using the bootstrap estimate mean squared error and select smoothing parameter in nonparametric problems. Journal of multivariate analysis, 32(2):177-203.
Hong, H. (1999). Lecture 11: Bootstrap.
Jiang, N. and Li, L. (2016). Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652-661. PMLR.
Kallus, N. and Zhou, A. (2020). Confounding-robust policy evaluation in infinite-horizon reinforce- ment learning. Advances in neural information processing systems, 33:22293-22304.
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., and Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(1 1471661776
Kostrikov, I., Nair, A., and Levine. S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint
Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-lcarning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191.
Le, H., Voloshin, C., and Yue, (2019). Batch policy learning under constraints. In International Conference Machine Learning, pages 3703-3712. PMLR.
Lee, J. N., Tucker, G., Nachum, O., Dai, B., and Brunskill, E. (2022). Oracle inequalities for model selection in offline reinforcement learning. arXiv preprint arXiv:221 1.00216...
Li, H. and Maddala, G. (1999). Bootstrap variance estimation of nonlinear functions of parameters: an application to long-run elasticities energy demand. Review of Economics and Statistics, 81(4):728-733.
Liu, Q. Li, L., Tang, Z., and Zhou, D. (2018a). Breaking the curse of horizon: Infinite-horizon off-po estimation. Advances in Neural Information Processing Systems, 31.
Liu, Y., Gottesman, O., Raghu, A., Komorowski, M., Faisal, A. A., Doshi-Velez, F., and Brunskill, E. (2018b). Representation balancing mdps for off-policy policy evaluation. Advances in neural information processing systems, 31.
Luo, Z., Pan, Y., Watkinson, P., and Zhu, T. (2024). Position: reinforcement learning in dynamic treatment regimes needs critical reexamination. Journal of Machine Learning Research.
Mandel, T., Liu, Y.-E., Levine, S., Brunskill, E.,, and Popovic, 7. (2014). Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 1077.
Marchetti, S., Tzavidis, N., and Pratesi, M. (2012). Non-parametric bootstrap mean squared error estimation for m-quantile estimators of small area averages, quantiles and poverty indicators. Computational Statistics & Data Analysis, 56(10))2889-299)
Mikusheva, A. (2013). Time series analysis. lecture 9: Bootstrap. https://ocw mit. edu/ courses/14-388 time series-anall aateees-naalssss-lllllla faw.aaao0o/reeeoreeeeeeeeeeeeeee 14_ lec9/.

---

Nachum, O., Chow, Y., Dai, B., and Li, L. (2019). Dualdice: Behavior Behavior-agmostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32.
Nair, Y. and Jiang, N. (2021). A spectral approach to off-policy evaluation for pomdps. arXiv preprint arXiv:2109, 10502.
Namkoong, H., Keramati, R., Yadlowsky, S., and Brunskill, E. (2020), Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 19-18831.
Nie, X. and Wager. S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319)
Oberst, M. and Sontag, D. (2019). Counterfactual off off-policy evaluation with gumbel max structural causal models. In International Conference on Machine Learning, pages 4881-4890. PMIR.
Päduraru, C. (2007). Planning with approximate and learned models of markov decision processes. PhD thesis, University of Alberta.
Paduraru, C. (2013). Off-policy evaluation in Markov decision processes. PhD thesis, McGill University.
Precup, D. (2000). Eligibility traces for off- policy evaluation. Computer Science Department Faculty Publication Series, page 80.
Ruan, S., Nie, A., Steenbergen, W., He, J., Zhang, J., Guo, M., Liu, Y., Dang Nguyen, K., Wang. C. Y., Ying, R., Landay. J., and Brunskill, E. (2024). Reinforcement learning tutor better supported lower performers in a math task. Machine Learning, pages 1-26.
Shao, J. (1990). Bootstrap estimation of the asymptotic variances of statistical functionals. Annals of the Institute of Statistical Mathematics, 42:737-752.
Shi, C., Uehara, M., Huang, J., and Jiang, N. (2022). A minimax learning approach to off-policy eval- uation confounded partially observable markov decision processes. In International Conference on Machine Learning, pages 20057-20094. PMLR.
Shi, X. (2012). Econ 715. lecture 10: Bootstrap. https: htop::///www... www.wwwwwww.......w...w.......wp wetpeedddd/xxx/e//////// decture_10_bott 10__toottoottppppppp pdf.
Su. Y., Srinath, P., and Krishnamurthy, A. (2020). Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages 1195-9205. PMLR.
Tennenholtz, G., Shalit, U., and Mannor, S. (2020). Off- evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283.
Thomas, P. and Brunskill, E. (2016). Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139-2148. PMLR.
Thomas, P., Theocharous, G., and Ghavamzadeh, M. (2015). High confidence policy improvement. In International Conference on Machine Learning, pages 2380-2388. PMLR.
Tucker, G. and Lee, (2021). Improved estimator selection for off- off-po evaluation. Workshop on Reinforcement Learning Theory at the 38th International Conference on Machine Learning.
Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. (2019). Empirical study of off- policy evaluation for reinforcement learning. arXiv preprint 1.0844.
Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. (2021). Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round) l).
Wang, Y.-X., Agarwal, A., and Dudık, M. (2017). Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference Machine Learning, pages 3589-3597. PMLR.
12

---

Williams, C. J. (2010). The bootstrap method for estimating mse. pdf.
uidaho.
Wolpert, D. H. (1992). Stacked generalization. Neural networks, 5(2):241-259.
Xie, I. and Jiang, N. (2021). Batch value-function approximation with only realizability, In International Conference Machine Learning, pages 11404-11413, PMLR. PMLR.
Yang, M., Nachum, O., Dai, B., Li, L.,, and Schuurmans, D. (2020). Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561.
Yu, T., Thomas, G., Yu, L.. Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. (2020). Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems, 13:14129-1414.
Yuan, C., Chandak, Y., Giguere, S., Thomas, P. S., and Niekum, S. (2021). Sope: Spectrum of off-policy estimators. Advances in Neural Information Processing Systems, 34:18958-1896,
Zhang, J. and Bareinboim, E. (2021). Non-parametric methods for partial identification of causal effects. Columbia CausalAl Laboratory Technical Report.
Zhang, S. and Jiang, N. (2021). Towards hyperparameter-free policy selection for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 34.
13

---

A Appendix
A.1 OPERA Diagram
We describe the OPERA framework as a two-stage process in Figure The first stage involves using black-box statistical methods (such as Bootstrap) to estimate the quality of each OPE estimator. The information then used estimate a weight a through convex optimization objective Eq 9. At stage two, we use the learned a to combine each estimator's s score to output the OPERA score.
Bootstrap
OPE
Optimization
OPE
OPEs
OPE2
D
D
OPERA
OPEs
OPE
Stage I: Estimating
Stage 2: Aggregating OPEs
Figure 3: OPERA framework two-stage process.
A.2
Notation Table
We provide list of important notations used in this paper.
Notation
Description
We use n, nı to refer to the number of trajectories in the dataset.
We use & to refer to the number of estimators in the ensemble.
Dn
D, = {+ = {st, a,, s, r. be the trajectories sampled from и on an MDP.
D
A bootstrapped sample from Dn, with nı data points, 12 n.
A
The cross-estimator bias variance matrix of k estimators. A E RR
The cross-estimator bias variance matrix of k estimators using D Å B
Coefficient for i estimators. a E R
a*
Coefficient for k estimators for solving a² arg ming a Aa.
Coefficient for i estimators for solving å E arg min Aa.
VE
True performance of the policy ग,, V = (
Let i-th OPE estimate of x's performance be V (D,) = Vi(AT, Dn).
VI
We use V/-T = V." (D,) where D, is the full offline dataset.
(*(a))
Weighted average of k estimators with given V (a) = Ein aiVe ER
OPERA's estimated performance æ. V = vā))). A shorthand notation for V V=(a").
V
The mean-squared error of the i-th OPE estimator.
MSE(V())
Based on how we constructed A, we have MSE(VA)) = A
The mean- equare error of रे (OPERA score). MSE(İ²) < A. + A..
MSE(Ò")
The estimated mean- aecan-suaredd error of the i-th estimator using bootstrapped dataset D Based on how we constructed Á, we have MSE(VV,) A
MSE(V())
The change in mean-squared error of OPERA, 인a from using dataset D
A- = ED. (v v=)
The change in mean-squared emor of OPERA, fT, from using sub-optimal
A
A, = ED, (5 pa
14

---

A.3 Bootstrap Convergence
In this section, we provide a high-level discussion of the bootstrap procedure and its asymptotic validity. We refer the readers to the works by (Cao, 1993; Hall, 1990) for more fine-grained analysis and convergence rates when estimating MSE using statistical bootstrap. Individual treatment of bias (Efron, 1990; Efron and Tibshirani, 1994; Hong, 1999; Shi. 2012: Mikusheva, 2013) and variance (Chen, 2017b; Gamero et al., 1998; Shao, 1990; Ghosh et al., 1984; Li and Maddala, 1999) can also be found.
In the following, we will discuss the consistency of Å estimated using bootstrap,
Å. As. O.
(12)
Towards this goal, we will consider the following conditions imposed on the set of the base estimators
U J
Vi, V is uniformly bounded.
Vi, VT "* Cy-
Vi, Vi is smooth with respect to data distribution.
avg VV " ck
V*.
Recall from equation
Arj E
(V (Vr (17 ]m
(13)
E [V, EV;"" + B Vm)
(14)
(V7 B1VJ E153] VT))
(15)
EE [vT - -EV (vr -
(16)
+ E [E\277) V*) (E[V))- - v*)]
(17)
Let Xn = (v. B(TTT) and Y, = ( As Vi-T -e G and Vi- is uniformly bounded, using (Thomas and Brunskill, 2016, Lemma 2), we have B[(:"] a.e c. Similarly, we have B[VT]] a.33 Cj Vir a.s, Cj. Then using continuous mapping theorem,
Y,YY - (c: ca)ee c2) O.
(18)
Now using (Thomas and Brunskill, 2016, Lemma 2),
E [(CT -EV;7) (VT - E(VTT)] = - [(( و3 0.
(19)
Similarly,
[[[ v*) [B|Vg] -v*) 0.4 (c. (C - VT)
(20)
Therefore, using equation 19 and equation 20,
Aij a... 0+ (c- V-
(21)
Now we consider the asymptotic property of the bootstrap estimate À of A.
A = ED; Ip. (VT - Vx (V, Vz
(22)
where VTT is known be consistent estimator, i.e., ViTT 0. Va. Here, V could be the WIS or IS or doubly-robust estimators that are known to provide consistent estimates of V* = ((T). For brevity, we drop the conditional notation on the subscript, and write equation 22 as,
Å. Eng [(VF(D).) Vz (V7 VZ ]
(23)
15

---

Simplifying equation 23.
Aij = ED; (Rr(DE.) EDi,
-D,; (D(,)) }] + EDDs [T -V??' -VF)
(24)
(v)))i))) - -DD;, [D)() + +ED; (r 3 - -Va))
(25)
= =Bp (VT(Di))) - EDE, (VT(Din)))
(26)
(v7,)) - -EDA [VT(Di 1]
(27)
+ ED:, (TT(),)) - V EDD (D - VzT
(28)
Let X,, (.. (D%. ED: (D%, )) and Yn, (V, EDD; Vi (D%) As the
empirical distribution Di, converges to the population distribution, i.e., Dn. 0.s; D, the resampled distribution D2 from Dn also converges to the population distribution, i.e., Dł, when the estimator V (D is smooth, using the continuous mapping theorem,
" D. Therefore,
Vi, lim (T V."
lim Dú, a1
(T) Gi.
(29)
1100
Therefore, similar to before,
Xn, Yn, 0.8 (c: - (i) cj) 0,
(30)
and subsequently.
EDs, [Nn] Yn.l 0.S; O.
(31)
Further, Vi a.4 V*,
(D,,), - VT a.g - V".
(32)
Therefore,
ED; (D*,)) - -VēJ EDi, (DT(Din) VT
(33)
-e (c. V* - V**)
(34)
Using equation 31 and equation 34 in equation 28,
A 0 + (q )c; - VT).
(35)
Finally, combining equation 21 and equation 35.
A. A a.g 0.
(36)
which gives the desired result. It is worth highlighting that, theoretically, this result relies upon assumptions that the base estimators satisfy regularity conditions and are consistent. In practice, such assumptions might not hold (for e.g., when using FQE to do policy evaluation if the function approximation is )araeeeiarameer.zedd)).. Nonetheless, in Section we empirically illustrate that even when these assumptions are not directly satisfied, OPERA can be effective.
A.4 Finite Sample Analysis of OPERA
Without loss of generality, let VT E I1, [m() < such that we can always consider Vi, | < 1 (this can be trivially achieved by normalizing each estimator's output by [Vmax)). Let V be weighted sum of Vi"" with a", where the total number of estimators in the ensemble is k.
16

---

In the following, we show how the enror in estimating the optimal weight coefficients a" affects the MSE of the resulting estimator V Given | we assume that Å obtained uisng the bootstrap procedure of OPERA will produce á via Equation 9 (and resulting estimate of VT). In contrast, using A would have produced a* (and a resulting estimate of V*). To provide a finite sample characterization of OPERA's mean squared error, consider the setting where given samples in dataset D, there exists À > 0, such that
Vi, EDD - oz S n
(37)
where the expectation is over the randomness due to data D,, that governs the estimates V.3 and thus also the weights á and a used to combines these estimates. We now provide proof of Theorem To bound the MSE of OPERA's estimate ,/2 observe that,
(5--v*) V") 27
MSE( = ED.
(38)
(5= -TTh + 7TT - -V-) )
27
En,.
(39)
S Ep..
- (İm-v=T)
+ ED,, - (D=-V*)²
(40)
We isolate the error of र into two terms: Aa and A. Ae is the gap between the best estimate OPERA can give with a* and the true estimate of the policy performance V If V can be expressed
linear combination of V77 then A, 0. Aa is the term want further analyze because depends on the difference between à and a*.
27
Aa ED,
(41)
k
EDn
Σ V Σ a V
(42)
i=1
27
E.,
Σ (a: - at)
(43)
k Σ (āa a;
Σ
< Epn
(44)
where the last inequality follows from Kruchy-Schwarc inequality. Now by using the fact that |0.| S 1 and by plugging equation 37 into equation 44:
A. < ED,
k
Σ i=1
(ā: - az)² (a;=-aa)
(45)
k
EDn Epn((ā: -
(46)
k2
(47)
n2x
Therefore, combining equation 40 and equation 47
k2 n2 + A..
MSE (SS((T)) <
(48)
This bound factors the MSE using the term De, which is the best a linear combination of estimators can do. Notice that A. S min, MSE(V²), ), as the best linear combination of the estimators can at least achieve the MSE of the best estimator, by assigning weight of 1 to the best estimator and 0 to
17

---

the rest. Therefore, the rate of decay of Ae bounded above by the rate of convergence of the best estimator our ensemble.
The other term h² /²/n² in equation 48 results due to the eπror in estimating a' because of the bootstrapping process used for estimating A of A in equation 8. This is dependent on the number of estimators - as we include more estimators in our ensemble, the combination weights a E R that need to be estimated becomes higher dimensional, thereby introducing more errors. However, the overall term decreases as the dataset size n increases.
A.5 Proofs on Properties of OPERA
A.5.1 Invariance
In the following, we illustrate an important property of OPERA, that the resulting combined estimate f is invariant the addition of redundant copies the base estimators [V,)==n Without loss of generality, let V3 [(*+1)x1 the stack unique estimators { {V: with Ve+ being a redundant copy of the V
Theorem (Invariance). If Å is positive definite, then Vz = V", where,
k+
3 B
E
R,
where, 8 arg min B BB.
(49)
BERÍR
Proof. We prove this by contradiction. Recall that á E IR* are the weights that minimize the bootstrap estimate of MSE of कx consisting of l: estimators.
MSE(áa V GzVZ)= à Åa.
(50)
As V is redundant copy of V
MSE(STVT B3VE + Bh+1 vit1) M +... (9) + 8 ()
(51)
(52)
Finally, 8* R is the weight that minimizes the bootstrap estimate of MSE of Vz. Now, if make equation 52 = equation 50. Further, notice that as both â and 33 are within the same feasible set of solutions, the above reassignment is also within the feasible
equation 50 equation 52, then could assign 83 == c; for E (1,....k) and (
of solutions. Similarly, if equation 50 > equation 52, then one could assign a; == Br for i E {I,..., 1}. and â₁ 0 + 3i++ to make equation 52 equation 50. Hence, equation 50 does not equal equation 52, then either or is not optimal and that would be contradiction. This ensures that MSB(V (VT) = MSE((T)
As Å is positive definite, implies that equation is strictly convex with linear constraints. Thus the minimizer á of equation 9 is unique, and V 인œ Note that due to redundancy, B will not be PD despite being PD. This would imply that there can be multiple values of 8% and 9 Nonetheless, since 37 + 3h++ = âg implies that v = -
A.5.2 Performance Improvement
Theorem (Performance improvement). Ifā ffā="; a², Vi E (l....k), MSE(V*) < MSE(V,).
(53)
Proof. With slight overload of notation?, make the dependency of weights explicit and let (Ta(a)) V. Let (SSE(V(()) = T Aa, where defined as equation
Note that in Section A.4, we used 5 to denote (T("). Here make this dependence explicit.
18

---

Now from equation and equation 2, we know that for Σ 1= ag 1,
a' arg min MSE V (a)). DER
(54)
Therefore, for any Rex1 such that Ei=1 =1
MSE(V (ā)) = (SE(V"((")) S MSE(V=(A))
à a
(55)
(56)
Notice that for = [0,0, 1,.,, 0], where there is 1 in the 8t position and zero otherwise, (=((e,) V, Therefore,
MSE(VV & MSE(V"(e))
Vi
(57)
MSE(V))
Vi.
(58)
Therefore, as v V (a), we have the desired result that Vi E {I,..,k) k). MSE((*) MSE(V)).
A.6 Empirical Properties of OPERA
In Figure 4: (a) We show that as dataset sizes increase, our bootstrap estimation of MSE approaches true MSE for each OPE estimator. (b) We show the MSE of OPERA with true and estimated A matrix. Note both MSB(V*) and MSE( VT) are near and overlap each other. (c) We show how a changes between different estimators dataset size grows.
sfft
Hal
622-
31-
Suused
0:110
3.0
01-
862
350
sin
sic
see oce 1088
0000
100
260
203
482 537
620 737
800 300 1848
-34
330
300
360
990 1068
(a)
(b)
(c)
Figure 4: Properties of OPERA
A.7 Graph Experiment
Voloshin et al. (2019) introduced a ToyGraph environment with a horizon length T and an absorbing state Tabs = 2T. Rewards can deterministic or stochastic, with for odd states, -1 for even states plus one based on the penultimate state. We evaluate the considered methods on a short horizon H=4, varying stochasticity of the reward and transitions, and MDP/POMDP settings. We evaluate a single policy in this domain.
Table 5: We report the Mean-Squared Error (MSE) for the Graph domain. We conduct the experiment on Graph over trials. We underscore the estimator that has the lowest MSE in the ensemble.
Stochasticity
Observability
OPERA
BestOPE
AvgOPE
IS
WTS
Deterministic
MDP
0.0339
0.0509
0.2872
0.7398
0.0509
Stochastic
MDP
0.4625
0.4838
0.7021
1,0803
0.4755
Deterministic
POMDP
1,4651
1.2193
2.3425
6.6273
0.4487
Stochastic
POMDP
0.3327
0.3516
0.3889
0.5634
0.3516
We report the results in Table 5. In three out of four setups, OPERA is able to produce an estimate that has a lower MSE compared to BestOPE and AvgOPE and lower than the estimators used in the ensemble. In the deterministic POMDP setting, the IS estimate is significantly off, and OPERA
19

---

Table 6: Root Mean Sean-S Error (RMSE) of the FQE estimators with different hyperparameter configurations.
Env/Dataset
FQE
FQE 2
FQE 3
FQE
Hopper
medium-replay
30.2 52.2
15.5
133.5
153.4
medium
12.5
242.9
237.6
HalfCheetah
medium-replay medium
126.0 158.6
65.0
439.7
318.8
111.8
491.6
386.5
Walker2d
medium-replay
185.8
167.4 .
301.6
167.7
medium 184.9
192.0
406.7
183.8
worse than BestOPE. This provides an important insight into the strengths and limitations of our particular procedure. In small tabular POMDPs, WIS can be quite good. Here, the error in the MSE estimation means that OPERA inaccurately balances the two estimators instead of relying only on WIS. We note that BestOPE does not match the performance of WIS in the ensemble. This because the error in MSE estimation makes BestOPE erroneously choose the wrong estimator in some trials, resulting in larger MSE. We note that on three other setups, the MSE estimation is fairly accurate. Therefore, OPERA is able to get a lower MSE than any of the OPEs in the ensemble.
These results suggest that it is important to consider the tradeoff between bootstrap estimation error and the benefit from combined estimators. When we compare the estimation quality over many policies (Sepsis and D4RL), OPERA does well, but for particular policy, it might not outperform other estimators. This is an interesting area for future work: if and when possible for such meta-algorithm provably match the performance of any individual estimator without further assumptions.
A.8 D4RL Experiment
Setup D4RL (Fu et al., 2020) an offline RL standardized benchmark designed and commonly used to evaluate the progress of offline RL algorithms. We use 6 datasets of different quality from three environments: Hopper, HalfCheetah, and Walker2d. We choose the medium and medium- replay datasets. Medium dataset has 200k samples from policy trained to approximately 1/3 the performance a policy trained to completion with SAC. Mediun-replay dataset takes the transitions stored in the experience replay buffer of policy this dataset can be thought of dataset sampled by a mixture of policies.
Policy Training We train policies from these three algorithms with different hyperparameters for the neural network, Q-learning (CQL) (Kumar el al., 2020), implicit Q-learning (Kostrikov et al., 2021), and TD3+BC (Fujimoto et al., 2018). We initialize all neural networks (including both actor and critics, if the algorithm uses both) with the hidden dimensions of [256, 256, 256). We train with batch size of 512, with Adam Optimizer. We train for 100 epochs on each dataset. We only change one important hyperparameter per algorithm. We report the discounted return of each policy Table 1, 12. We report these scores because they are the prediction target the FQE algorithm. We report the un-discounted return of each policy in Table 15,16.17.
Alg
Initial
Alg
Expectile
Alg
Alpha
CQL
1.0
1 IQL
0.7
TD3+BC 0.7 TD3+BC
CQL 2
10
IQL 2
0.5
0.5
Table
Table 8
Table 9
FQE Training We train Fitted Q learning for each policy. As discussed in the main text, FQE has few hyperparameter choices. We choose hyperparameters for Hopper and HalfChectah. We
20

---

Hopper (medium-rephle
Hopper (medium)
Walker2D (medium-replayy
Walker2D
Policy
Policy
(medium)
CQL
193.47
242.24
CQL 1
252.68
85.39
CQL
123.76
243.57
IQL
238.77
253.19
IQL
239.20
246.26
IQL 2
130.29
243.51
IQL 2
239.85
240.05
CQL
247.03
198.92
TD3+BC 1 TD3+BC
183.48
231.81
TD3+BC
211.28 183.38
247.22
208.16
234.19
TD3+BC
237.85
Table 10: Discounted perf of different policies on Hopper task.
Table 11: Discounted perf of different policies Walker2D task.
HalfCheetah (medium-replay)
HalfCheetah (medium)
Policy
CQL
363.35
601.59
IQL
394.06
436.52
IQL 2
362.65
423.37
CQL 2
354.23
539.03
TD3+BC I TD3+BC2
407.96 318.22
441.20
422.65
Table 12: Discounted perf of different policies on HalfCheetah task.
choose another 4 hyperparameters for Walker2D. The reason is that noticed the Q-value for Walker2D exploded if we used the same hyperparameters for the two other tasks. We should note that since OPERA does require OPEs to be the same across tasks. The hyperparameter choices around the Q-function neural network's 's hidden sizes and how many epochs we train each Q-function. Generally, training too long over-training leads exploding Q-values.
Hopper/ HalfCheetah
Q-Function Network
Training Epochs
Q-Function Network
Training Epochs
Walker2D
FQE
[256, 256, 256]
2
FQE 1
[128, 256, 512]
2
FQE 2
[256, 256, 256]
3
FQE
[128, 256, 512]
5
FQE
[512, 512]
1
FQE
[512, 512]
1
FQE 4
[512, 512]
2
FQE 4
[512, 512]
2
Table 13: FQE Hyperparameters. Training epochs were chosen to be an early checkpoint and a late checkpoint (before exploding Q- values).
Table 14: FQE Hyperparameters. Training epochs were chosen to be an early checkpoint and late checkpoint (before exploding Q- values).
A.9 Sepsis and Graph Experiment Details A.9.1 Sepsis
The first domain is based the simulator and works by Oberst and Sontag (2019) and revolves around treating sepsis patients. The goal of the policy for this simulator discharge patients from the hospital. There three treatments the policy choose from antibioties, vasopressors, and mechanical ventilation. The policy can choose multiple treatments at the same time or no treatment at all, creating different unique actions.
The simulator models patients as combination of four vital signs: heart rate, blood pressure, oxygen concentration and glucose levels, all with discrete states (for example, for heart rate low, normal and high). There is latent variable called diabetes that present with a 20% probability which drives the likelihood of fluctuating glucose levels. When a patient has at least 3 of the vital signs simultancously
21

---

Hopper
Hopper (medium-v2)
Walker2D (mediuu-reeplay-v2)
Walker2D
Policy
Policy
(medium-replla-vv2
(medium-v2)
CQL
433.40
2550.03
CQL 1
3732.01
145.25
CQL
439.56
2787.95
IQL
2383.09
3044.03
IQL
3144.02
1768.19
IQL 2
776.79
3194.87
IQL 2
2177.90
2028.27
CQL
3073.49
1409.01
TD3
1104.04
1977.88 1751.87
TD3 TD3 2
2250.07
3920.79
TD32 2
910.26
1656.82
3732.23
Table 15:
Undiscounted perf of different poli-
Table 16:
Undiscounted perf of different poli- cies Walker2D task.
cies Hopper task.
HalfCheetah (medium-replay-vv)
HalfCheetah (medium-v2)
Policy
CQL
4053.04
7894.69
CQL
4192.01
6875.66
IQL
4995.02
5704.12
IQL
4657.00
5475.88
TD3 1
5324.46
5758.83
TD3 2
5002.90
5420.27
Table 17: Undiscounted perf of different policies HalfCheetah task.
out of the normal range, the patient dies. If all vital signs are within normal ranges and the treatments all stopped, the patient discharged. The reward function is +1 if patient discharged, -1 if patient dies, and ( otherwise. We truncate the trajectory to 20 actions (H=20). For this simulator, early termination means we don't 't get observe a positive or negative return on the patient.
We follow the process described by Oberst and Sontag (2019) to marginalize an optimal policy's ' action over 2 states: glucose level and whether the patient has diabetes. This creates the Pepsis-POMDP environment. We sample 200 and 1000 patients (trajectories) from Sepsis-POMDP environment with the optimal policy that has 5% chance of taking random action. We also sample trajectories from the original MDP using the same policy; we call this the Sepsis-MDP environment.
use tabular FQE. Therefore, there is no representation mismatch. We additionally use fross-fitting, form of procedure commonly used in causal inference (Chernozhukov et al., 2016). Cross-fitting a sample-splitting procedure where we swap the roles of main and auxiliary samples to obtain multiple estimates and then average the results. The main goal of cross-fitting is to reduce overfitting. We notice significant performance improvement of our FQE estimator after using cross-fitte cross-fitting. We present the RMSE of each of our trained FQE estimator Table 6.
FQE Training
A.9.2 Graph
For the graph environment, set the horizon H=4, with either POMDP MDP and ablate on the stochasticity of transition and reward function. The optimal policy for the Graph domain is simply the policy that chooses action 0. All the experiments reported have 512 trajectories.
A.10
Compute Resource
The training was done small cluster of 6 servers, each with 16GB RAM, and 4-8 GPUs of Nvidia A5000. D4RL was the most computationally expensive experiment.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions are clearly stated in the introduction and detailed in Section 4, and claimed improvements are validated in Section 5. Some claims empirically verified in Section 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitation is discussed in Section 6, and we discussed again in the last paragraph of the Appendix Section A.7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
23

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theorems are provided in Section 4. Full proofs are included in Appendix Section A.I, A.2, and A.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experiments are described in Section and additional details are in Appendix Section A.5, A.6, A.7, and A.8.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes. The code is submitted as part of the supplementary material. We did not use any non-open-source data model. Additional details are in Appendix Section A.5, A.6, A.7, and A.8.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
www.aasssonnoiigiioo (alloliyy))) cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: ptaps.cc/public/guue wwtiiiccggue///eeeeeeeeeeeeaeeee http://ggggide//oeeeuutnn ssi wsw.nnsoisleeeceeeooo onPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide all the details in Section and additional details are Appendix Section A.5, A.6, A.7, and A.8.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard error of the mean (SEM) in Figure 2. We do not conduct any statistical test in this paper.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report it in Appendix Section A.9.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
26

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://rrr....cpaaaa//aacaaataaa
Question:
Answer:
[Yes]
Justification:
The contributions of this paper (OPERA) do not have direct safety security implications.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Sections 6 and 7, discuss both the aspirational goals and their broader impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We not releasing data models that have high risk for misuse.
27

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: datasets used in the experiments are properly attributed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
We not introducing assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
28

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no crowdsourcing or human subject studies conducted in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: he paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "KbP61bAuvK", "reviewer": "Reviewer_Rbow", "summary": "The paper introduces OPERA, an algorithm for offline policy evaluation (OPE) in reinforcement learning (RL). OPERA addresses the challenge of selecting the best OPE estimator by adaptively combining multiple estimators using a statistical procedure that optimizes their weights to minimize mean squared error (MSE). The authors prove OPERA's theoretical consistency and desirable properties, ensuring it is at least as accurate as any individual estimator. The algorithm employs a bootstrapping method to estimate MSE, circumventing the need for direct access to true performance measures.", "questions": "1.The theoretical guarantees of OPERA rely on the assumption that at least one base estimator is consistent. How does OPERA perform when this assumption does not hold? Are there any mechanisms within OPERA to detect and handle inconsistent base estimators?\n\n2. What practical considerations should be taken into account when implementing OPERA in real-world scenarios? Are there specific guidelines for choosing the initial set of estimators or criteria for including new estimators in the ensemble?", "limitations": "This paper seems to lack a section for limitation. I recommend adding 1 short paragraph to the conclusion section to summarize the border limitations and social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper presents a novel approach to offline policy evaluation by introducing OPERA, a method that adaptively blends multiple OPE estimators using a statistically optimized weighting mechanism. This approach is innovative as it leverages the strengths of various estimators without requiring explicit selection. The use of bootstrapping to estimate mean squared error (MSE) and optimize the estimator weights is a simple yet creative combination of existing statistical techniques in a new context.\n\n2. The paper is well-written and clearly structured. The problem statement, methodology, theoretical analysis, and experimental results are presented in a logical and coherent manner. The use of mathematical formulations and proofs is precise, aiding in the clear communication of complex concepts. Additionally, the inclusion of diagrams and pseudocode for the OPERA algorithm enhances understanding and provides readers with a clear roadmap of the proposed method.", "weaknesses": "**Assumption of Consistent Estimators**\nThe theoretical guarantees provided for OPERA rely on the assumption that at least one of the base estimators is consistent. However, in practice, this assumption may not always hold, especially in complex or noisy environments where all available estimators could be biased or inconsistent. The paper could benefit from a discussion on how OPERA performs under such conditions and whether there are ways to relax this assumption while still maintaining acceptable performance. For example, could the authors design an experiment where the state, action and/or reward are severely imbalanced, which leads to insufficient data coverage and approximation error to behaviour policy? A practical reflection can be found in a critical study of real-world ICU treatment [1].\n\n[1] Luo, Zhiyao, et al. \"Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback! We incorporated these discussions into the paper but respond to them individually here. \n\n> OPERA rely on the assumption that at least one of the base estimators is consistent. However, in practice, this assumption may not always hold. The paper could benefit from a discussion on how OPERA performs under such conditions and whether there are ways to relax this assumption while still maintaining acceptable performance. For example, could the authors design an experiment where the state, action and/or reward are severely imbalanced, which leads to insufficient data coverage.\n\n> How does OPERA perform when this assumption does not hold? Are there any mechanisms within OPERA to detect and handle inconsistent base estimators?\n\nThanks for raising this important issue. In general, if there is good coverage over states and actions, then including an IS estimator is sufficient to ensure that a consistent base estimator exists. However, if there is not good coverage, that will result in poor inconsistent estimators. OPERA is fundamentally an ensemble method (similar to stacking in statistics) that combines multiple estimators. As long as we can accurately estimate the MSE of the estimators in the reward/action imbalanced conditions, then OPERA will be able to improve upon base estimates.\n\nAs for mechanisms, we can introduce ideas such as KL divergence between the behavior/sampling policy and the evaluation policy, which will indicate when state/action/reward imbalance might occur. If this is the case, most OPE methods, in general, will have issues returning an accurate estimate. We would recommend collecting additional data before the evaluation or including estimators that can produce reliable OPE estimates in this case.\n\n> What practical considerations should be taken into account when implementing OPERA in real-world scenarios? Are there specific guidelines for choosing the initial set of estimators or criteria for including new estimators in the ensemble?\n\nThank you for this question! We are adding guidelines as a section in the appendix, and we summarize them here:\n1. The estimators should be selected based on the properties of the domain well. For example, if the task horizon is short, then IS estimators are often useful to include.\n2. If there are estimators known to under or over-estimate the policy performance, then selecting a balanced set of such estimators can allow OPERA to cancel out the bias of these estimators (see the discussion in Sec 4.1 Interpretability).\n3. Choosing a reasonable number of estimators (starting with K=3) and only include more when the dataset size grows.\n\n> This paper seems to lack a section for limitation. I recommend adding 1 short paragraph to the conclusion section to summarize the border limitations and social impact.\n\nWe have added text in the conclusion to address the limitation. Thank you for the suggestion!\n\nThank you for the review. Did this answer your questions? We are also happy to answer more questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "66FaVj7hOv", "reviewer": "Reviewer_QbuQ", "summary": "The authors propose a novel offline policy evaluation algorithm, that linearly blends the estimates from many OPE estimators to produce a combined estimate that achieves a lower MSE.", "questions": "Considering the first paragraph in weaknesses, can you discuss whether OPERA systematically favors biased estimators? \n\nRecent work of Cief et al. (2024) provides a new way of estimating MSE, which can potentially improve OPERA.  As this was published after the NeurIPS deadline, I do not consider it an issue.\n\nCief, Matej, Michal Kompan, and Branislav Kveton. “Cross-Validated Off-Policy Evaluation.” arXiv, May 24, 2024. http://arxiv.org/abs/2405.15332.", "limitations": "The authors provide a thorough discussion on the limitations in Section 6 and Appendix A.7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written, complete, and easy to read.\n\nThe experiments are well executed, the methods are evaluated in numerous different domains and compared against other relevant baselines in each of them.\n\nI specifically like the discussion on different MSE estimation strategies in Section 6, which shows that the authors really thought about the problem and various choices made in their method.", "weaknesses": "The authors propose to estimate MSE of \\hat{V}_i by estimating \\pi using bootstrapped D_n and calculating the squared error from an estimate of \\pi done by \\hat{V}_i using the original D_n dataset. I think this underestimates bias. For example, if \\hat{V}_i returns a constant for any dataset and policy, then its MSE would be 0, and I assume \\hat{\\alpha}_i would be 1, meaning the ensemble reward would correspond to this constant. I appreciate the variants of OPERA presented in 6.2 that partially address this.\n\nMinor:\n* [L114] D_n is defined both as 1 to n and 0 to n\n* [L125] \\theta_* is undefined\n* [L136] Eq. 4 is referred although it is not labeled when defined (and many others)\n* [L169] \\theta_* is still undefined\n* [L189] The word equation is repeated twice, I recommend using the LaTeX package cleveref", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and pointing out minor errors. We have corrected the inconsistent symbols and clerical mistakes. Much appreciated!\n\n> The authors propose to estimate MSE of \\hat{V}_i by estimating \\pi using bootstrapped D_n and calculating the squared error from an estimate of \\pi done by \\hat{V}_i using the original D_n dataset. I think this underestimates bias. For example, if \\hat{V}_i returns a constant for any dataset and policy, then its MSE would be 0… I appreciate the variants of OPERA presented in 6.2 that partially address this.\n\n> …Considering the first paragraph in weaknesses, can you discuss whether OPERA systematically favors biased estimators?\n\nThank you for asking about this. We don’t believe that OPERA systematically favors biased estimators— for example, if there is an estimator which always returns -10^6 or 10^6 with 50/50 probability, it would have a high variance (in addition to being biased) and OPERA would likely place low weight on this estimator. We do completely agree that the method (prior to 6.2) can underestimate the bias. Indeed, using Bootstrap to estimate MSE reliably assumes that the estimator will approach the true target asymptomatically. Therefore, an estimator that only produces a constant value invalidates the Bootstrap procedure, and its MSE cannot be reliably estimated with Bootstrap. One way to address this is by using estimators that have known convergence guarantees.\n\nWe agree that empirically estimators that have very low variance but are potentially very biased (i.e., model-based estimators or FQEs) might be favored by OPERA. As the reviewer notes, this helped motivate the modifications we present in Section 6.2, where we use a consistent estimator as the centering variable/estimator.\n\nWe will expand our discussion of these issues in the paper.\n\n> Recent work of Cief et al. (2024) provides a new way of estimating MSE, which can potentially improve OPERA. As this was published after the NeurIPS deadline, I do not consider it an issue.\n\nThank you for suggesting this paper– we look forward to going through it in more detail, though, as the reviewer notes, it was released after the deadline. \n\nThank you for the review. Did this answer your questions? We are also happy to answer more questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VxWEcfaCTq", "reviewer": "Reviewer_BP3B", "summary": "The paper \"OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators\" deals with the challenge of evaluating new decision-making policies using past data, which is vital in areas like healthcare and education where mistakes can be costly.\nThe authors introduce OPERA, a new algorithm that improves the accuracy of policy evaluation by combining several existing estimators instead of relying on just one. This approach makes the evaluations more reliable.\nThey provide a solid theoretical basis for OPERA, showing that it effectively minimizes errors and meets key evaluation criteria. A standout feature is using bootstrapping to estimate the mean squared error (MSE) of each estimator, which helps in finding the best way to combine them.\nThe paper demonstrates OPERA's effectiveness through tests in various fields, such as healthcare and robotics. It consistently outperforms other methods by selecting better policies and achieving lower errors.\nOne of the key contributions is OPERA's flexibility. It can be used in many applications without needing strict conditions on the input estimators, making it a versatile tool for different offline reinforcement learning tasks.\nIn summary, the paper presents a new method for policy evaluation that combines multiple estimators to enhance accuracy and reliability, backed by strong theoretical and practical evidence.", "questions": "The paper primarily compares OPERA with traditional methods. Have you considered comparing OPERA with more recent state-of-the-art models? Including such comparisons could provide a clearer picture of OPERA's competitive edge.", "limitations": "The authors have done a great job addressing the limitations of their work. They recognize the challenges of combining multiple estimators and provide a solid theoretical foundation to back up their approach. They also discuss using bootstrapping to tackle issues like bias and variance, which is a key part of their method.\nHowever, it would be helpful to add a separate section that dives into potential edge cases and limitations in more detail. This could cover situations where OPERA might have difficulties, such as dealing with very noisy or incomplete data, and practical challenges users might face during implementation.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: \nThis paper brings an innovative approach to offline policy evaluation with the OPERA algorithm. Instead of relying on a single estimator, OPERA cleverly combines multiple estimators to improve accuracy. This innovation taps into the strengths of different estimators, solving a major limitation of existing methods. The creative use of bootstrapping to estimate mean squared error (MSE) adds to the algorithm's robustness, making it a standout contribution.\nQuality: \nThe quality of this paper is excellent. The authors do a fantastic job of defining the problem of offline policy evaluation and explaining why current methods fall short. They clearly describe their proposed solution, the OPERA algorithm, and walk the reader through each step in a logical and detailed manner.\nThe theoretical groundwork is solid. The authors back up their claims with rigorous proofs, showing that OPERA effectively minimizes mean squared error (MSE) and meets important criteria for policy evaluation. They also anticipate potential issues with existing methods, like bias and variance, and introduce innovative solutions to tackle these challenges using bootstrapping.\nThe empirical validation is thorough and convincing. The authors test OPERA across various fields, including healthcare (like sepsis treatment) and robotics, demonstrating its effectiveness. They detail their experimental setup, the datasets used, and the baseline methods for comparison. The results are clear: OPERA consistently outperforms other methods by selecting better policies and achieving lower errors, which strongly supports the practical value of their algorithm.\nClarity: \nThe paper is well-structured and clearly written. The flow is logical and easy to follow, from problem definition to solution proposal and validation.\nSignificance: \nThis paper makes a significant impact in offline reinforcement learning and policy evaluation. By combining multiple estimators, the authors solve a crucial problem, especially in critical fields like healthcare and education. The results show that OPERA consistently outperforms existing methods, proving its value for both researchers and practical applications.", "weaknesses": "While the theoretical and empirical aspects are well-covered, the paper could really use a more detailed discussion on how to implement OPERA in practice. Offering guidelines or best practices for using OPERA in different situations would be very helpful for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review.\n\n> the paper could really use a more detailed discussion on how to implement OPERA in practice. Offering guidelines or best practices for using OPERA in different situations would be very helpful for practitioners.\n\nThank you for the suggestion. We are adding guidelines as a section in the appendix, and we summarize them here:\n1. The estimators should be selected based on the properties of the domain well. For example, if the task horizon is short, then IS estimators would be great; but if the task horizon is long, we should not include IS estimators.\n2. If there are estimators known to under or over-estimate the policy performance, then selecting a balanced set of such estimators can allow OPERA to cancel out the bias of these estimators (see the discussion in Sec 4.1 Interpretability).\n3. Choosing a reasonable number of estimators (starting with K=3) and only include more when the dataset size grows.\n\n> The paper primarily compares OPERA with traditional methods. Have you considered comparing OPERA with more recent state-of-the-art models? \n\nThank you for the suggestion. We do compare with methods such as SLOPE [1] and BVFT [2] which to our knowledge are state of the art. On the contextual bandit domain, we show that OPERA outperforms SLOPE across 180 conditions when the dataset size is larger than 300. For more realistic robotic control tasks (D4RL), OPERA outperforms BVFT on three tasks: Hopper, HalfCheetah, and Walker2d. If there are other additional algorithms the reviewer was thinking of, please let us know as we’d be happy to look into these. \n\n[1] Su, Y., Srinath, P., and Krishnamurthy, A. (2020). Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages 9196–9205. PMLR. \n\n[2] Xie, T. and Jiang, N. (2021). Batch value-function approximation with only realizability. In International Conference on Machine Learning, pages 11404–11413. PMLR.\n\nThank you for the review. Did this answer your questions? We are also happy to answer more questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o59snsBKUi", "reviewer": "Reviewer_6uiB", "summary": "The paper addresses the challenge of evaluating new sequential decision-making policies using OPE techniques. The authors propose a new algorithm, OPERA, which adaptively blends multiple OPE estimators to improve the accuracy of policy performance estimates without relying on explicit selection methods. This approach leverages bootstrapping to estimate the mean squared error of different estimator weightings and optimizes it as a convex problem. The proposed method is shown to be consistent and demonstrates superior performance in selecting higher-performing policies in healthcare and robotics compared to existing approaches. This contributes to a more general-purpose, estimator-agnostic framework for offline reinforcement learning.", "questions": "How should we construct the set of estimators to perform the proposed algorithm? Can the authors propose any general guideline? \n\n\nWhat would be the intuition of the better effectiveness of the proposed method against standard OPE? Some reasonable accuracy of the proposed method means that we can estimate the MSE somewhat accurately, meaning that we can do OPE similarly accurately, in my intuition.\n\n\nIn theorem 1, the error rate of \\alpha estimation is not guaranteed, right?\n\n\nWhen combining different estimators, can we consider more complex functions such as polynomials or more general function classes rather than just linear combinations as done in the paper?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper formulates an interesting problem of how to aggregate different OPE estimators to achieve better(lower) MSE in a data-driven way\n\n\nThe paper provides interesting empirical investigations regarding the tuned alpha in Figure 1, which is interesting and helpful to understand how the method works\n\n\nThe paper provides promising empirical results on both contextual bandit and reinforcement learning environments, showing that, by coming multiple estimators, we can achieve better estimations in a range of situations", "weaknesses": "There still remains an important question of how to construct an appropriate set of estimators before performing the proposed ensemble algorithm \n\n\nI still do not get an intuition of if the proposed method solves fundamentally an easier problem than standard OPE. I mean, if we know the MSE, then it is really straightforward that the proposed algorithm works well, but we need to estimate it somehow, which includes the true policy value, which is the estimand of OPE. Therefore, intuitively, MSE estimation has the same difficulty as OPE, but the proposed algorithm works better than OPE. I’d like to know in what sense the proposed algorithm makes OPE easier.\n\n\nIn theorem 1, it seems that the error rate of \\alpha estimation is given, which is not guaranteed, if my understanding is correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review.\n\n> How should we construct the set of estimators to perform the proposed algorithm? Can the authors propose any general guideline?\n\nThank you for the suggestion. We are adding guidelines as a section in the appendix, and we summarize them here:\n1. The estimators should be selected based on the properties of the domain well. For example, if the task horizon is short, then IS estimators would be great, but if the task horizon is long, we should not include IS estimators.\n2. If there are estimators known to under or over-estimate the policy performance, then selecting a balanced set of such estimators can allow OPERA to cancel out the bias of these estimators (see the discussion in Sec 4.1 Interpretability).\n3. Choosing a reasonable number of estimators (starting with K=3) and only include more when the dataset size grows.\n\n>  What would be the intuition of the better effectiveness of the proposed method against standard OPE? \n\nThank you for the good question. The reviewer is correct that OPERA uses estimates of the MSE, which can seem intuitively to be the same hardness as OPE estimation. However, OPERA fundamentally is an ensemble method (similar to stacking in statistics) that combines multiple estimators. For OPERA to offer an improvement over some of the input OPE estimators, we do not have to have a perfect estimate of their MSE, but a good enough estimate that we can combine across them. Note that OPERA with an estimated MSE per estimator is not guaranteed to be as good as picking the (unknown) estimator with the true best MSE. However, this estimator is unknown because, as the reviewer notes, we do not know the true MSE.  We find empirically that the bootstrap estimates provide a reasonable enough approximation that we can find a decent weighting using OPERA that yields an estimator that often (see e.g. Table 1) improves over all individual input estimators, as Figure 1 (and the last part of Section 4) discusses should be possible in some cases given the true (unknown) MSE. We are happy to add additional discussion around this point in the paper. We also note that we think further work on better MSE estimation could further improve OPERA. \n\n> In theorem 1, the error rate of \\alpha estimation is not guaranteed, right?\n\nThe reviewer is correct, and we will make sure this is clear. The MSE of OPERA has a factor of $\\lambda$.\n\n> When combining different estimators, can we consider more complex functions such as polynomials or more general function classes rather than just linear combinations as done in the paper?\n\nYes, we completely agree this would be an interesting direction for future work. We choose to focus initially on a linearly weighted estimator, which allows us to decompose OPERA’s MSE as a function of the underlying estimators’ MSE: see the derivation in Remark 1. This also allowed the optimization objective to be a convex quadratic program. Other choices of combination might result in different non-convex optimization objectives which, depending on the solution techniques, might introduce additional approximations in OPERA’s policy value estimate. Still, we agree more complicated functions would be worth investigating since they give additional modeling flexibility.\n\nDid this answer your questions? We are also happy to answer more questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents OPERA, a novel and estimator-agnostic approach to offline policy evaluation that significantly improves accuracy by adaptively blending multiple OPE estimators. The innovative use of bootstrapping to estimate the mean squared error of different estimator weightings, and its subsequent optimization as a convex problem, demonstrates a sound theoretical foundation. Comprehensive empirical evaluation across domains like healthcare and robotics showcases OPERA's superior performance in selecting higher-performing policies compared to existing approaches. This work represents a valuable contribution to the field of offline reinforcement learning, and thus I recommend acceptance of this paper.
Decision: Accept (poster)