OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

Authors: allen  nie
yash  chandak
christina j yuan
anirudhan  badrinath
yannis  flet-berliac
emma  brunskill
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:39:22

Abstract: Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

Main paper: 
 introduction: 
 Offline reinforcement learning (RL) involves learning better sequential decision policies from logged historical data, such as learning a personalized policy for math education software (Mandel et al., 2014; Ruan et al., 2024) , providing treatment recommendations in the ICU (Komorowski et al., 2018; Luo et al., 2024) or learning new controllers for robotics (Kumar et al., 2020; Yu et al., 2020) . Offline policy evaluation (OPE), in which the performance J(π e ) of a new evaluation policy π e is estimated given historical data, is a common subroutine in offline RL for policy selection, and can be particularly important when deciding whether to deploy a new decision policy that might be unsafe or costly. Offline policy evaluation methods estimate the performance of an evaluation policy π e given data collected by a behavior policy π b . There are many existing OPE algorithms, including those that create importance sampling-based estimators (IS) (Precup, 2000) , value-based estimators (FQE) (Le et al., 2019) , model-based estimators (Paduraru, 2013; Liu et al., 2018b; Dong et al., 2023) , doubly robust estimators Jiang and Li (2016) ; Thomas and Brunskill (2016) , and minimax-style estimators (Liu et al., 2018a; Nachum et al., 2019; Yang et al., 2020) .This raises an important practical question: given a set of different OPE methods, each producing a particular value estimate for an evaluation policy, what value estimate should be returned? A simple approach is to avoid the problem and pick only one OPE algorithm or look at the direction of a set of OPE algorithms' scores as a coarse agreement measure. Voloshin et al. (2021) offered heuristics based on high-level domain structure (e.g., horizon length, stochasticity, or partial observability), but this does not account for instance-specific information related to the offline dataset adn policies.In this paper we seek to aggregate the results of a set of multiple off-policy RL estimators to produce a new estimand with low mean square error. This work is related to several streams of prior work: (1) multi-armed bandit and RL algorithms that combine two estimands to yield a more accurate estimate;(2) multi-armed bandit and RL algorithms that select a single estimand out of a set of estimands, and (3) stacked generalization / meta-learning / super learning methods in machine learning.Research in (1) builds on doubly robust (DR) estimation in statistics to produce an estimand that combines important sampling and model-based methods (Jiang and Li, 2016; Gottesman et al., 2019; Farajtabar et al., 2018) . Similarly, accounting for multiple steps, the MAGIC estimator blends between IS-based and value-based estimators within a trajectory (Thomas and Brunskill, 2016) . The second line of work (2) does not combine scores but instead introduces an automatic estimator selection subroutine in the algorithm. However, such methods typically assume strong structural requirements on the input estimators. For example, Su et al. (2020) ; Tucker and Lee (2021) assume as input a nested set of OPE estimators, where the bias is known to strictly decrease across the set. Zhang and Jiang (2021) leveraged a set of Q-functions trained with fitted Q-evaluation (FQE) and cross-compare them in a tournament style until one Q-function emerged. None of these methods allow mix-and-match of different kinds of OPE estimators.Our work is closest to a third line of more distant work, that of stacked generalization (Wolpert, 1992 ) / meta-learning and super learning across ensembles. There is a long history in statistics and supervised learning of combining multiple input classification or regression functions to produce a better metafunction. Perhaps surprizingly, there is little exploration of this idea to our knowledge in the context of RL or multi-armed bandits. The one exception we are aware of was for heterogeneous treatment effect estimation in a 2-action contextual bandit problem, where Nie and Wager (2021) utilized linear stacking to build a consensus treatment effect estimate using two input estimatands.In this paper we introduce the meta-algorithm OPERA (Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators). Inspired by a linear weighted stack, OPERA combines multiple generic OPE estimates for RL in an ensemble to produce an aggregate estimate. Unlike in supervised learning where ground truth labels are available, in our setting a key choice is how to estimate the mean squared error of the resulting weighted ensemble. Under certain conditions, bootstrapping (Efron, 1992) can approximate finite sample bias and variance. We use bootstrapping to compute estimates of the mean squared error of different weightings of the underlying input estimators, which can then be optimized as a constrained convex problem. OPERA can be used with any input OPE estimands. We prove under mild conditions that OPERA produces an estimate that is consistent, and will be at least as accurate as any input estimand. We show on several common benchmark tasks that OPERA achieves more accurate offline policy evaluation than prior approaches, and we also provide a more detailed analysis of the accuracy of OPERA as a function of choices made for the meta-algorithm.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The contributions are clearly stated in the introduction and detailed in Section 4, and claimed improvements are validated in Section 5. Some claims are empirically verified in Section 6.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Offline policy evaluation Most commonly used offline policy estimators can be divided into a few categories depending on the algorithm. An important family of estimators focuses on using importance sampling (IS) and weighted importance sampling (WIS) to reweigh the reward from the behavior policy (Precup, 2000) . These estimators are known to produce an unbiased estimate but have a high variance when the dataset size is small. For a fully observed Markov Decision Process (MDP), a model-free estimator, such as fitted Q evaluation (FQE), is proposed by Le et al. (2019) , and one can also learn a model given the data to produce a model-based (MB) estimate (Pȃduraru, 2007; Fu et al., 2021; Gao et al., 2023) . When the behavior policy's probability distribution over action is unknown, a minimax style optimization estimator (DualDICE) can jointly estimate the distribution ratio and the policy performance (Nachum et al., 2019) . For a partial observable MDP (POMDP), many of these methods have been extended to account for unobserved confounding, such as minimax style estimation (Shi et al., 2022) , value-based estimation (Tennenholtz et al., 2020; Nair and Jiang, 2021) , uses sensitivity analysis to bound policy value (Kallus and Zhou, 2020; Namkoong et al., 2020; Zhang and Bareinboim, 2021) , or learns useful representation over latent space (Chang et al., 2022) .Choosing the right estimators has become an issue when there are many proposals even under the same task setup and assumptions. Voloshin et al. (2021) proposed an empirical guide on estimator selection. One line of work tries to combine multiple estimators to produce a better estimate by leveraging the strengths of the underlying estimators, for example, (weighted) doubly robust (DR) method (Jiang and Li, 2016) . For contextual bandit, Wang et al. (2017) proposed a switch estimator that interpolates between DM and DR estimates with an explicitly set hyperparameter. For sequential problems, MAGIC blends a model-based estimator and guided importance sampling estimator to produce a single score (Thomas and Brunskill, 2016) . Another line of work tackles the many-estimator problem by reformulating multiple estimators as one estimator. Yang et al. (2020) reformulated a set of minimax estimators as a single estimator with different hyperparameter configurations. Yuan et al. (2021) constructed a spectrum of estimators where the endpoints are an IS estimator and a minimax estimator and proposed a hyperparameter to control the new estimator. This line of approaches does not leverage multiple estimators or solve the OPE selection problem because they recast the OPE selection problem as a hyperparameter selection problem. The last line of work provides an automatic selection algorithm that chooses one estimator from many, relying on an ordering of estimators (Tucker and Lee, 2021) or being able to compare the output (such as Q-values) directly Zhang and Jiang (2021) .Bootstrapping for model selection Using bootstrap to estimate the mean-squared error for model selection was initially proposed by Hall (1990) , for the application of kernel density estimation. The idea was subsequently used by others for density estimation (Delaigle and Gijbels, 2004) , selecting sample fractions for tail index estimation (Danielsson et al., 2001) , time-series forecasting (dos Santos and Franco, 2019) and other econometric applications (Marchetti et al., 2012) . Similar ideas have been explored by Thomas et al. (2015) to construct a confidence interval for the estimator. We extend this idea to use bootstrapping to combine multiple OPE estimators to produce a single score.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitation is discussed in Section 6, and we discussed again in the last paragraph of the Appendix Section A.7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
notation and problem setup: 
 We define a stochastic Decision Process M = ⟨S, A, T, r, γ⟩, where S is a set of states; A is a set of actions; T is the transition dynamics; r is the reward function; and γ ∈ (0, 1) is the discount factor. LetD n = {τ i } n i=1 = {s i , a i , s ′ i , r i } n i=1be the trajectories sampled from π on M . We denote the true performance of a policy π as its expected discounted return J(π) = E τ ∼ρπ [G(τ )] where G(τ ) = ∞ t=0 γ t r t and ρ π is the distribution of τ under policy π. In an off-policy policy evaluation problem, we take a dataset D n , which can be collected by one or a group of policies which we refer to as the behavior policy π b on the decision process M . An OPE estimator takes in a policy π e and a dataset D n and returns an estimate of its performance, where we mark it as V : Π × D → R. We focus on estimating the performance of a single policy π. We define the true performance of the policy V π = J(π), and multiple OPE estimates of its performance asV π i (D n ) = Vi (π, D n ) for the i-th OPE's estimate. 
opera: 
 In this section, we consider combining results from multiple estimators { V π i } k i=1 to obtain a better estimate for V π . Towards this goal, given { V π i } k i=1 , we propose estimating a set of weightsα * i ∈ R such that V π := k i=1 α * i V π i ∈ Rhas the lowest mean squared error (MSE) towards estimating V π . Formally, let V ∈ R k×1 be a vector whose elements correspond to values from different estimators, and let V ∈ R k×1 correspond to a vector where each element is the same and corresponds to V π . Let α * ∈ R k×1 be a vector with values of all α * i 's and let α ∈ R k×1 be an estimate of α * . For any estimator V π i , the mean-squared error is denoted by,EQUATIONwhere we make V π i explicitly depend on D n to indicate that the expectation is over the random variables V π i which depend on the sampled data D n . With this formulation, estimating α * can be elicited as a solution to the following constrained optimization problem.Remark 1. Let k i=1 α i = 1, then α * ∈ arg min α∈R k×1 α ⊤ Aα , where A := E V -V V -V ⊤ ∈ R k×k .(2)Using the fact thatEQUATIONNow re-writing the equation above equation 3 in vector form,EQUATIONFinally, simplifying equation 4 furtherEQUATIONTherefore, α that minimizes MSE V π is equivalent to α that minimizes α ⊤ Aα.It is worth highlighting that the optimization problem in Remark 1 is convex in α with linear constraint and thus can be solved by any off-the-shelf solvers (Diamond and Boyd, 2016) .Estimating A: An advantage of Remark 1 is that it provides the objective for estimating α * .Unfortunately, this objective depends on A, and thus on V π , which is not available. Further, observe that A can be decomposed asEQUATION)where the first term corresponds to co-variance between the estimators {V π i } k i=1 and the second term corresponds to the outer product between their biases. One potential approach for approximating A could be to ignore biases. While this could resolve the issue of not requiring access to V, ignoring bias can result in severe underestimation of A, especially in finite-sample settings or when function approximation is used. Further, even if we ignore the biases, it is not immediate how to compute the covariance of various OPE estimators, e.g., FQE.We propose overcoming these challenges by constructing Â ∈ R k×k , an estimate of A ∈ R k×k , using a statistical bootstrapping procedure (Efron and Tibshirani, 1994) . Subsequently, we will use the Â as a plug-in replacement for A to search for the values of α as discussed in Remark 1. There is a rich literature on using bootstrap to estimate bias (Efron, 1990; Efron and Tibshirani, 1994; Hong, 1999; Shi, 2012; Mikusheva, 2013) and variance (Chen, 2017b; Gamero et al., 1998; Shao, 1990; Ghosh et al., 1984; Li and Maddala, 1999) of an estimator that can be leveraged to estimate the terms in equation 6. Instead of estimating the bias and variance individually, we directly use the bootstrap MSE estimate (Chen, 2017a; Williams, 2010; Cao, 1993; Hall, 1990) to approximate A.For bootstrap estimation to work, two key challenges need to be resolved. Even if provided with V π , the regular bootstrap is not guaranteed to yield an MSE estimate which is asymptotic to the true MSE if the distribution has heavy tails (Ghosh et al., 1984) . Furthermore, V π is unknown in the first place.To address these challenges we follow the work by Hall (1990) , where the first issue is resolved by using sub-sampling based bootstrap resamples of size n 1 < n, where n 1 is of a smaller order than n. Therefore, we draw data D * n1 = {τ * 1 , ..., τ * n1 } from D n = {τ 1 , ..., τ n } with replacement. To resolve the second issue, we leverage the MSE estimate by Hall (1990) , and approximate equation 1 usingEQUATIONBuilding upon this direction, we propose using the following estimator Â for A,Â := E D * n 1 V(D * n1 ) -V V(D * n1 ) -V ⊤ D n , ∈ R k×k (8)and we substitute α for A in equation 2 to obtain the weights for combining estimatesEQUATIONThere are two key advantages of the proposed procedure: (1) Bias: it does not require access to the ground truth performance estimates θ * . In the supervised learning setting, a held-out/validation set can provide a way to infer approximation error, However, for the OPE setting there is no such held-out dataset that can be used to obtain reliable estimates of the ground truth performance. (2) Variance: Depending on the choice of the estimator (e.g., FQE), it might not be possible to have a closed-form estimate of the variance, especially when using rich function approximators. Using statistical bootstrapping, OPERA mitigates both these issues and thus is particularly suitable for off-policy evaluation.Estimating α * : We now consider how error in estimating the optimal weight coefficient α * affects the MSE of the resulting estimator V π . Without loss of generality, we consider | V π i | ≤ 1, since we can trivially normalize each estimator's output by |V max |. We now prove that under the mild assumption that the error in the estimated α can be bounded as some function of the dataset size, that we can bound the mean squared error of the resulting value estimate:Theorem 1 (Finite Sample Analysis). Assume given n samples in dataset D, and let ∆ c :=EQUATIONEQUATION)The error of OPERA is divided into two terms. First note that ∆ c is the approximation error: the difference between the true estimate of the policy performance V π and the best estimand OPERA can yield when using the optimal (unknown) α * . If V π can be expressed as a linear combination of the input OPE estimands θi , then there is zero approximation error and ∆ c = 0. The second term in the bound comes from the estimation error due to estimating α * -this arises from the bootstrapping process used for estimating A in equation 8. For this second term we compute an upper bound using a Cauchy-Schwartz inequality. This term decreases as the dataset size n increases. The resulting error depends on the rate at which the estimated α converges to the true α as a function of the dataset size. For example, if λ = 0.5, (a n -.5 rate), the MSE will converge at a n -1 rate in the first term, and if λ = 0.25 (a n -.25 rate) the MSE will converge at a n -0.5 rate in the first term. We provide the full proof in Appendix A.4.We show a full practical implementation of OPERA in Algorithm 1, where we demonstrate how to efficiently construct Â and compute α.For Â obtained from the bootstrap procedure in equation 8 to be an asymptotically accurate estimate of A, (a) a consistent estimator of V is required, and (b) the estimators V need to be smooth. We discuss these points in more detail in Appendix A.3. In the following, we theoretically establish the properties of OPERA on performance improvement and consistency. We also demonstrate how OPERA allows us to interpret each estimator's quality. Further, in Section 6, we empirically study the effectiveness of OPERA even when we do not have any consistent base estimators, or V π i is constructed using deep neural networks.Performance Improvement It would be ideal that the combined estimator V π does not perform worse than any of the base estimators { V π i } n i=1 . As OPERA optimizes for the MSE, we can directly obtain the following desired result.Theorem 2 (Performance improvement). If α = α * , ∀i ∈ {1, ..., k}, MSE( V π ) ≤ MSE( V π i ).However, observe that due to bootstrap approximation, Â may not be equal to A, and thus α may not be equal to α * . Nonetheless, as we will illustrate in Section 6, even in the non-idealized setting OPERA can often achieve MSE better than any of the base estimators {V π i } n i=1 . Consistency Some prior works that deal with multiple OPE estimators assume that there is at least one known consistent estimator (Thomas and Brunskill, 2016) . Under a similar assumption that ∃ V π i : V π i p -→ J(π), OPERA can be made to fall back to the consistent estimator after a large n, such that V π is also consistent, i.e., V π p -→ J(π). Naturally, as V π is a weighted combination of the base estimators { V π i } k i=1 , if all the base estimators provide unreliable estimates, even in the limit of infinite data, then there is not much that can be achieved by weighted combinations of these unreliable estimators.Algorithm 1: OPERA with BootstrapInput: offline RL data Dn; evaluation policy π; a set of OPE estimators [OPE1, OPE2, ..., OPE k ]; number of bootstrap B; a subsample coefficient η ∈ [0, 1]. Output: estimated π performance sOPERA for i ← 1...K do s * i = OPEi(Dn) si = ∅ for j ← 1...B do n1 = |D| η Dj ← Bootstrap (Dn, n1) si = si ∪ OPEi( Dj) end end M ← [s1, s2, ..., sk ] ∈ R K×B M ← [s * 1 , s * 2 , ..., s * k ] ∈ R K×1 δ ← [(s1 -s * 1 , s2 -s * 2 , ..., sk -s * k ] ∈ R K×B A ← 1 B n 1 n δδ ⊤ ∈ R K×K α = arg min α αAα ⊤ s.t. α = 1 sOPERA = α ⊤ M return sOPERAInterpretability With a linear weighted formulation for V π , OPERA allows for the inspection of the assigned weights to which give further insights into the procedure. In Figure 1 we provide a synthetic example to illustrate the impact of bias and variance of the input estimators on the values of α.Consider a case where there are two OPE estimators ( V π 1 and V π 2 ) and two corresponding weights α 1 and α 2 ). Let the true unknown quantity be V π = 0. As we can see below, when both estimators have low bias, but one has higher variance, OPERA assigns a higher magnitude of α i for V π i with a lower variance (Figure1, left). When both estimators have similar variance, and their biases have opposite signs with similar magnitude, then α 2 ≈ α 1 (Figure1, middle left).Interestingly, unlike related prior work (Thomas and Brunskill, 2016) , our optimization procedure in equation 2 does not require α i ≥ 0. Therefore the resulting estimator V π may assign negative weights for some of the estimators. This can be observed for the case when the sign of the Bias( V π 1 ) and Bias( V π 1 ) are the same. In such a case, using a positive and a negative weight can help cancel out the biases of the base estimators, as observed in Figure1 (middle right). When one estimator has no bias and the other has no variance, α values are inversely proportional to their contributions towards the MSE (Figure1, right).  
experiment: 
 We now evaluate OPERA on a number of domains commonly used for offline policy evaluation. Experimental details, when omitted, are presented in the appendix.5.1 Task/Domains Contextual Bandit. We validate the performance of OPERA on the synthetic bandit domain with a 10-dimensional feature space proposed in SLOPE (Su et al., 2020) . This domain illustrates how OPERA compares to an estimator-selection algorithm (SLOPE) that assumes a special structure between the estimators. The true reward is a non-linear neural network function. The reward estimators are parametrized by kernels and the bandwidths are the main hyperparameters. As in their paper, we ran 180 configurations of this simulated environment with different parametrization of the environment. Each configuration is replicated 30 times.Sepsis. This domain is based on a simulator that allows us to model learning treatment options for sepsis patients in ICU (Oberst and Sontag, 2019) . There are 8 actions and a +1/-1/0 reward at the episode end. We experiment with two settings: Sepsis-MDP and Sepsis-POMDP, where some crucial states have been masked. We evaluate 7 different policies: an optimal policy and 6 noised suboptimal policies, which we obtain by adding uniform noise to the optimal policy.Graph. Voloshin et al. (2019) introduced a ToyGraph environment with a horizon length T and an absorbing state x abs = 2T . Rewards can be deterministic or stochastic, with +1 for odd states, -1 for even states plus one based on the penultimate state.We evaluate the considered methods on a short horizon H=4, varying stochasticity of the reward and transitions, and MDP/POMDP settings.D4RL-Gym. D4RL (Fu et al., 2020 ) is an offline RL standardized benchmark designed and commonly used to evaluate the progress of offline RL algorithms. We use 6 datasets (200k samples each) from three Gym environments: Hopper, HalfCheetah, and Walker2d. We use two datasets from each: the medium-replay dataset, which consists of samples from the experience replay buffer, and the medium dataset, which consists of samples collected by the medium-quality policy. We use conservative Q-learning (CQL) (Kumar et al., 2020) , implicit Q-learning (IQL) (Kostrikov et al., 2021) , and TD3 (Fujimoto et al., 2018) . We train 6 policies from these three algorithms with 2 different hyperparameters for the neural network. We selected 2 FQE hyperparameters for each task and picked 2 checkpoints (one early, one late) to obtain 4 estimators to build the OPE ensemble.We compare to using single OPE estimators as well as two new baseline algorithms that combine OPE estimates together. AvgOPE: We can compute a simple average estimator that just outputs the average of all underlying OPE estimates. If an estimator in the ensemble outputs an arbitrarily bad value, this estimator has no implicit mechanism to ignore such an adversarial estimator. BestOPE:We select the OPE estimator that has the smallest estimated MSE. This estimator can be better than AvgOPE as it can ignore bad estimators. In addition, in different domains, we compare to other OPE strategies such as BVFT (Batch Value Function Tournament): making pariwise comparisons between different Q-function estimators with the BVFT-loss (Xie and Jiang, 2021; Zhang and Jiang, 2021) . SLOPE: an estimator selection method that based on Lepski's method, assuming the estimators forming an order of decreasing variance and increasing bias (Yuan et al., 2021) . DR (Doubly Robust): a semi-parametric estimator that combines the IS estimator and FQE estimator to have an unbiased (Jiang and Li, 2016; Gottesman et al., 2019; Farajtabar et al., 2018) . All of these methods place explicit constraints on the type of OPE estimator to include. We report the result in Figure 2 . Figure 2a shows that as the dataset size grows, the bootstrapping procedure employed by OPERA can quickly estimate the performance each estimator and compute a weighted score that is better than a single estimator. In the ultra-small data regime, OPERA is worse than single-estimator selection style algorithms, mainly because OPERA does not explicitly reject estimators. We can add an additional procedure to reject bad estimators and then combine the rest with OPERA, using a rejection algorithm by Lee et al. (2022) .Sepsis We report the results in Table 1 . In this domain, OPERA is able to produce an estimate, on average, across many policies with different degrees of optimality, that matches and exceeds the best estimator in the ensemble. Even though in three out of four tasks, OPERA MSE is close to the MSE of the best estimator in the ensemble, in the MDP (N=200) setup, OPERA is able to get a significantly lower MSE than any of the estimators in the ensemble, suggesting a future direction of carefully choosing a set of weak estimators to put in the ensemble to obtain a strong estimator.Graph We report the graph domain result in Appendix A.9 and in Table 5 . We find a similar result to the Sepsis domain. OPERA is able to outperform AvgOPE and BestOPE in different setups.We report the results in Table 2 . We choose this domain because, in continuous control tasks, the horizon is often very long. Many OPE estimators that rely on short-horizon or discrete actions will not be able to extend to this domain. A popular OPE choice is FQE with function approximation, but it is difficult to determine hyperparameters like early stopping, network architecture, and learning rate. We can see that even though FQE used in D4RL is not a consistent estimator and does not satisfy OPERA's theoretical assumption, we are still able to combine the estimations to reach an aggregate estimate with lower MSE.6 Discussion: Different MSE Estimation Strategies 6.1 Estimating MSE with MAGIC Part of our algorithm implicitly involves estimating the MSE of each OPE estimator. In our algorithm we do this using bootstrapping but other alternatives are possible. For example, prior work by (Thomas and Brunskill, 2016) provided a way to estimate the bias and variance of an OPE estimator are computed through per-trajectory OPE scores and used this as part of their MAGIC estimator. However, this method cannot estimate the MSE of self-normalizing estimators (such as WIS) or minimax-style estimators (such as any estimator in the DICE family (Yang et al., 2020) ). We denote this estimator as MSE MAGIC (V π ) and now explore how our approach of using boostrapping compares to this method in an illustrative setting.In particular, we consider estimating the MSE of the FQE and IS estimands on the Sepsis-POMDP and Sepsis-MDP domains. MAGIC estimates the bias of an OPE as the distance between the OPE We use a percentile bootstrap to construct a 50% confidence interval CI around WIS.(V π ) MSEMAGIC(V π ) MSE(V π ) MSE(V π ) MSEMAGIC(V π ) MSE(V π ) IS 0.Our bootstrap MSE(V π ) procedure is able to provide a consistently more accurate estimate of the true MSE of the FQE estimate compared to MSE MAGIC (V π ) and a comparable or better one for the IS estimate (see Table 3 ). We suspect that this is due to MAGIC's unique way of computing bias. Specifically, MAGIC computes bias by comparing two estimates (in this case, FQE and the uppper/lower bounds on WIS) which may significantly misestimate the bias in some situations. 
variants of opera with different strategies: 
 We now explore two alternative strategies to estimate the MSE of each estimator. The first strategy is, instead of using the estimator's own score as the centering variable V, we use a consistent and unbiased estimator's score as V. We call this OPERA-IS. Another strategy is to use the idea from (Thomas and Brunskill, 2016 )'s MAGIC algorithm, where the bias estimate of each estimator compares the estimand to the upper or lower confidence bound of a weighted importance sampling estimator, as above. We call this OPERA-MAGIC. These are two new variants of our OPERA algorithm that will may lead to learning different α weights and producing different linearly stacked estimates. We use these two new methods, and compute the true MSE of the resulting stacked estimate, compared to OPERA and other baseline estimates. We use the Sepsis domains to illustrate the results and use as input IS, WIS and FQE OPE estimates.The true MSE of the resulting estimates are presented in Table 4 . While using an unbiased consistent estimator as the centering variable can help further improve OPERA's estimate, sometimes it also hurts the performance (MDP N=1000 setting). OPERA-MAGIC however almost always performs worse than the best estimator in the ensemble. This suggests that when combining OPE scores this bound on the bias, which will provide a distorted estimate of the estimator bias especially in low data regimes, can lead to learning less effective weightings of the input OPE estimands. OPERA remains a solid option across all settings presented in the table. 
conclusion: 
 We propose a novel offline policy evaluation algorithm, OPERA, that leverages ideas from stack generalization to combine many OPE estimators to produce a single estimate that achieves a lower MSE. Though such stacked generalization / meta-learning has been frequently used to create better estimates from ensembles of input methods in supervised learning, to our knowledge this is the first time it has been explored in offline reinforcement learning. One challenge is that unlike in supervised learning, we do not have ground truth labels for offline policy learning. OPERA uses bootstrapping to estimate the MSE for each OPE estimator in order to find a set of weights to blend each OPE's estimate. We provide a finite sample analysis of OPERA's performance under mild assumptions, and demonstrate that OPERA provides notably more accurate offline policy evaluation estimates compared to prior methods in benchmark bandit tasks and offline RL tasks, including a Sepsis simulator and the D4RL settings. There are many interesting directions for future work, including using more complicated meta-aggregators.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report standard error of the mean (SEM) in Figure 2 . We do not conduct any statistical test in this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We report it in Appendix Section A.9.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: There are no crowdsourcing or human subject studies conducted in this paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: he paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We describe the OPERA framework as a two-stage process in Figure 3 . The first stage involves using black-box statistical methods (such as Bootstrap) to estimate the quality of each OPE estimator. The information is then used to estimate a weight α through a convex optimization objective in Eq 9. At stage two, we use the learned α to combine each estimator's score to output the OPERA score. D D * OP E 1 OP E 2 OP E k Â α Bootstrap D OP E 1 OP E 2 OP E kWe provide a list of important notations used in this paper.We use n, n 1 to refer to the number of trajectories in the dataset. kWe use k to refer to the number of estimators in the ensemble.D n D n = {τ i } n i=1 = {s i , a i , s ′ i , r i } n i=1be the trajectories sampled from π on an MDP. D * n1 A bootstrapped sample from D n with n 1 data points, n 1 < n.The cross-estimator bias variance matrix of k estimators. A ∈ R k×k ÂThe cross-estimator bias variance matrix of k estimators usingD * n1 . Â ∈ R k×k α Coefficient for k estimators. α ∈ R k×1 α *Coefficient for k estimators for solving α * ∈ arg min α∈R k×1 α ⊤ Aα. αCoefficient for k estimators for solving α ∈ arg min α∈R k×1 α ⊤ Âα.True performance of the policy π, V π = J(π).V π i Let i-th OPE estimate of π's performance be V π i (D n ) = Vi (π, D n ). We use V π i = V π i (D n ) where D n is the full offline dataset. V π (α)Weighted average of k estimators with given α:V π (α) := k i=1 α i V π i ∈ R V π OPERA's estimated performance of π. V π = V π (α). V π A shorthand notation for V π = V π (α * ). MSE( V π i )The mean-squared error of the i-th OPE estimator.Based on how we constructed A, we have MSE(V π i ) = A i,i . MSE( V π ) The mean-squared error of V π (OPERA score). MSE( V π ) ≤ ∆ α + ∆ c . MSE( V π i )The estimated mean-squared error of the i-th estimator using bootstrapped dataset D * n1 . Based on how we constructed Â, we have MSE( V π i ) = Âi,i .The change in mean-squared error of OPERA, V π , from using dataset D n .∆ c := E Dn V π -V π 2 .The change in mean-squared error of OPERA, V π , from using sub-optimal α.∆ α := E Dn ( V π -V π ) 2 .In this section, we provide a high-level discussion of the bootstrap procedure and its asymptotic validity. We refer the readers to the works by (Cao, 1993; Hall, 1990) for a more fine-grained analysis and convergence rates when estimating MSE using statistical bootstrap. Individual treatment of bias (Efron, 1990; Efron and Tibshirani, 1994; Hong, 1999; Shi, 2012; Mikusheva, 2013) and variance (Chen, 2017b; Gamero et al., 1998; Shao, 1990; Ghosh et al., 1984; Li and Maddala, 1999) can also be found.In the following, we will discuss the consistency of Â estimated using bootstrap, Âi.j -A i,j a.s.-→ 0.(EQUATIONTowards this goal, we will consider the following conditions imposed on the set of the base estimators{ V π i } k i=1 , • ∀i, V π i is uniformly bounded. • ∀i, V π i a.s. -→ c i . • ∀i, V π i is smooth with respect to data distribution. • ∃ V π k : V π k a.s. -→ c k = V π .Recall from equation 2,EQUATION)EQUATION) Let X n := V π i -E[ V π i ] and Y n := V π j -E[ V π j ] . As V π i a.s.-→ c i and V π i is uniformly bounded, using (Thomas and Brunskill, 2016 , Lemma 2), we haveE[ V π i ]a.s.-→ c i . Similarly, we haveE[ V π j ]a.s.-→ c j as V π j a.s.-→ c j . Then using continuous mapping theorem,EQUATION)Now using (Thomas and Brunskill, 2016, Lemma 2) ,EQUATIONSimilarly,E[ V π i ] -V π E[ V π j ] -V π a.s. -→ (c i -V π )(c j -V π ) (20)Therefore, using equation 19 and equation 20,EQUATION)Now we consider the asymptotic property of the bootstrap estimate Â of A.Âi,j = E D * n 1 |Dn V π i (D * n1 ) -V π k V π j (D * n1 ) -V π k (22)where V π k is known to be a consistent estimator, i.e., V π k a.s.-→ V π . Here, V π k could be the WIS or IS or doubly-robust estimators that are known to provide consistent estimates of V π = J(π). For brevity, we drop the conditional notation on the subscript, and write equation 22 as,EQUATION)+ E D * n 1 V π i (D * n1 ) -V π k E D * n 1 V π j (D * n1 ) -V π k (28) Let X n1 := V π i (D * n1 ) -E D * n 1 V π i (D * n1 ) and Y n1 := V π j (D * n1 ) -E D * n 1 V π j (D * n1) . As the empirical distribution D * n1 converges to the population distribution, i.e., D n a.s.-→ D, the resampled distribution D * n1 from D n also converges to the population distribution, i.e., D * n1 a.s.-→ D. Therefore, when the estimator V π i (D * n1 ) is smooth, using the continuous mapping theorem, ∀i,EQUATIONTherefore, similar to before,EQUATIONand subsequently,E D * n 1 [X n1 Y n1 ]a.s.-→ 0.(31)Further, as V π k a.s.-→ V π ,EQUATION)Therefore,EQUATION)Using equation 31 and equation 34 in equation 28, Âi,j a.s.EQUATION)Finally, combining equation 21 and equation 35,EQUATION)which gives the desired result. It is worth highlighting that, theoretically, this result relies upon assumptions that the base estimators satisfy regularity conditions and are consistent. In practice, such assumptions might not hold (for e.g., when using FQE to do policy evaluation if the function approximation is under-parameterized). Nonetheless, in Section 6 we empirically illustrate that even when these assumptions are not directly satisfied, OPERA can be effective.Without loss of generality, let ∀π ∈ Π, |J(π)| ≤ 1, such that we can always consider ∀i, | V π i | ≤ 1 (this can be trivially achieved by normalizing each estimator's output by |V max |). Let V π be a weighted sum of V π i with α ⋆ , where the total number of estimators in the ensemble is k.In the following, we show how the error in estimating the optimal weight coefficients α * affects the MSE of the resulting estimator V π . Given { V π i } k i=1 , we assume that Â obtained uisng the bootstrap procedure of OPERA will produce α via Equation 9(and a resulting estimate of V π ). In contrast, using A would have produced α ⋆ (and a resulting estimate of V π ). To provide a finite sample characterization of OPERA's mean squared error, consider the setting where given n samples in dataset D, there exists λ > 0, such thatEQUATION)where the expectation is over the randomness due to data D n that governs the estimates V π i and thus also the weights α and α * used to combines these estimates. We now provide a proof of Theorem 1. To bound the MSE of OPERA's estimate V π observe that,MSE( V π ) := E Dn V π -V π 2 (38) = E Dn V π -V π + V π -V π 2 (39) ≤ E Dn ( V π -V π ) 2 =∆α + E Dn V π -V π 2 =∆c (40)We isolate the error of V π into two terms: ∆ α and ∆ c . ∆ c is the gap between the best estimate OPERA can give with α ⋆ and the true estimate of the policy performance V π . If V π can be expressed as a linear combination of V π i , then ∆ c = 0. ∆ α is the term we want to further analyze because it depends on the difference between α and α * .EQUATIONwhere the last inequality follows from Cauchy-Schwarz inequality. Now by using the fact that | θi | ≤ 1 and by plugging equation 37 into equation 44:EQUATIONTherefore, combining equation 40 and equation 47,EQUATIONThis bound factors the MSE using the term ∆ c , which is the best a linear combination of estimators can do. Notice that ∆ c ≤ min i MSE( V π i ), as the best linear combination of the estimators can at least achieve the MSE of the best estimator, by assigning weight of 1 to the best estimator and 0 to the rest. Therefore, the rate of decay of ∆ c is bounded above by the rate of convergence of the best estimator in our ensemble.The other term k2 /n 2λ in equation 48 results due to the error in estimating α * because of the bootstrapping process used for estimating Â of A in equation 8. This is dependent on the number of estimators k -as we include more estimators in our ensemble, the combination weights α ∈ R k that need to be estimated becomes higher dimensional, thereby introducing more errors. However, the overall term decreases as the dataset size n increases.A.5 Proofs on Properties of OPERA A.5.1 Invariance In the following, we illustrate an important property of OPERA, that the resulting combined estimate V π is invariant to the addition of redundant copies of the base estimatorsEQUATIONProof. We prove this by contradiction. Recall that α ∈ R k are the weights that minimize the bootstrap estimate of MSE of V π consisting of k estimators.EQUATIONAsEQUATIONFinally, as β * ∈ R k+1 is the weight that minimizes the bootstrap estimate of MSE of V π β . Now, if equation 50 < equation 52, then one could assign β * i := αi for i ∈ {1, ..., k}, and β * k+1 = 0 to make equation 52 = equation 50. Further, notice that as both α and β * are within the same feasible set of solutions, the above reassignment is also within the feasible set of solutions. Similarly, if equation 50 > equation 52, then one could assign αi := β * i for i ∈ {1, ..., k -1}, and αk = β * k + β * K+1 to make equation 52 = equation 50. Hence, if equation 50 does not equal equation 52, then either α or β * is not optimal and that would be a contradiction. This ensures that MSE(V π β ) = MSE( V π ).As Â is positive definite, it implies that equation 9 is strictly convex with linear constraints. Thus the minimizer α of equation 9 is unique, and V π β = V π . Note that due to redundancy, B will not be PD despite Â being PD. This would imply that there can be multiple values ofEQUATIONProof. With a slight overload of notation 2 , we make the dependency of weights α explicit and letV π (α) = k i=1 α i V π i . Let MSE( V π (α)) := α ⊤ Aα,where A is defined as in equation 2.Now from equation 1 and equation 2, we know that forEQUATIONTherefore, for any λ ∈ R k×1 such thatEQUATIONNotice that for e i := [0, 0, .., 1, .., 0], where there is a 1 in the i th position and zero otherwise,EQUATIONTherefore, asV π = V π (α), we have the desired result that ∀i ∈ {1, ..., k}, MSE( V π ) ≤ MSE( V π i ).A.6 Empirical Properties of OPERA In Figure 4 2019) introduced a ToyGraph environment with a horizon length T and an absorbing state x abs = 2T . Rewards can be deterministic or stochastic, with +1 for odd states, -1 for even states plus one based on the penultimate state.We evaluate the considered methods on a short horizon H=4, varying stochasticity of the reward and transitions, and MDP/POMDP settings. We evaluate a single policy in this domain. We report the results in Table 5 . In three out of four setups, OPERA is able to produce an estimate that has a lower MSE compared to BestOPE and AvgOPE and is lower than the estimators used in the ensemble. In the deterministic POMDP setting, the IS estimate is significantly off, and OPERA is worse than BestOPE. This provides an important insight into the strengths and limitations of our particular procedure. In small tabular POMDPs, WIS can be quite good. Here, the error in the MSE estimation means that OPERA inaccurately balances the two estimators instead of relying only on WIS. We note that BestOPE does not match the performance of WIS in the ensemble. This is because the error in MSE estimation makes BestOPE erroneously choose the wrong estimator in some trials, resulting in a larger MSE. We note that on three other setups, the MSE estimation is fairly accurate. Therefore, OPERA is able to get a lower MSE than any of the OPEs in the ensemble.These results suggest that it is important to consider the tradeoff between bootstrap estimation error and the benefit from combined estimators. When we compare the estimation quality over many policies (Sepsis and D4RL), OPERA does well, but for a particular policy, it might not outperform other estimators. This is an interesting area for future work: if and when it is possible for such a meta-algorithm to provably match the performance of any individual estimator without further assumptions.A.8 D4RL Experiment Setup D4RL (Fu et al., 2020) is an offline RL standardized benchmark designed and commonly used to evaluate the progress of offline RL algorithms. We use 6 datasets of different quality from three environments: Hopper, HalfCheetah, and Walker2d. We choose the medium and mediumreplay datasets. Medium dataset has 200k samples from a policy trained to approximately 1/3 the performance of a policy trained to completion with SAC. Medium-replay dataset takes the transitions stored in the experience replay buffer of policy -this dataset can be thought of as a dataset sampled by a mixture of policies.Policy Training We train 6 policies from these three algorithms with 2 different hyperparameters for the neural network, Q-learning (CQL) (Kumar et al., 2020) , implicit Q-learning (Kostrikov et al., 2021) , and TD3+BC (Fujimoto et al., 2018) . We initialize all neural networks (including both actor and critics, if the algorithm uses both) with the hidden dimensions of [256, 256, 256] . We train with a batch size of 512, with Adam Optimizer. We train for 100 epochs on each dataset. We only change one important hyperparameter per algorithm. We report the discounted return of each policy in Table 11 ,10,12. We report these scores because they are the prediction target of the FQE algorithm.We report the un-discounted return of each policy in out of the normal range, the patient dies. If all vital signs are within normal ranges and the treatments are all stopped, the patient is discharged. The reward function is +1 if a patient is discharged, -1 if a patient dies, and 0 otherwise. We truncate the trajectory to 20 actions (H=20). For this simulator, early termination means we don't get to observe a positive or negative return on the patient.We follow the process described by Oberst and Sontag (2019) to marginalize an optimal policy's action over 2 states: glucose level and whether the patient has diabetes. This creates the Sepsis-POMDP environment. We sample 200 and 1000 patients (trajectories) from Sepsis-POMDP environment with the optimal policy that has 5% chance of taking a random action. We also sample trajectories from the original MDP using the same policy; we call this the Sepsis-MDP environment.We use tabular FQE. Therefore, there is no representation mismatch. We additionally use cross-fitting, a form of procedure commonly used in causal inference (Chernozhukov et al., 2016) . Cross-fitting is a sample-splitting procedure where we swap the roles of main and auxiliary samples to obtain multiple estimates and then average the results. The main goal of cross-fitting is to reduce overfitting. We notice significant performance improvement of our FQE estimator after using cross-fitting. We present the RMSE of each of our trained FQE estimator in Table 6 .For the graph environment, we set the horizon H=4, with either POMDP or MDP and ablate on the stochasticity of transition and reward function. The optimal policy for the Graph domain is simply the policy that chooses action 0. All the experiments reported have 512 trajectories.The training was done on a small cluster of 6 servers, each with 16GB RAM, and 4-8 GPUs of Nvidia A5000. D4RL was the most computationally expensive experiment.NeurIPS Paper ChecklistEmails: {anie,ychandak,ebrun}@cs.stanford.edu. 3 Work done while at Stanford. 38th Conference on Neural Information Processing Systems (NeurIPS2024).Note that in Section A.4, we used V π to denote V π (α * ). Here we make this dependence explicit. 


Decision: This paper presents OPERA, a novel and estimator-agnostic approach to offline policy evaluation that significantly improves accuracy by adaptively blending multiple OPE estimators. The innovative use of bootstrapping to estimate the mean squared error of different estimator weightings, and its subsequent optimization as a convex problem, demonstrates a sound theoretical foundation. Comprehensive empirical evaluation across domains like healthcare and robotics showcases OPERA's superior performance in selecting higher-performing policies compared to existing approaches. This work represents a valuable contribution to the field of offline reinforcement learning, and thus I recommend acceptance of this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces OPERA, an algorithm for offline policy evaluation (OPE) in reinforcement learning (RL). OPERA addresses the challenge of selecting the best OPE estimator by adaptively combining multiple estimators using a statistical procedure that optimizes their weights to minimize mean squared error (MSE). The authors prove OPERA's theoretical consistency and desirable properties, ensuring it is at least as accurate as any individual estimator. The algorithm employs a bootstrapping method to estimate MSE, circumventing the need for direct access to true performance measures.
strength_weakness: strength_weakness: **Assumption of Consistent Estimators**
The theoretical guarantees provided for OPERA rely on the assumption that at least one of the base estimators is consistent. However, in practice, this assumption may not always hold, especially in complex or noisy environments where all available estimators could be biased or inconsistent. The paper could benefit from a discussion on how OPERA performs under such conditions and whether there are ways to relax this assumption while still maintaining acceptable performance. For example, could the authors design an experiment where the state, action and/or reward are severely imbalanced, which leads to insufficient data coverage and approximation error to behaviour policy? A practical reflection can be found in a critical study of real-world ICU treatment [1].

[1] Luo, Zhiyao, et al. "Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination." Forty-first International Conference on Machine Learning.
questions: questions: 1.The theoretical guarantees of OPERA rely on the assumption that at least one base estimator is consistent. How does OPERA perform when this assumption does not hold? Are there any mechanisms within OPERA to detect and handle inconsistent base estimators?

2. What practical considerations should be taken into account when implementing OPERA in real-world scenarios? Are there specific guidelines for choosing the initial set of estimators or criteria for including new estimators in the ensemble?
limitations: limitations: This paper seems to lack a section for limitation. I recommend adding 1 short paragraph to the conclusion section to summarize the border limitations and social impact.
paper_summary: paper_summary: The authors propose a novel offline policy evaluation algorithm, that linearly blends the estimates from many OPE estimators to produce a combined estimate that achieves a lower MSE.
strength_weakness: strength_weakness: The authors propose to estimate MSE of \hat{V}_i by estimating \pi using bootstrapped D_n and calculating the squared error from an estimate of \pi done by \hat{V}_i using the original D_n dataset. I think this underestimates bias. For example, if \hat{V}_i returns a constant for any dataset and policy, then its MSE would be 0, and I assume \hat{\alpha}_i would be 1, meaning the ensemble reward would correspond to this constant. I appreciate the variants of OPERA presented in 6.2 that partially address this.

Minor:
* [L114] D_n is defined both as 1 to n and 0 to n
* [L125] \theta_* is undefined
* [L136] Eq. 4 is referred although it is not labeled when defined (and many others)
* [L169] \theta_* is still undefined
* [L189] The word equation is repeated twice, I recommend using the LaTeX package cleveref
questions: questions: Considering the first paragraph in weaknesses, can you discuss whether OPERA systematically favors biased estimators? 

Recent work of Cief et al. (2024) provides a new way of estimating MSE, which can potentially improve OPERA.  As this was published after the NeurIPS deadline, I do not consider it an issue.

Cief, Matej, Michal Kompan, and Branislav Kveton. “Cross-Validated Off-Policy Evaluation.” arXiv, May 24, 2024. http://arxiv.org/abs/2405.15332.
limitations: limitations: The authors provide a thorough discussion on the limitations in Section 6 and Appendix A.7.
paper_summary: paper_summary: The paper "OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators" deals with the challenge of evaluating new decision-making policies using past data, which is vital in areas like healthcare and education where mistakes can be costly.
The authors introduce OPERA, a new algorithm that improves the accuracy of policy evaluation by combining several existing estimators instead of relying on just one. This approach makes the evaluations more reliable.
They provide a solid theoretical basis for OPERA, showing that it effectively minimizes errors and meets key evaluation criteria. A standout feature is using bootstrapping to estimate the mean squared error (MSE) of each estimator, which helps in finding the best way to combine them.
The paper demonstrates OPERA's effectiveness through tests in various fields, such as healthcare and robotics. It consistently outperforms other methods by selecting better policies and achieving lower errors.
One of the key contributions is OPERA's flexibility. It can be used in many applications without needing strict conditions on the input estimators, making it a versatile tool for different offline reinforcement learning tasks.
In summary, the paper presents a new method for policy evaluation that combines multiple estimators to enhance accuracy and reliability, backed by strong theoretical and practical evidence.
strength_weakness: strength_weakness: While the theoretical and empirical aspects are well-covered, the paper could really use a more detailed discussion on how to implement OPERA in practice. Offering guidelines or best practices for using OPERA in different situations would be very helpful for practitioners.
questions: questions: The paper primarily compares OPERA with traditional methods. Have you considered comparing OPERA with more recent state-of-the-art models? Including such comparisons could provide a clearer picture of OPERA's competitive edge.
limitations: limitations: The authors have done a great job addressing the limitations of their work. They recognize the challenges of combining multiple estimators and provide a solid theoretical foundation to back up their approach. They also discuss using bootstrapping to tackle issues like bias and variance, which is a key part of their method.
However, it would be helpful to add a separate section that dives into potential edge cases and limitations in more detail. This could cover situations where OPERA might have difficulties, such as dealing with very noisy or incomplete data, and practical challenges users might face during implementation.
paper_summary: paper_summary: The paper addresses the challenge of evaluating new sequential decision-making policies using OPE techniques. The authors propose a new algorithm, OPERA, which adaptively blends multiple OPE estimators to improve the accuracy of policy performance estimates without relying on explicit selection methods. This approach leverages bootstrapping to estimate the mean squared error of different estimator weightings and optimizes it as a convex problem. The proposed method is shown to be consistent and demonstrates superior performance in selecting higher-performing policies in healthcare and robotics compared to existing approaches. This contributes to a more general-purpose, estimator-agnostic framework for offline reinforcement learning.
strength_weakness: strength_weakness: There still remains an important question of how to construct an appropriate set of estimators before performing the proposed ensemble algorithm 


I still do not get an intuition of if the proposed method solves fundamentally an easier problem than standard OPE. I mean, if we know the MSE, then it is really straightforward that the proposed algorithm works well, but we need to estimate it somehow, which includes the true policy value, which is the estimand of OPE. Therefore, intuitively, MSE estimation has the same difficulty as OPE, but the proposed algorithm works better than OPE. I’d like to know in what sense the proposed algorithm makes OPE easier.


In theorem 1, it seems that the error rate of \alpha estimation is given, which is not guaranteed, if my understanding is correct.
questions: questions: How should we construct the set of estimators to perform the proposed algorithm? Can the authors propose any general guideline? 


What would be the intuition of the better effectiveness of the proposed method against standard OPE? Some reasonable accuracy of the proposed method means that we can estimate the MSE somewhat accurately, meaning that we can do OPE similarly accurately, in my intuition.


In theorem 1, the error rate of \alpha estimation is not guaranteed, right?


When combining different estimators, can we consider more complex functions such as polynomials or more general function classes rather than just linear combinations as done in the paper?
limitations: limitations: Yes

Comments: comment: Thank you to the authors for the useful clarification. I will maintain my positive assessment.
I have read the Rebuttal, thanks for answering my question.
Thank you for answering my question. After reviewing other reviews and their discussions, I decided to keep my score and vote for the paper's acceptance.