Simultaneous Statistical Inference for Off-Policy Evaluation in Reinforcement Learning
Authors: Tianpai Luo
Xinyuan Fan
Weichi Wu
Venue: NeurIPS 2025 poster ()
Keywords: off-policy evaluation
simultaneous confidence region
convex Gaussian approximation
bootstrap
reinforcement learning
Primary area: reinforcement_learning
TL;DR: This work presents the first asymptotically correct simultaneous confidence region for off-policy evaluation in reinforcement learning.

Abstract: This work presents the first theoretically justified simultaneous inference framework for off-policy evaluation (OPE). In contrast to existing methods that focus on point estimates or pointwise confidence intervals (CIs), the new framework quantifies global uncertainty across an infinite or continuous initial state space, offering valid inference over the entire state space. Our method leverages sieve-based Q-function estimation and (high-dimensional) Gaussian approximation techniques over convex regions, which further motivates a new multiplier bootstrap algorithm for constructing asymptotically correct simultaneous confidence regions (SCRs). The widths of the SCRs exceed those of the pointwise CIs by only a logarithmic factor, indicating that our procedure is nearly optimal in terms of efficiency. The effectiveness of the proposed approach is demonstrated through simulations and analysis of the OhioT1DM dataset.

Simultaneous Statistical Inference for Off-Policy Evaluation in Reinforcement Learning
Tianpai Luo" Xinyuan Fan" Weichi Wu
Department of Statistics and Data Science
Tsinghua University Beijing 100084, China
(1tp21, fxy22) Smails. tsinghua. edu en, wuweichi @tsinghua.edu.cn
Abstract
This work presents the first theoretically justified simultaneous inference framework for off-policy evaluation (OPE). In contrast to existing methods that focus on point estimates or pointwise confidence intervals (CIs), the new framework quantifies global uncertainty across infinite continuous initial state space, offering valid inference over the entire state space. Our method leverages sieve-basee Q-function estimation and (high-dimensional) Gaussian approximation techniques over convex regions, which further motivates a new multiplier bootstrap algorithm for constructing asymptotically correct simultaneous confidence regions (SCRs). The widths of the SCRs exceed those of the pointwise CIs by only a logarithmic factor, indicating that our procedure is nearly optimal in terms of efficiency. The effectiveness of the proposed approach is demonstrated through simulations and analysis the OhioT DM dataset.
Introduction
Off-policy evaluation (OPE) is a fundamental topic in reinforcement learning (RL), aiming to assess the performance of target policy using data collected under different behavior policy, before adopting the target policy in practice. For this purpose, much effort has been made on the statistical inference of the value of the target policy, including obtaining an accurate estimate and valid confidence intervals for quantifying the uncertainty. See Uchara et al. 2022) for a comprehensive
review.
In many real-world applications, such as healthcare (Murphy et al. 2001 Matsouaka et al. 2014, Shi
Lu & Song 2020), ridesharing (Xu al. (2018)), and autonomous driving (Sallab al. (2017) ). is often necessary to evaluate a policy across a range of initial states. For instance, in the Ohio'l DD dataset (Marling & Bunescul (2000), each patient begins in different state of continuous glucose monitoring (CGM) blood glucose levels and self-reported life events. The evaluation of a potentially effective off-policy must be conducted without direct deployment, and it requires quantification of uncertainties across multiple initial states. However, constructing pointwise confidence intervals (CIs) for each state with Bonferroni correction inflates the overall significance level, which is well-known as the multiple testing problem. The inflation becomes especially pronounced when the state space is infinite, for example, R. To address this, we consider the following question:
Is possible simultaneously quantify the uncertainty of off eff-policy value estimators over the entire state space?
'Equal contribution.
1 'Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this paper, we provide an affirmative answer. Specifically, this can be achieved by constructing simultancous confidence regions (SCRs) that cover the whole value functions at a given significance level.
1.1 Related work
Existing methods for statistical inference in reinforcement learning can be categorized into three categories: (i) Direct estimation: This approach constructs CIs by directly learning the system dynamics or Q-function under the target policy. The estimations include kernel-based Q-function methods (Feng et al. (2020)), batch learning (Le al. 2219)), and sieve estimation methods (Chen (2007), Shi et al. 2021) I) or equivalently. called linear function approximation [Sutton et al. (2008), Lagoudakis (0077). )) (ii) Importance sampling: This method re-weights the observed rewards with the density ratio of the target and behavior policies. Bootstrap methods, concentration inequalities, and empirical likelihood-based methods have been applied construct CIs for importance sampling estimators (Thomas et al. (2015). Hanna et al. (2017), Dai et al. )2 (iii) Double reinforcement learning (DRL): This framework combines the first two for more robust and efficient value evaluation (Jiang & Li 6) Thomas & Brunskill (2016), Jiang & Huang (2020)) For instance, |Kallus & Uchara (2022) achieves consistent DRL estimation the value function and computes a marginalized
density ratio to build CI.
While existing methods largely focus on point estimation pointwise intervals, approaches tailored for a large number of states remain limited. Works such as Duan et al. (2020) and [hi et al. (2021) advanced this direction by constructing confidence intervals not only pointwise (for the value at given state) but also for integrated value functions under a known reference distribution of initial states,
However, both leave important gaps. The asymptotic theory in [Shi al. (202T) establishes validity in large samples but does not provide non asymptotic error control. In contrast, Duan et al. (2020) supports finite-sample inference, but its confidence bounds are conservative. Neither framework provides the simultancous inference that is uniformly valid across all states. Our work addresses these gaps by developing a framework that enables distribution- for the value function at any state simultaneously, while also delivering saiple guarantees through the non -symptotioiicc bound obtained from the Gaussian approximation.
asymptotically correct inference
1.2 Contributions
In this paper, we propose a novel framework for constructing asymptotically correct SCRs for the OPE, To the best of our knowledge, this is the first work to introduce a simultaneous statistical inference framework in policy evaluation of RL. Our method shares similar spirit to Q-learning, which estimates the state-action value function (Q-function) under the target policy. The estimation of Q- function is achieved by the linear function approximation (i.e., sieve method). Our key contributions are as follows:
We establish a convex Gaussian approximation result for the sieve estimation of the Q- function. This approximation enables us to characterize the distribution of the sieve estimator over arbitrary convex sets, thereby facilitating simultaneous inference when the initial state is not fixed. Moreover, the convex Gaussian approximation theory only requires the number of trajectories or decision points to diverge, which naturally allows the infinite-horize setting. Our theoretical results are built upon non-asymptotic results, which do not involve any convergence from extreme value theory in statistics.
2. Based on the convex Gaussian approximation, we construct an asymptotically correct SCR whose stochastic behavior is depicted by the maxima of Gaussian random field. The width of the SCR exceeds that of the pointwise confidence intervals only by a logarithmic factor.
3. To implement our methodology, we develop a multiplier bootstrap algorithm for constructing SCRs, which avoids the need to estimate the limiting joint distribution of policy value estimators across different initial states. We further assess the performance of the proposed simultaneous inference framework through both numerical simulations and real data analysis.
The rest of the article is organized as follows. We introduce the model setup in Section 2 In Section 3 we present the construction of SCR based on sieve estimation, convex Gaussian approximation, and the bootstrap algorithm. Simulation studies and real data analysis on the OhioTIDM dataset are

---

conducted in Section 9 Finally, we conclude our paper in Section All proofs, along with additional simulation results, are given in the supplementary material.
2 Preliminaries
Consider a Markov Decision Process (MDP) represented by the tuple M = (S, A. Rt. where S denotes the state space, A the action space, and R x A IK the reward function. In this paper, we assume that S is a subspace of I2 with a fixed dimension d, and A is the discrete set denote the state-action-rewant triplet collected at time t. In the MDP framework, the following Markov assumption is imposed:
{0, I.... m 1} with a fixed cardinality m. Let (S Ao.
Ro,s) A..2,.o.)
P(So,t+1 E So, = 8, Ao. = a; (So.* (Ao,* (Ro,
= P(B) 8, s,a),
(2.1)
where P denotes the transition probability kernel, which is time-homeen tine-homogeneous. Additionally, we assume that the conditional mean of the reward R depends only on the current state and action, i.e.,
B(Ro,,|So:: s, Ao, a, (So Ao, Ro,k)
E ( So, s, A,, a) r(s (2.2)
where r(.) is reward function r S x A + R. We note that if the reward Ro.: is deterministic
function of So.r, Ao. S0.c-1. condition (.2g follows directly from (2.T) Both (2.1) and (.2 are
standard assumptions in the reinforcement learning literature.
Let (-I--) denote a policy which satisfies [[ass) 2 0 for all E E, and EAEA (als)) for any 8 E S. The objective of RL can then be expressed through the following value function:
V(T;))= Σ "EE (Ro,c = s),
(2.3)
120
where the expectation ET is taken under the rule that actions selected according the policy and r refers to a given discount factor, S 1.
In this paper, we consider an offline setting where data is pre-collected and can be written as (Ry,,, A.,,, Si,,, S1,+1), 0 S t S Tú 1 i
where n denotes the number of trajectories, and T3 is the termination time of the i-th trajectory. For the sake of brevity, we assume T; T, I,... n, and the sample size is denoted as N nl'. Our framework only requires that either T' or diverges (namely, N 20).
3 Simultaneous inference for OPE
In this paper, shall construct the asymptotically correct SCR for the OPE significance level 1 - a, a E 1) via finding Ca (which might depend on N) such that
lim P Cc,L()) < ((T;s) s ((T;s) C. =1 1 1 (3.1) N-000
where (T;; s) is the estimated policy values and L(s) is scaling factor related to the covariance. When only a fixed so E S is considered (instead Vs E S), (3.I) reduce to the pointwise confidence interval. Since the state space S can be continuous and infinite, to achieve asymptotic correct simultaneous coverage, we need to well control the family- -wie error rate in contrast with previous pointwise Cls in RL (e.g.. Luckett [ucketLLkkkett al. (2020), Shi et al. (2021) Shi et al. (2024)).
Without loss of generality, we focus on stationary policies .T( that do not vary with time (. For the justification, we refer to Lemma Shi, Wan, Song, Lu & Leng (020) and proof of Theorem 6.2.10 10 in Puterman| (1994). To enable simultaneous confidence inference, we impose three main assumptions, which are adopted from the literature pointwise inference (e.g.. SSh et al. (2021) The detailed assumptions and illustrations are listed as (AI)-(A3) in Section 3.2
3.1 Q-learning with linear function approximation
We adopt a Q-learning approach to develop valid inference procedures for both deterministic and random policies. The Q-function under policy is defined as
Q(T:s Σ
"ET.(Rol Ao.o a).
(3.2)
3

---

Under conditions 2. and 2.2 the Q-function satisfies the Bellman equation:
( = r(s, a) + + IQ (T;; 1 Ao.1) So,0 = 8, Ap,0 a]-
(3.3)
We consider the linear function approximation for learning the Q-function. s (s), ...)...... (x)s) be a collection of K basis functions and (s) (91 (8). We approximate Q(T;; 8, a) based on a linear combination of the basis functions, i.e..
Let
2x
(s))
QT:: s, a) = ((s) 3r.a Vs S,aEA
(3.4)
Related approximation results are presented in Section F.1 of the supplementary materials, assuming that : belongs to Hölder space of smoothness p for any policy and action E A. This condition holds under standard assumptions on the transition probability P and a smooth reward function r(s, a) (see Section F.1 for details). The basis functions be chosen from orthogonal splines, Legendre polynomials, or wavelets, forming a sieve basis commonly used in sieve estimation
(Chen 2007 Huang 1998 Cohen et al. 1993 Timan 2014).
By
and 3.4), the wwK-ddmmnnioonl vector 8 = ( 3T
satisifies
B Ri, + 7 Σ (( (S (t (S((s.)
(( Ba,a' ((S.,))(AA,, =a'= =0, (3.5)
aEA
for all d' A. Denote Et. {(S,, Un.11 = (Sr,) where
E(s, = (9(s) 0), (()) I(a = .,. ((5) II(a m 1m11)
(s) = (9(s) [(0)s) (s) ((s) T (m
-1|s}}
Then 3.5 reduces to E (R y F., - -T,,,,) O, and 3 can be estimated by
T
1
3x = 1
S.c.B...
(3.6)
Σ
T
1=0
where B, = (B.,,..... AT
T) and
n.m
n T
E Σ.
Exe (Eiee yUm.)e11
(3.7)
Consequently, the value for policy can be estimated by
(T::)) = Σ =())s)Q(T:)
Σ (cals)((s))A T...
(3.8)
aEA
aEA
O(ex) where 02 3.r B and < max SuPsEs QQ 8, a) - {s) 3t.al- By Chen (2007), there exists 19% such that EK = O(K p/d) when the Q-function lies a d-dimensional space with Hölder smoothness p.
By 3.4 we have (T;;) - ((T;s)
(s)s)
3.2 Convex Gaussian approximation
In this section, we establish general convex Gaussian approximation theory for learning the distribution behavior of θ, 3. 8 for all Euclidean convex sets in RmK To allow K to diverge, we apply convex Gaussian approximation theorem (Fang (2016), Fang & Koike (2024)), which supports moderately high-dimensional scenarios. We consider the state within compact region c Rdd For unbounded S. modifications such as introducing a weighting or mapping function are discussed in, e.g.. Tjøstheim & Auestad (1994) Huang & Shen (2004). Chen & Christensen (2015).
We impose the following assumptions.
(A1) The Markov chain { has an unique invariant distribution with some density function p(s). Denote (s(s) as the probability density function of So,0- The density functions p(s) and (s)s) are uniformly bounded away from 0 and oc.

---

(A2) Suppose the following (i) and (ii) hold when T - oo and (i) holds when T is bounded.
(i) Xmin E S0, - u u (So.t. A u ( Au.)] 2 Tč for some constant é > O, where "z a) E (Ux (Sb.1) S0,0 x, Ao.o = a} and Amiu (M) denotes the minimum eigenvalue of matrix M. (ii) (So, is geometric ergodicity in dependence measure.
(A3) Define (+ E { Rooo + +2 EaEAT(a ((1 S,,1, 2(T=;So.) Ao,o)
}
Assume w wx(s, a) 2 60 and P(maxosisr-11 RRo S co) = for some constant Co 2
Remark 3.1. Assumptions (AI))A3 are mild assumptions and serve as the minimal requirement for the goodness of the offline dataset to support feasible evaluation. The first condition in Assumption (Al) ensures that the Markov chain would not be trapped a small subset of the entire space. Moreover, the second conditon ensures that every state is possible to be the initial state. Assumption (A2) relaxes the condition on sample size. Previous work (Jiang & Li requires the number of trajectories co. (A2) additionally allows fixed n,, but length T + oo when the action variety is sufficiently large on each chain. The geometrical decay is similar with the geometrical ergodic for the Markov chain, which is technical assumption in theoretical deduction, and is commonly assumed as a weaker requirement of i.idd in deriving limit theory. Assumption (A3) requires the reward signal diversity. (r(s,,) a) 2 6o' requires that the reward random variable is nondegenerate (not always the same). P(maXosesT-11 Rooe S. co) means the rewards are bounded. The detailed definitions of the geometrical ergodicity and dependence measure presented in Section G of the supplementary materials to save space.
The following Theorem shows that there exists -ii---innsssonal Gaussian random vector Z such that probability of 0₂ = 3xx - B can be approximated by Zg over any convex sets. Theorem 3.1. Denote Zg as the mK -imenssionl Gaussian random vector possesses the same covariance structure of v VNO., ie.,
2 Nmx (0, Ax =E ( (1 (ET))
(3.9)
where
1 T Ωr =
S { Bax + AUU - -TT,1077 1²
(3.10)
1=1
1=
Under Assumptions (AI), (A2), and (A3), suppose that K = o NN)
then we have
sup
PP(VNO, 0) P(Zzr [0]l= +,
(3.11)
DED
where D the collection of all the convex sets in R.m.
Remark 3.2. Note that the SCR based on estimation (T; = Emes ( can be written as NoeSt VNO E Or,,) where
O = ( ERRK ((8)
Σ t(als)0 - <[(s)
(3.12)
asd
Or,s is a convex set since for any 0,0' E On.sy XO + (1 2)0' E Dr.s for any X E [0, Therefore, D is a convex set and the probability P(nses(vNO e (x,,)) ) can be learned by P(NGes(Z (Z, (r,,)))
Remark 3.3. Theorem 3. provides a higher-onder convex Gaussian upproximation for 0,, - 33
in the OPE estimation error (TT;)) 3 (TT; s) 2(s) Enc, π ().... Existing approaches for constructing pointwise confidence intervals typically rely on the central limit theorem, deriving the limiting distribution of the inner product 2(s) T EneA ((a)s) 0z for each fixed E S. However. extending these results from fixed to arbitrary E S is nontrivial, as it requires controlling
DO = P (NNO, E EO) - P (Zz E 0)) for some convex set (see Remark 3.2 for details).
5

---

Regarding the sanite properties, we provide the following bound on Dg with respect to the sample size N and the number of basis functions K, derived from the proof of Theorem 3.1
sup A SC DED
1-9eg "K
SK,N K " SK K a log N).
From the proof of Theorem 3.7 it follows that the above bound converges N 20 when K o (N2/T-c) for any given c 0.
By Theorem 3.] the SCR in (3. can be achieved by finding appropriate critical value Co, (which may depend on N) such that
a = P
(s) Σ
S CC..
aEA
(s)
EREA L(s)
{a)s)
P
sup #ES
S &Co.N
(3.13)
The probability (313) involves the supremum functional linear combinations of the high- approximate Ca,N in (3.13) by generating
dimensional Gaussian vector Z, In practice, simulations of the Gaussian random vector Z and computing the empirical quantile of the supremum.
This approach, known the Gaussian multiplier bootstrap, is detailed in Section 3.3 In theory,
leverage properties of Gaussian processes along with approximation techniques from Sun & Loader (1994) to analyze the desired Ca.N.
Proposition 3.2. For any two positive real sequences and bno we write On x b, if there exists constants c < <, oo such that S lim inf,--s an! /bn I lim sup, p- sup, an/On, C. We write an S b, (an Z bn) if there exists constant C > O such that On Cb (Con 2 bn) for all n. Denote matrix
0 ((s)
= ((,)))..,,,((s)), M2(s) = 0sj [
(3.14)
Under same conditions Theorem
if there exists constant co. 01, C22 C 2 such that
sup VD(s)) Na,sup V2 S N inf [((s) 2 No, / Amin (M MMdss NS, (3.15)
SES
SE
SES
then we have appropriate Ca.N = log 1/2 1/2 N such that
(T::)) (TT:))]
Ca. Ys Ys E -1
lim N0000
(3.16)
(U>(s) A,,(s))AUUUs))))
VN
where a is the given significance level and a E (0,
Remark 3.4. The scaling factor I(()) = (U*(s) ( (s)=(s) aligns with the pointwise Cls in Shi et al. (2021) so that we only need to compare the critical value Ca.N with that in pointwise Cls. The rates No, _es, Noo in condition (315)) are mild assumptions which have been frequently used in the literature of sieve nonparametric estimation and inference; see Assumption of Chen & Christensen (2015) and Example 1-2 in Quan & Lin (2024) for more details. The rate N for Amin (M Mjdsean be derived in practice given basis (s), which would be verified in Section F.2 of the supplementary materials.
Proposition 3.2 speciffes the essential scale the width of SCR. In contrast with previous pointwise confidence intervals proposed in Shi al.) (2021). Ca.N Vlog N shows that only an additional logarithmic rate vlog N is introduced to extend the pointwise confidence in Shi et al. (2021) to the global region S.
3.3 Bootstrap implementation
In the asymptotically correct SCR provided by 3316 calculating approximation of C..N N is rather complicated, and the convergence would be slow. We propose the Gaussian multiplier bootstrap algorithm to circumvent these problems and derive feasible SCR in Algorithm

---

Algorithm 1 Gaussian multiplier bootstrap for SCR
Input:
Observed data {Ra,,, St,, JosiST
Step 1: Calculate Br,, Σn, f2x according to (3.6). (37 and 3.10
Obtain the estimator of the
value function
(T; = ((s) x (als)
(3.17)
AEA
Generate mK -i---minniional Gaussian random vector - ~ N (o, 2 (2,)
Step 2:
Step 3:
Repeat Step 2 for B times and document the outcomes - b = L..... ........ B.
Step 4: For given level (0, 1), denote -1-a as the (1 - - sample quantile of
B
sup #ES
L(s) (b()s)
Σ π (als)
aEA
b=1
Output: (1-a)-th SCR (T; 91 (1-0/())/VV
It is worth noting that by the convex Gaussian approximation results, Algorithm I can yield different asymptotically correct SCRs by modifying the scaling factor L(s). The function L(s) provides flexibility to adjust the relative weighting of states $ S, where larger L(s) prioritizes tighter confidence bounds for state s. For instance, if one is interested in the uncertainty of maximal deviation supses (V(;;)) (x;;))] then L(s) = can be a convenient choice.
Remark 3.5 (Computational remarks). Our procedure is computationally efficient and can, for instance, be executed on personal laptop. The term 3x in (3.6) analogous to a least squares estimate and can be computed efficiently. Moreover, Steps and 4 of the bootstrap procedure are linear due to the use of a linear approximation. Overall, the time complexity of our method is O(NK² + K + BK) and the space complexity is O(N + BK).
4
Experiments
4.1 Simulation studies
In this section, we conduct numerical studies to evaluate the performance of the proposed SCR. Both univariate (d = 1) and multivariate (d 1) scenarios are considered. The code is available
http://gthhbbcom//iiiuaaaaaaa haaaiiaaaaaat///Smmaaanoouu Saaereeouss-ttiisttil-nneeeeeeee -for-OOo--ollcc--yallaaltt in ne-Reinforemeet-LLaammi
In our settings, the state vector So, may have bounded support. To address this, we apply sigmoid transformation, defined as s s(i)) 1+exp(-S$t)) for 1 j S d, to obtain features
with bounded support. The basis functions constructed using the tensor product of K Legendre spline functions. The number basis functions is determined through cross-validation (Qiu et al.) 202I). We put the detailed cooss-validttion procedure in Section D of the supplement. Moreover, performed sensitivity analyses and found that both the empirical coverage and the average length of the SCRs are robust to the choice of K.
We evaluate the SCRs using two metrics, each computed across 500 independent replications: (i) Empirical Coverage Probability (ECP): The proportion of times the true value function lies within the SCR across multiple simulations. (ii) Average Length (AL): The average width of the SCRs, approximated by averaging the widths at equally spaced grid points. The experiments can be readily conducted on a standard workstation, for example, an Apple M1 machine with 16 GB of RAM running macOS Sonoma.
For the method in Shi et al. (022) (referred to as SAVE), we apply the Bonferroni correction to adjust the pointwise confidence intervals. For each setting, we compute the SCRs over equally spaced grid points. We emphasize that, principle, pointwise confidence intervals cannot be naturally extended to SCRs, due the continuous nature of our state space. Overall, the proposed SCR
7

---

achieves coverage close to the nominal level (we set a 0.05), while the munferroni-adjuster SAVE results in a coverage rate well above 0.95.
(Scenario I (univariate).) Let = 0.5, n = 22, 25,50, 75, T 30, 50, 70, and
1 6
S0 0.1+1
S -o, (2A 1)U0, Ro
2 66,41
S6. 0,
for / 2 O, where Uo.t iid. UO, 1) and So,0 ~ U(-2,2). We consider a completely randomized behavior policy, i.e., Ao, iid. Bernoulli(0.5) for 0. The target policy designed as (1[ss)
1 - [(x > 0). We construct SCRs for V(T, 8) over the domain & [-2, 2, 2]. The true value function approximated from Monte Carlo simulation. We generate 10000 of independent trajectories with initial state being s 2 + 4i/999 for each i = 0,... 999. Actions are selected according to the target policy. We approximate V (TT, s)) by taking the average over the 10000 trajectories, and use linear interpolation to approximate V(T,, for & {s), i ...... 999).
For Scenario we employ the Legendre basis, and the results for ECPs and ALs are presented in Figure 1 Moreover, we perform the sensitivity analysis by taking (n. T) = (25, 50), (50, 50) two illustrative examples and examining the results by varying the specification of K over a relatively wide range. The corresponding results are presented in Table S.1 in the supplement. Figure I shows
1.00-
0.20
0.95
SAVE (n=25)
SCR (n=25)
0.10
SAVE (n:50)
(n-50)
SAVE (n.75)
(o.75)
2000
2000
4000
5000
Figure 1: Comparison of the methods based on empirical coverage probability (ECP, left) and average length (AL., right) for Scenario 1.
that SCR consistently achieve the nominal coverage level in various choices of T). In contrast, the Bonferroni-adjusted SAVE method exhibits substantial over-coverage. As = n? increases, the empirical performance of method converges more closely the nominal target. Additionally, the average width of the SCRs produced by SAVE is approximately 20% greater than that of ours, which highlights the improved efficiency of our approach. Table S.I further shows that our method is robust to the choice of the number of basis functions, enhancing its practical applicability.
(Scenario 2 (multivariate).) Let 7 = 0.5,
(2A0.t -
0 0 1 1-2 ) So, + 20,1, Ro.: = So
(2A0.r - 1),
1
So,t+1
4
for 2 O, where 20,t i..dd (0. 4I2) and So,0 U( where the two components are independent. For behavior policy, we consider Ao, Bernoulli(po.)) independently, where Po. = 0.5 (Sigmoid(S).)) + Sigmoid(s(?))) TThe target policy designed as =(1[s) = I(s())
(2) > 0). We construct SCRs for V(T,, 8) over the domain E [-1, 1J2. Similar to that in Scenario we simulate 10000 independent trajectories, each initialized at a point in the grid {s g()) = -11 2i/29, s(2) -1 + 2j/29, for 1 < i,j 30). to approximate the true value function s), E 1 We construct SCRs using tensor products of Legendre and spline basis functions, respectively. The results summarized in Table [T] Moreover, we conducted additional simulations employing SAVE with the Sidak correction (Abdi et al. 2007), and the results summarized in Table S.2 in the supplement.
Furthermore, we modified the state transition rule to assess the performance of our method under high noise and non-Gaussian emrrr.. Specifically, set 20, be an i.i.d. two-dimensional t(8) random
8

---

variable, while keeping all other components unchanged. The results are presented in Table S.3 where the coverage and length remain robust.
In addition to the comparison with SAVE, we also evaluated our method against the importance sam- pling approach (Jiang & Li [Hanna et al. 2017) based Scenario 2. The detailed experimental settings and results are provided in Section B in the supplement.
Table 1: Results for Scenario 2. Format: ECP(AL).
Legendre
Spline
n2
T
SCR
SAVE
SCR
SAVE
30
50
0.926 (8.472)
0.982 (9.445)
0.936 (9.452)
0.978 (9.793)
50
30
0,946 (9.553)
0,970 (10.492)
0.922 (11.101)
0.942 (11.131)
40
50
0.924 (7.225)
0.976 (8. 193)
0,938 (8.087)
0.976 (8.606)
50
40
0.944 (7.138)
0,990 (8. 136)
0.930 (7.247)
0.984 (7.753)
50
50
0.942 (7.299)
0.978 (8.249)
0.930 (8.156)
0.966 (8.630)
50
150
0.952 (6.985)
0,966 (7.402)
0.934 (5.771)
0.978 (6.080)
50
200
0.944 (5.978)
0.978 (6.436)
0,950 (7.733)
0.960 (7.418)
50
250
0.942 (5.957)
0,968 (6.234)
0.930 (5.177)
0.960 (5,446)
200
70
0.934 (4.924)
0,988 (5.370)
0.926 (4.766)
0.968 (5.045)
250
70
0.936 (4.374)
0.986 (4.800)
0.906 (4.245)
0.968 (4.521)
300
70
0.934 (4.430)
0.974 (4.737)
0.936 (5.029)
0.978 (5.031)
Table provides several key insights. First, it illustrates the theoretical claim that our method is primarily governed by the product N = nl' In addition, it indicates that both spline and Legendre bases lead to similar results, with the SCRs constructed using spline bases being slightly wider. This suggests some robustness of our method to the choice of basis functions, which is appealing for practical applications.
Note that the empirical coverage probability (ECP) the mean of binary outcomes. Therefore, we can derive the confidence interval for it. Specifically, the 95% confidence interval for ECP is given by (p 1.96 pp)5000, p+ 1.96 1:066p((1111 pp)/500), where denotes the empirical coverage.
Remark 4.1.
4.2 Real data application
In this section, we apply our method to the OhioT IDM datase J] which contains records of continuous glucose monitoring (CGM), insulin administration, and self-reported life events for six individuals diagnosed with type diabetes. The data is partitioned into consecutive three-hour intervals and has a three-dimensional state variable St, for each patient i at time step f. Due to the space limitation, leave the specific construction in the supplement. The action A,, is constructed as binary variable.
Ar, 1 if the cumulative insulin administered during the interval exceeds one unit; otherwise A = O. The discount factor is set as y = 0.5 to weight future outcome. The reward, Ri,r, is derived from the Index of Glycemic Control (IGC), a piecewise function that penalizes both hypoglycemia and hyperglycemia while assigning zero cost to glucose values within a clinically optimal range. i.c.,
2 a - (80 1
1.35
/30 <00)
(5 140)
((5()) 22100/11///300
The downloaded dataset has been separated as training group and testing group. Our objective is to conduct the simultaneous OPE on the testing group under the target policies obtained from the training group. In specific, we evaluate two kinds of target policies on the testing group. The first is an optimal policy л nht obtained by the double fitted Q-iteration algorithm (Härdle & 2010) in the training group; implementation details are provided in Section E of the supplementary material. The second is the behavior policy b obtained by the random forest from the training data. We then estimate value functions (p SS) and (6, So) on the testing set by (3.)). SCRs for ( Sp)
and (b, So) are constructed for all states in the test set by Algorithm
The results show that (x So) exceeds (b, Sp) by an average of 2.61, and improvements are observed in 87.1% % of the initial states. To characterize uncertainty, we examine the proportion of states under which the SCRs do not cover 0 (i.e., the average CGM blood glucose level is not within

---

the normal range) for both target policies. Owing to the uniform property of the SCR, the proportion of states for which the SCRs do cover zero reflects the fraction of patients who remain significantly poor condition under the target policy. The results show that, at the 5% significance level, for policy b, the value function (b, So) is significantly less than 0 in 90.7% of the states, whereas for policy ((r So), this proportion is 23.3%. We visualize the SCRs where the upper bound of 95% SCR is below than O, sorted by the value estimates, in Figure In terms of the average length, for V ( So), our method yields an averaged length of 27.0, while SAVE with Bonferroni correction produces an AL of 32.4, which is 20% larger than ours. Moreover, for (b, So), our method yields an average length of 7.02, compared to 7.58 for SAVE (approximately 8% longer). These findings suggest that, in the medical context, applying reinforcement learning algorithms alongside simultancous inference could improve health outcomes for patients.
Estimated Value
-30-
Estimated Value
-50
-100
-90-
-120
100
200
300
400
20
40
60
State Index
State Index
Figure 2: Left: Visualization of values where (b, So) sufficiently negative (the upper bound of 95% SCR is below 0). Right: Visualization of values where (iTTrt SS)) is sufficiently negative (the upper bound of 95% SCR is below 0).
5 Conclusion and future work
In this work, we present a novel simultaneous statistical inference framework for off-policy evaluation, proving that our SCRs are asymptotically correct via convex Gaussian approximation. The SCRs have widths exceeding pointwise confidence intervals by only logarithmic factor. This establishes near-optimal efficiency while achieving uniform coverage. The method's validity and efficiency are demonstrated both theoretically and empirically.
The current results limited offline settings. Extending this framework to online RL represents natural next research direction. Additionally, the simultaneous inference framework shows potential for extension more general Q-learning estimation in RI., including robust value estimation (e.g., Panaganti et al. 2022 Cayci & Eryilmaz (2023)).
Acknowledgments and disclosure of funding
This work was supported by the High Performance Computing Center, Tsinghua University. Weichi Wu, the corresponding author, is supported by the NSFC No. N2212271
References
Abdi, H. et al. (2007), 'Bonferroni and šidák corrections for multiple comparisons', Encyclopedia of measurement and statistics 3(01), 2007.
Cayci, S. & Eryilmaz, A. (2023), 'Provbbyy robust temporal difference learning for heavy-tailed
rewards", Advances in Neural Information Processing Systems 36, 25693-25711.
Chen, X. (2007), 'Large sample sieve estimation of semi-nonparametric models", Handbook of Econometrics 6, 5549-5632.
10

---

Chen, X. & Christensen, T. M. (2015), "Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions' Journal of Econometrics (2), 447-465.
Cohen, A., Daubechies, 1. & Vial, P. (1993), "Wavelets the interval and fast wavelet transforms', Applied and computational harmonic analysis.
Dai, B., Nachum, O., Chow, Y., Li, L., Szepesvári, C. & Schuurmans, D. (2020), 'Coindice: Off-policy confidence interval estimation', Advances in neural information processing systems 33, 9398-94911 9918-.
Duan, Y., Jia, Z. & Wang, M. (2020), Minimax-optimal off-policy evaluation with linear function approximation, in 'International Conference on Machine Learning'. PMLR, pp. 2701-2709.
Fang, X. (2016), 'A Multivariate CLT for Bounded Decomposable Random Vectors with the Best Known Rate", Journal of Theoretical Probability 29(4), 1510-1523.
Fang, X. & Koike, Y. (2024), 'Large-dimensional central limit theorem with fourth-moment error bounds on convex sets and balls', The Annals of Applied Probability 34(2), 2065 - 2106.
Feng, Y., Ren, T., Tang, Z. & Liu, Q. (2020), Accountable off-policy evaluation with kernel bellman statistics, in "International Conference on Machine Learning'. PMLR, pp. 3102-3111.
Hanna, J.. Stone, P. & Niekum, S. (2017), Bootstrapping with models: Confidence intervals for off off-po evaluation, in 'Proceedings of the AAAI Conference on Artificial Intelligence', Vol. 31.
Härdle, W. K. & Song, S. (2010), 'Confidence bands in quantile regression', Econometric Theory 26(4), 1180-1200.
Huang, J. Z. (1998), 'Projection estimation in multiple regression with application functional anova models', The annals of statistics 26(1), 242-272.
Huang, J. Z. & Shen, H. (2004), 'Functional coefficient regression models for non-linear time series: A polynomial spline approach', Scandinavian Journal of Statistics 31(4), 515-534.
Jiang, N. & Huang. J. (2020), 'Minimax value interval for off off-policy evaluation and policy optimiza- tion', Advances in Neural Information Processing Systems 33, 2747-2758.
Jiang, N. & Li, I.. (2016), Doubly robust off-policy value evaluation for reinforcement learning, in 'International conference on machine learning'. PMLR, pp. 652-661.
Kallus, N. & Uehara, M. (2022), 'Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning', Operations Research 70(6), 3282-3302.
Lagoudakis, M. G. (2017), Least-Squares Reinforcement Learning Methods, Springer US, Boston, MA, pp. 738-744.
Le. H.. Voloshin, C. & Yue, Y. (2019), Batch policy learning under constraints, in 'International Conference on Machine Learning', PMLR, pp. 3703-3712.
Luckett, D. J.. Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E. & Kosorok, M. R. (2020), 'Estimating dynamic treatment regimes in mobile health using v-learning'. Journal of the American Statistical Association
Marling, C. & Bunescu, R. (2020), "The ohiot dm dataset for blood glucose level prediction: Update 2020', CEUR workshop proceedings 2675, 71-74.
Matsouaka, R. A., J. & Cai, T. (2014), 'Evaluating marker-guided treatment selection strategies', Biometrics 70(3), 489 499.
Murphy, S. A., van der Laan, M.J., Robins, M. & Group, C. R. (2001). 'Marginal mean models for dynamic regimes', Journal of the American Statistical Association 96(456), 1410-1423.
Panaganti, K., Xu, 7.,, Kalathil, D. Ghavamzadeh, M. (2022), 'Robust reinforcement learning using offline data", Advances in neural information processing systems 35, 3221 1-32224.

---

Puterman, M. L. (1994), 'Markov decision processes', Wiley Series in Probability and Statistics
Qiu, H., Luedtke, A. & Carone, M. (2021), 'Universal sieve-based strategies for efficient estimation using machine learning tools', Bernoulli: journal of the Bernoulli Society for Mathematical Statistics and Probability 27(4), 2300.
Quan, M. & Lin, Z. (2024), 'Optimal one-pass nonparametric estimation under memory constraint', Journal of the American Statistical Association 119(545), 285-296.
Sallab, A. E., Abdou, M., Perot, F. & Yogamani, S. (2017), 'Deep reinforcement learning framework for autonomous driving'- Electronic Imaging (9() 70-76.
Shi, C., Lu, W. & Song, (2020), BBrekking the curse of nonregularity with subagging- -eeeginn-iiffrencee of the mean outcome under optimal treatment regimes', Journal of Machine Learning Research 21(176), 1-67.
Shi, C., Wan, R., Song, R., Lu, W. & Leng, L. (2020), Does the markov decision process fit the data: Testing for the markov property in sequential decision making, in 'International Conference Machine Learning', PMLR, pp. 8807-8817.
Shi, C., Zhang, S., Lu, W. Song, R. (2021), 'Statistical inference of the value function for reinforcement learning in infinite-horizon settings', Journal of the Royal Statistical Society Series B: Statistical Methodology 84(3), 765-793.
Shi, C., Zhu, J., Shen, Y., Luo, S., Zhu, H. & Song, R. (2024), 'Off-policy confidence interval estima- tion with confounded markov decision process', Journal of the American Statistical Association 119(545), 273-284.
Sun, J. & Loader, C. R. (1994), 'Simultaneous Confidence Bands for Linear Regression and Smooth- ing', The Annals of Statistics 22(3), 1328 - 1345.
Sutton, R. S., Szepesvári, C. Maci, H. R. (2008), 'A convergent o (n) algorithm for off-policy temporal-difference learning with linear function approximation' Advances in neural information processing systems 21(21), 1609-1616.
Thomas, P. & Brunskill, E. (2016), Data-efficient off-policy policy evaluation for reinforcement learning, in "International conference machine learning.. PMLR, pp. 2139-2148.
Thomas, P., Theocharous, G. Ghavamzadeh, M. (2015), High-confidence off- off-policy evaluation, in PProceedings of the AAAI Conference on Artificial Intelligence', Vol. 29.
Timan, A. F. (2014), Theory of approximation of functions of real variable, Elsevier.
Tjøstheim, D. & Auestad, B. H. (1994). 'Nonparametric identification of nonlinear time series:
projections', Journal of the American Statistical Association (922)) 1398-1409.
Uehara, M., Shi, C. & Kallus, N. (2022), 'A review of off-policy evaluation in reinforcement learning'. arXiv preprini arXiv:2212.063:
Xu, Z., Li, Z., Guan, Q., Zhang, D., Li, Q., Nan, J., Liu, C., Bian, W. Ye, (2018), Large- scale order dispatch in on-demand ride- aade-haiinng platforms: A learning and planning approach, in 'Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining', pp. 905-913.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions and scope introducing theoretically justified simultaneous inference framework for off-policy evalua- tion. These claims are well supported by the theoretical analyses and empirical evaluations presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses several limitations in the final section of the main article.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: The paper clearly states the assumptions required for its theoretical results. All theorems formally stated and proofs are provided in full in the supplementary material. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental setup, including data generation processes, parameter set- tings, algorithms, and evaluation metrics, is described in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code available at http://gshhubbbcoooiiiyuaaaataan
The datasets
used were obtained from Kaggle.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.c//wwgeeeeeeee / http:///ppi......i..//ooo//teaaa htt////ooeSSSiiseeeeeeeeeee SonpoddeummsssiooPPe.. wwriiisssiinpiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper provides complete details of the experimental setup, including data generation schemes, model parameters, and the rationale behind their selection (e.g.. cross-validation).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper provides a way to measure the randomness of the simulation results in Remark 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper specifies that experiments were conducted on a standard personal computer. The computational requirements minimal, and all experiments can be efficiently on CPUs without specialized hardware.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code of Ethics. The dataset used is publicly available on Kaggle, and the algorithms proposed do not pose known ethical or societal risks.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
16

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any high-risk models or datasets and poses no identifiable risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The dataset used is publicly available on Kaggle.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeoooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new datasets or models. It proposes new theoretical framework.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any human subjects crowdsourced data collec- tion.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve human subjects and does not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No large language models (LLMs) were used as part of the method develop- ment, experimentation, or analysis.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) (btr:p///eerrrpp.cc///ooofereeer /ctp:///e/eeeeeece////////.... for what should or should not be described.
19

---

Supplementary materials for "Simultaneous Statistical Inference for Off-Policy Evaluation in Reinforcement Learning"
Tianpai Luo" Xinyuan Fan Weichi Wu' Department of Statistics and Data Science
Tsinghua University Beijing 100084, China
(1tp21, fxy22) @mails.....mmmmmmmmmmmmmmmllmm.1 tsinghua, edu. cn, wuweichi @tsinghua.edu.cn
The supplementary materials organized follows. In Section A] we discuss the key challenges in policy evaluation for offline reinforcement learning and describe methods for assessing dataset
quality. In Section we report the additional simulation results described in the main article. In Section C we detail the construction of the state space in the real data example. In Section D we present the cross-validation method for selecting the number of basis functions. In Section provide the double fitted Q-learning algorithm. In Section F we introduce commonly used basis functions and verify the related conditions stated the main paper. Section G contains the proofs of the theorems, along with the relevant lemmas.
Notations in this supplement are summarized as follows. For a vector =: (vi, 02, ....... Up) E RP. let |v| (/) 1/2 For random vector V and probability measure P. denote | [Er ([V|3)) > 0 where (-(() is the expectation with respect to probability P. For simplicity, we shall use EE.). 11. || instead of Ep(-). || Ilp... || |lp.2, respectively if no confusion arises. For matrix A, the determinant of matrix A is denoted as (A). If the matrix A is real and symmetric, we use Amin (A) (Amax (A)) denote the smallest (largest) eigenvalue of A. For any two positive real sequences an and bn. write on b, if there exists 0 <
C < x such that e lim inf, On/bn < lim sup an/Up, C. We write an S bn (an 2 b,,) to mean that there exists universal constant C > 0 such that On, < Cl,, (Can 2 bw) for all n.
A Offline Reinforcement Learning: Evaluation Challenges and Dataset Quality
Unlike online environments (e.g., Go or MiniGrid), collecting data through online interaction in many real-world applications, such as healthcare or autonomous driving, can be costly or even hazardous. This limitation hinders the widespread adoption of traditional online RL methods. As alternative, offline RL leverages large historical datasets, and it is particularly suited to situations where online interaction is infeasible but cxisting data is available for learning.
Nevertheless, offline RL introduces unique challenges compared online RL. From the learning perspective, key difficulties include distributional shift, where the behavior policy used to collect the dataset may differ from the target policy being learned, potentially leading to poor performance or overfitting; and sample inefficiency, since learning relies entirely on static dataset, preventing online exploration. From the evaluation perspective, estimating policy performance without online deployment necessitates off-policy evaluation (OPE), which is the focus of our work. A central challenge in OPE is the gap between estimated off ofllcy performance and real-world outcomes. Recent methods, such pointwise confidence intervals, aim to quantify this gap probabilistically.
"Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our work extends these tools from pointwise to global inference, providing more reliable guarantees for decision-makinn
The quality of an offline dataset critically affects learning and evaluation. Important factors include state-space coverage, reward signal diversity, action variety, and compatibility between the data distri- bution and the RL algorithm. As noted in Levine et al. (2020). formalizing non-trivial sufficiency condition for dataset quality remains open problem. In framework, these considerations are captured through Assumptions (AI)-A3), under which the dataset's quality can be roughly quantified by its sample size N. Our empirical results demonstrate reliable performance of our surge-sample theory with dataset sizes around N 2000, outperforming modified classical methods such as SAVE when sample sizes are larger. This scale is modest compared to typical offline RL applications: for instance, the HiRID Rodemund et al. (2023) and MIMIC-III Johnson et al. (2016) datasets contain extensive ICU records collected over multiple years, and datasets like D4RL. (Datasets for Deep Data-Driven Reinforcement Learning) provide large-scale data for computer science applications. These examples highlight that our approach can be reliably applied across wide range of real- worlw scenarios.
B
Additional simulation results
B.1 Sensitivity analysis for scenario 1
We perform the sensitivity analysis for scenario 1 by taking (n, (25, 50), (50, 50) as two illustrative examples and examining the results by varying the specification K over a relatively wide range. The results presented in Table SS.1 It can be seen that method is not sensitive to the choice of the number of basis functions K.
Table S.1: Sensitivity analysis under Scenario for different values of K.
n
T
K
ECP
AL
10
0.92
0.121
11
0.93
0.127
25
50
12
0.94
0.133
13
0.92
0.137
20
0.93
0.120
22
0.94
0.127
50
50
24
0.96
0.135
26
0.95
0.143
B.2 SAVE with the Sidak correction for Secnario 2
The results for SAVE with the Sidak correction in Scenario are summarized in Table S.2 Results with high noise and non- Gan-aassiinn errors are reported in Table S.3
Table S.2:
Results for Scenario 2 using SAVE with Sidak correction. Format: ECP(AL).
n
T
Legendre SAVE (Sidak) Spline SAVE (Sidak)
30
50
0,980 (9.431)
0,976 (9.778)
50
30
0.968 (10.476)
0.942 (11.115)
40
50
0.976 (8. 181)
0,974 (8,593)
50
40
0.990 (8. 124)
0.984 (7.741)
50
50
0.978 (8.237)
0,966 (8.617)
50
150
0.974 (7.393)
0.972 (6.075)
50
200
0.974 (6.435)
0.966 (7.385)
50
250
0.952 (6.223)
0.980 (5.431)
200
70
0.972 (5.360)
0,978 (5,050)
250
70
0.972 (4.804)
0.958 (4.503)
300
70
0.986 (4.726)
0.972 (5.020)

---

Table S.3: Results for Scenario 2 with t(8) noises.
Format: ECP(AL).
Legendre SAVE
Legendre SAVE
Spline SAVE
Spline SAVE
T
Legendre SCR
Spline SCR
(Bonferroni)
(Sidak)
(Bonferroni)
(Sidak)
30
50
0,940 (4.706)
0,934 (4.431)
0,980 (4.424)
0.936 (3.716)
0.934 (3,967)
0.986 (3.961)
50
30
0.944 (2.721)
0.932 (3.316)
0.990 (3.311)
0.934 (3.694)
0.932 (3.959)
0.988 (3.953)
40
50
0.946 (3.349)
0.934 (3.658)
0.992 (3.652)
0.952 (3.632)
0.950 (3.755)
0.990 (3.749)
40
0.932 (2.876)
0.928 (3.279)
0.990 (3.274)
0,944 (3.104)
0.942 (3.393)
0.988 (3.388)
50
50
0.934 (3.488)
0.926 (3.448)
0.986 (3.443)
0.934 (2.776)
0.932 (3.048)
0.994 (3.044)
50
150
0,948 (2.333)
0.944 (2.498)
0.988 (2.494)
0.948 (2.415)
0.944 (2.524)
0.988 (2.520)
50
200
0,942 (2.012)
0,946 (2.168)
0.984 (2.16))
0.936 (2.078)
0.934 (2.184)
0.988 (2.181)
50
250
0.932 (1.962)
0.934 (2.111)
0.994 (2.188)
0.950 (2.330)
0.944 (2.332)
0.986 (2.329)
200
70
0,946 (1.970)
0.996 (2.024)
(2.021)
0,944 (2.176)
0.944 (2.196)
0.980 (2.192)
250
70
0,948 (1.644)
0.980 (1.784)
0.980 (1.780)
0.944 (1.943)
0,944 (1.968)
0.982 (1.965)
300
70
0,930 (1,495)
0.986 (1.625)
0.986 (1.623)
0.948 (1.770)
0.944 (1.800)
0.980 (1.797)
B.3
Comparison with the importance sampling method for Scenario
We evaluated our method against the importance sampling (IS) approach (Jiag & 2016 Hanna al. 2017] under Scenario 2. Specifically, we set S. (-2 2- 0.44) where 0 < j 10. For each combination of and j. we generated no = 100 trajectories, each of length 10, while keeping all other settings unchanged. For bootstrapping IS, we employed the algorithm from Hanna al.) (2017). which provides confidence intervals for each V (TT,SS) So). The Bonferroni correction was then applied to obtain the simultaneous confidence bands (SCB).
is worth noting that IS approach estimates the value function by directly reweighting trajectories, whereas Scenario 2, the target policy <(1ss) = I(s(1) > 0, > 0) is discontinuous in s. Moreover, the choice of target policy frequently results in weights of zero, reducing the effective sample size and causing the IS estimates to be dominated by a small subset of samples. This leads to biased estimates and, consequently, degraded performance. We mention that Hanna et al. (2018) also highlighted the same issue in their Section 7. Our method is not impacted by this problem, further showcasing its practical applicability.
From the results, under this setting, the IS method achieves an empirical coverage probability (ECP) 0.638, with an average length (AL) of 5.300. This is below the nominal level of 0.95. Increasing the sample size can improve the coverage of the IS method, however, the associated bootstrap procedure becomes computationally expensive. In contrast, our method performs well even with smaller sample size (no = 10), achieving an empirical coverage probability (ECP) of 0.954 and an average length (AL) of 4.575.
C Specific construction for states in the real data example
In the real data example in the main article, we construct three-dimensional state variable S for each patient at time step L. Specifically, s! represents the average CGM blood glucose level over the preceding three-hour interval. s(2) is a decayed sum of carbohydrate intake within the same period, where each meal's carbohydrate estimate is discounted according to its temporal distance from the current interval. Specifically, if meals are recorded at times f1, f2...., ty [t ft-11 1, with corresponding carbohydrate estimates CE₁, CE CEN, then s Σ CE₃ 0.536(t)-t+))
s(3) denotes the average basal rate over the same three-hour window, capturing the background level.
D Cross-validation for choosing the number of basis functions
The method of cross-validation is widely used in machine learning and sieve methods (see, for
example, Van Der Laan & Dudoit (2003). Hansen (2014), [Bates et al. (2024)). Based on the key
equation (3.5) in the main article, adopt the following 5-fold cross-validation approach, as described in (S.
Algorithm S.1 5-Fold Cross-Validatio
3

---

1: Input: Observed data D = {(Re,,, Ar,, S,,,, Si.1 igigni candidate set of choices
Kaan {K....... k-
2: Randomly partition DD into 5 approximately equal-sized folds: D1, D2, 23, Da, Ds.
3: for j to 5 do
4: Set the j-th fold as validation set: Vval <- D
5: Set the remaining folds as training set: Drain + Uits D.-
6: for k = 1 1 do
7: Obtain (22 based on Daain using equation (366) with kg basis functions.
8: Let
-vG
CV(,k)= = Σ
Ris + Σ
(Sa.+11)
T 3(.R) P7.
1 ((a)Ste-1)
Dut
aEA
((E,)) T ((Se.))
9: end for end for
11: Let k* arg ming=1,... Ej=1 CV(
12: Output: Select ky as the number of basis functions.
E
Double-fitted Q-iteration algorithm
The double-fitted Q-iteration algorithm (Hasselt 2010) is presented in Algorithm S.2 The Q-function Q(::;)) can be specified using any model indexed by 0, and linear model with basis functions to approximate Q
Algorithm S.2 Double Fitted Q-Iteration Algorithm
1: Input: Observed data ((Rr,, Ai,t, St,t, 1lo4s)), isigai initialize parameters d AB.
2: repeat
3: Step 1: For all i, compute target values:
Q R4, + 7Q (S,,41,, arg max Q(
Q - RA, + 7Q (i,,41, arg max
4: Step 2: Update parameters by minimizing squared errors:
0 < arg min 0A
ΣΣ
Q(Xx, Acr) -Q21
0B <- arg min 8B
Σ>
Q(xxs A,,;0 -Q.
5: until convergence
6: Output: Learned parameter A_
Sieve method
In this section, we introduce commonly used sieve basis and verify the related conditions in the main paper. We list several commonly used sieve basis follows, which can be used in our simultaneous inference framework.
Example F.1 (Legendre). Define Legendre polynomials
1 di
P₂(x) 22 dzy (x2 1y, x x [-1,]
Then continuous function f(x) on [-1, can be written as
f(x)= = Σ (x)) aj G 1+ :/ S P,(x))(z)dz.

---

Example F.2 (Fourier). Consider real-valued function f(z) E 121-1,11 ic. f()dd < cc. By Fourier transformation, f(x can be written
f(x) Σ a3os(1), aj a== - (
where {0)(x)) {(cos(jTZ) + ssi((jxx) /2 forms an orthonormal basis for T2[-1,1}
Example F.3 (Harr wavelet). The Haar sequence was proposed in 1909 by Haar 1910 Haar
used these functions to give an example of an orthonormal system for the space of functions. For every pair n, k of integers in Z, the Haar function hn is defined on the real line I. by the formula
h (t) 2 (2"t k))
where h(t) is the Harr wavelet's mother wavelet function
0 Ost<
h(t)
2 sik1
0
otherwise
The Haar system the real line is the of functions
(tn,a(t) n Z. 2),
which is orthonormal basis.
Example E.4 (Daubechies wavelet). For N E N, a Daubechies mother wavelet of class Daubechies- N is a function & E (R) defined by
2N
δ(x) = 22
Σ h22 1-22"h2N---- -k),
where hy, h1, h2N-1 E R are constant and satisfy Σk= h22 = v2 =
-E2 h2441 as well
as, for 1 = 0, 1, N 1,
2N 2N-1+21
[1, 1=
hihh-
10, 1 1 0
2
The (p(x) is the scaling wavelet function supported on 20 1) and satisfies the recursion equation up(x) = 2 Σ he k)), as well as the normalization fg :(x)da = Ja p(2x k))p(2x 1112x-1jer l)dæ 0, k & I. As listed in Daubechhiss (1992), the filter coefficients ho.... h2N can be efficiently computed. The order N decides the support [0, 2N - 1) and provides the regularity condition
[ =2,6))dd=0..=00 0.j
=
The Harr wavelet as introduced above can be regarded special Daubechies wavelet with N 1. In our simulations and data analysis, we employ Daubechies wavelet with a sufficiently high order N to construct sequence of orthogonal sieve basis proposed in Daubechies (1888). For given J and Ju. we consider the following periodized wavelets on [0, 1]
(
2 -1: 1: SJ - 2/ 1}. where
())((1) = 20// Φ (2 1n/ - k) (x) = 22/2 Σ
# (2'x + 2 - k))
1E2
or equivalently, by Yves 1989.
The 2%, equals to our basis number Additionally, we refer to Chen (007 for a more general example of orthogonal wavelets.
5

---

F1 Sieve approximation
For the approximation (34), we show that the sieve method can approximate any function in the Hölder space with smoothness Given d-tuple a = (a1...., Q.) of nonnegative integers and [a] = 01 + ..... ad, the Hölder space with smoothness p. (S). defined as
Ag(S) = { h E (S): sup sup DD & C, sup sup [a]sm sES
Dah(2) - (Ph(2)-Dhh(u)) S SC
I 817
aaj=m a.
(S.1)
where C > O is constant, p = m 7 E (0, 1], (T)(S) is the class of m-times continuously differentiable real-valued functions on S, and the differential operator
Glai
D
Ozi ... 0z
For function Q(T;;, a) E (S). sup [Q - 5 0 p/d) if ((s) is the
tensor product of sieve bases such as B-splines, Legendre polynomials, orthogonal wavelets, or Fourier series if is periodic; see Section 2.3.1 Chen (2077) or Timan (2014) Yves (1989) Chen (0077)
As discussed in (hi, Wan, Chernozhukov & Song (021), there exists some transition density function q such that P(ds', a) = q(s'ss, ss, a a)ds if the transition kernel P(-|s, a) is absolutely continuous with respect to the Lebesgue measure. The following Lemma shows that (T; and reward r(s, a) follow certain mild conditions.
; a) E (S) if q(s'),,
Lemma F.1 (Lemma 1 in Shi, Zhang, Lu & Song (0011). If there exist some p. C O such that
r(,,),, q(s)).a) E (2(()) for any a A, E S, then there exists constant C" > such that Q(T;ia) (S) for any policy and a E A.
F.2 Geometric properties of sieve space
In this section, we verify the condition (3.15) in Proposition 3.2 Condition 3.15)
simplified
requirement on the sieve basis which will yield a polynomial rate No (c 2 0) for the geometric
quantities, including volume, curvature, and boundary of the manifold {*(s)/ [s)] E 8). For
simplicity, we only verify [s X 4/2
yoin (M M)ds Z N in condition 33.55 We refer to Assumption
4 of Chen & Christensen (2015) and Example 1-2 in Quan & Lin (2024) for the rest polynomial
rate conditions in (3.15). Define ÊK,N =: supses [s)s and AK,N =: sup.es (V)( Then there exists W, wo, W1, w 0 s.t. ÊK.N & NW1, and KNN S N and No < KK&M N
Lemma F.2 (Lemma E. in Shi, Zhang, Lu & Song (021 There exists some constant c* 2 1 such that
(c')-1 S Amain {L ((s) (s)ds S Amaxx (1e ((s) (s)ds sc²
and ENN S c²v CVK.
We verify condition (3.55) using trigonometric basis functions as a representative example. A similar procedure applies to other types of basis functions.
Suppose that d = 1,S = |- T]. the number of basis functions is K = 2k + 1, 2 1. and (s) (1,sin()) cos(s), ... sin( Ks), cos (Ks)) Then (+()) = VR + 1+)) = (0, cos(s), sin..).... K (os(Ks), K sin Rs)) / K + 1, and TTM- -ER (/(+1))
K2. As a result, we have Ss Aman (M M)ds 2 K ~
Now consider the case where d 2 and S
|-,, m| Suppose that the number
of basis functions K = (2K + 1)2. Then ((s) = o(s1) @ )(s2) where o(s) = (1, sin(s), cos(s), sin [ cos (Ks)) ( = (ss)) 8 ((2))/ + 1), My(s) = d(s1) &
(#2)/ K + 1) where (s)) = (0,cos(s), - sin(s),,.............. K (oss))s), - -K kin Ks))
(((s))]((s)))()))3(43 ((22) ((s1) T [{(s))(())) ((((2)
M T M =
(K' + 1)2 ((s2))(())))) (bs))) ((si) T (()sss)))))))) T(((s))
R(K + 1)2 +1) (1 0)
1) R(2 + (6 1
0
(K + 1)2

---

Then we have Is Aman LMMM)ds 2 K N
G Technical proofs
G.1 Dependence measure and geometric ergodicity
To measure the dependence in the Markov chain, we introduce the concept of physical dependence
measure in Wu (2005). For pair of jointly distributed random variables (X, Y). let yx)
P(X 2., Y y) v. y E IR, be the joint distribution function and Fy|x (y æ) = IP(Y S x x) the conditional distribution function of Y given X = For u E (0, 1), define the conditional quantile function G(x, u) inf {y E R Fylx (y x) 2 u). Let U be a uniform (0. 1) distributed random variable and assume that U and X independent. Then can view Y as the outcome of the bivariate function Y - G(X, U) such that
(X = (X, G(X,)))
(S.1)
For many standard constructions of stochastic processes (see e.g. Deák 1990). chapter 5), stochastic process {X,} can represented as
X1
X1 G2 U2)
H1 (U1) H2 (U2)
X₂
=d
=d
(S.2)
X,
Gn (X Un)
H, (U.))
where HI,..., .,, H,, are measurable functions. The above representation can characterize many Markov sequences (see e.g. Rüschendorf & de Valk (1993). If {Xe} } is a Markov chain, then the conditional quantile G2 (X2-1. can viewed as function of X,-1- Wiener (1958) first considered this representation problem for stationary and ergodic processes. For Markov chain [X,) with the form
X, G. (X,-1, U:). Wu & Mielniczuk 2010) asserts that, there exists copy X, of X, such that
X
(Xi) and X, is expressed as ( U--1, U4), function of iid random variables.
iEz
Lemma G.1 (Theorem 1 in Wu & Mielniczuk 2010)). Assume that (Xi} satisfies the recursion
X, G; (Xi-1, U.) =: F. (X
(S.3)
where Uj are i.i.d. standard uniform random variables. Here F, are independent random maps F((z) = G2 (x, U.). Assume that for some >> 0 we have
|[G((z,)) U) G.(,, U)Il.
sup Lj < 1, where L; = -sup xEz
and for some 2o.
sup |[G; (zo, U)l,o 00.
Then the backward iteration F, a F;-1 o F. converges almost surely and the limit forms non-stationary Markov chain which is a solution to S.3)
Let i E Z, be i.i.d. random variables and let H. -,..,,--,, G.). Based on Lemma G.I we can consider the irreducible and aperiodic Markov chain [X:) as
X, H4 (H4), E Z
(S.4)
where H; measurable functions. We view & as the input and X, as the output of the system. If H, does not depend on i, i.e., H, = H. then the process (X,) is stationary. Then we introduce the following definition measure the dependence of (X:) in S.44: Let (C) be i.i.d. copy of {G}. denote Hi,k = (Hi Gi-ktt,... (.).
Definition G.1 (physical dependence measure). Assume that sup, (H4)l < co for 0. then we can define the physical dependence measure of (X,) as
òR q) sup (HH:) H,(HH.)))p ig
(S.5)
where H4 (H4-k-1, S/- G...1...., G.).
7

---

Note that the Lemma G.I and Definition G.1 allow a nonstationary Markov chain; our theorem can actually be generalized to nonstationary cases, which can be promising future work.
For our stationary state observation (So.e). by Lemma G.1 our geometric ergodicity assumes {So,z) is an irreducible and aperiodic Markov chain where there exists {Su.}} S(H)) in the form of (S.4 such that {So.)) =d {So and the physical dependence measure is geometrically decaying, Le. 0g 1) O(x") for some constant X (0, 1). In fact, for contracting Markov chains (e.g.. autoregressive models), this assumption generally holds. Notice that
(s(k, S sup |s (s(( #ES
1))
(S.6)
0s(k, = O(x²) holds for any given E N+ since state space S is bounded. Furthermore, denote ((0,,) G(H,) 2 S(H,), then the physical dependence measure
óz (kk =: [[G(H,)) G(H,,))ll4 ( q)) = O(AK,NX))
Note that (0,,) S sup, [0()) = ÊK.N- Using the fact min(xr 1} S x 2 0 for any given
E (0, 1), we can have (k, q) = AR for any given E (0,1).
G.2 Proof of Theorem 3.1
We introduce the following lemmas before proving Theorem 3.1 In the proof of Theorem and the following Lemmas, we will omit the subscript in (+).). un (,(+)) Ex, Ex,, 3r, 3r Wr. etc, for brevity. For simplicity, we deduce our proof under the condition (34) In other words, we consider the Q-function Q" &TT (ss) f= instead Q(T;; 8, a). This can be achieved when the Q-function is smoothing enough discussed in Section F.1 We denote the dimension of 3 as p =: mK where x K since m is fixed. For the convex Gaussian approximation, we introduce a smoothed function
hA,e, (w) =: h iinfvE W )
(S.7)
E1
where w E IR,, convex set A C IP, and
1,
I < 0, 0 x 1
2x
h(x)
(S.8)
2(1 x))
2 < <<1, x 1>1. 1.
0,
To show the convex Gaussian approximations, we introduce the following Lemmas. Lemma G.2 (Lemma 5.3 in Fang & Koike (024 ) For any p-dimensional random vector W,
sup |P (W e 0) - P (Z e 0) S 4p 1/4 + sup |Eho,,, (W) - -Eho,,, (Z)), DEO
GED
where Z is p-dimennsonal Gaussian random vector with invertible covariance matrix and D is the collection of all the convex sets RP.
Lemma G.3 (Theorem 2.1 in Fang (016)). Let W = EE X, be sum of p-dimensional random vectors such that E (X,) = O and Cov(W)= = E. Suppose can be decomposed as follows:
1. Vi E [n], 3i E N; C [n), such that - XNN; is independent of Xú, where [n]] = {1, ,.).
2. Vi [n],j N₁, JN, C Ny C [n]]. such that W -X is independent of (X,,,,)
3. Vi [n],j E N
ANej C Nesk C [ri] such that W XNNsaa is independent of
Suppose further that for each E [[]],j N E Nig.
[X,] B, S nI, INil S n2, [Nijol ns
where| is the Euclidean norm of vector. Then there exists universal constant C such that
< P(W 0) -
x)) Cp x 1/2 8d
sup OED
d
where 7. a p-dimensional a- Gaussian random vector preserving the covariance structure of W and where D denotes the collection of all the convex sets in RR?

---

Lemma G.4 (Lemma E.2 in Shi, Zhang, Lu & Song (2021)). Suppose the conditions in The-
orem 3.1 hold. We have as
N + 00 that |- |>=' I 3c ||
0(1), Σ1|r
Op (12 log N.
5 - x-1
Op / log N} and 1--1 S 66-1
with probability approaching I.
Lemma G.S (Lemmaa E.3 in Shi, Zhang, Lu & Song (0011). Suppose the condi-
tions in Theorem 3.1 hold.
As N + 00, we have Xmax
T EiTo EEO.
0(1), Amax {N-1 Ex=1 Ei=o S.,55.t] = Op(1), Amin (T
2t=
2 2/2 and
N-1 2i=1 Et StaShh }
Amin
2 2/3 with probability approaching 1.
Proof of Theorem
3.1
By definition and the arguments in Section F.1 we have
0=
3 - B"
E-1
[Ra, (Ei. -U,+11)) 8*]
N
t=
n
T-1
(St,411) (a Shana
H
Sipe
R T (S +7 Σ T
aEAA
1
T-1
+ +
Σ
+ +O(K-R//)
(S.9)
N
N
mm0
0m0
where
f = Ri +
Σ (T; ( Si,41) Q" Si,S Ai.).
(S.10)
aEA
Denote Gi,r as the sub-dataset {S A U [[Rij Ais St, Sa.) Disser By the Bellman equation and conditions (2.1) and (.22). we have
Bsi, Fs,)) = E (er St,, Aai)) =0
Recall the definition Ex, ((S,, Ar,) in 3.5). we have for any 0 S t < t2 TT
Si,,,,,, fi,t2 = E (EL,, B F(,ta)) =0
Therefore, for any 0 S t1 < t2 < T - and 1 < i < i2 S n we have EE E22, ET Eiz = 0 and
Σ
Sir
Ee
L c
Σ Bo
i=1
By Assumption (A3) and Lemma
we have
Σ
Sipri
S E S nī sup (bss) O(NK)
i=1
t=
sES
By Markov's 's inequality, - we have
Di
Et Ei O,lvK/(K/))) K/N), and together with Lemma G.4
S (-11--11
Ste =Op
Op (KN-'log N)g
(S.11)
In the following, we show the convex Gaussian approximation Σ-1
VN ΣE Ei=
E,1E1,))
Denote ZN = VN En T-1 Zi,t and Zi,t = Ei define the truncated Zi, as
Zi.t
1 S nn otherwise.
Zi,1
(S.12)
10
Denote ZN Σn define
1i=1
=t=
T
Fish
N and Zi =: ZN EZN. Suppose T/m = k E E N.N.l.. w.l.o.g.. and
(m)
Fm....(F.(())
(S.13)

---

where (() = 00--mmm1......... .)) Then (m))
L. are independent as long as k --j > Further 1,
[m]
let Z/mi Ei= (/m) //N and Z (m) Z (m)
E
(m) then V x ZN -
N
Denote the covariance matrices f and ((()) of ZN and Z standard Gaussian random vector G and denote
(m) respectively. Introduce p-dimensional
GN = Q!/
( N = (D(m))))
1/2
G
so that GN and G (m) N preserve the covariance structure Ω2 P((m), respectively. We then introduce the convex Kolmogorov distance to measure the convex distribution probability difference between p-dimensional random vectors X and Y.
() sup P(x E 0) - P(Y E (O),
(S.14)
CED
where D is the collection of all the convex sets in R Combining (S.9) and (S.11) it suffices to show K(ZN. GN) = o11. By Lemma G.2 and / S 2c we can decompose the K(ZN. GN) as
K[ZN. GN) - 4K21/4 + sup OED
EE [ho,er (ZN) - ho,e, (Gx)ll
< K Ktei + sup GED
E
|hcaa (Zx) ho, (2x)
E |hoora (zN) hoo.
(Z(T))
sup DED
+ sup DED
E [issee (G((m) - ho,n (Gx)
+ sup GED
E
hoce
(z(2²) - -10.0
(m))
N
+ 1 Zn IN
EF
IN
((m)
+
E
((m)
(m) N
G
(Z( ho.e:
(G
(TA)
+ sup DED
E he.e1
(S.15)
Based on decomposition assertions as follows:
S.15), for q > and appropriate m x logn, we shall prove the following
(1) Truncation error
E ZN ZN| = =OVVNT (sk.)
(S.16)
(2) M-decomposition error
E In mmm) ol n -99 sk.n)
(S.17)
(3) Gaussian comparison
E (m)) GN =0 -NNT
2-2 5k.n)
(S.18)
(4) Gaussian approximation
(m)) )] =0 -O(1/AAN-124 ( log N - K (2xx) ho.a (G(()))
1
E ho.a
xN log" NN)
sup OED
61
(S.19)
Truncation error
In view of EZ, = 0 and Zi,t Zi, = Zi 1 |
we have for q > 1,
E ZN
ZyI
- E
Zn
ZNl + EZN
EZN1
2
E
Zi,t
1
VN
9-1
<2VNE
[zr,e]
TN
2
N
(S.20)
10

---

which yields (S.16) using the fact ||z,,ell - (suup, [ = OEK..) for any given & E N Moreover, also have for > 1,
[iirl]
2q-2
1/2
1N Zyrl 22 E
|i,e|
A
(T(na))
(S.21)
where TN.4 =: Vna 9 91 1|2..e1120
M-decomposition error Denote operator (()) Z,, =: E(z,, FFx - -(zz,. using the fact E lim ( (t)), we have on richer space,
T-1
IN ZN
Z Zit
R.T.
(S.22)
1=1 j=-
j=
where RI,,, =: Zi=(m-j+1j)) z(. By Jensen's 's inequality, for q > 1,
pp S $||ptt/ane1 = (B Gi 15 :,)
E (Zi, (1 (zilHe) He- 111] 1/a
do 1,4) (S.23)
Note that for given i, process filtration SS$-stu: If 2 by Burkholder's inequality, there exists constant Cg
(R....j. 2 m n + 1} is martingale difference with respect to
such that
R,T.
Cc
Σ
R
1++1
j=m-n++
< CCg
Σ
p
(S.24)
j=m-n+1
(=(m-j-1jv1
Using the fact do (k: = 0(4 N x [K,N) for any given a E (0,1), S.23 yields
pp( 2
O(EK,N N
(S.25)
Combining S.22).
(S.23).
S.24..
and S.25
elementary calculation yields
1
7 (m) ZN
(mm)
R
O6K,,N (S.26)
N
jmm-n-t m+1
Setting appropriate m-decomposition m x log N (e.g., m for q 4,
(g-4)lun (0-4)lurwwop /w0 log ), we have loo((1/x)
{k.n A K NX
& TNN
(S.27)
with < min(1, / Wi + wo/ 12 - 1/6]].
Gaussian comparison For a matrix A, denote
|| as the Frobenius norm of A i.e. ||A]| =
(tr(ATAN A)) 1/2
Recall Ω E
Σn T-1 Ziet)
Di ET-1 2,711 /N and
T
1
B
Σ>
(Z. - EZ,))
Σ Σ (Z, - -ZZ.))
N

---

Consider the difference of covariance matrix between NN' and Z based on Frobenius norm, using the fact E = and EZ (m)
EZN.
N
12 - (m)
s E ZN - -/²² (ZN) T + B2N" (m)
ZN -N-""
2 )7)
B (ZN ZN) E (ZN -Zx) =xx'
+
(S.28)
By S.21 and (S.26).
E ZN
Z (m)
(2N)
ZN 2 |- |Z |1 O (Tn.gSK,n Aknx
(S.29)
and similarly, we also have
1 S 1 km || N O 4 (S.30)
EZ
EZ (m)
ZN x
Besides, by (S.20).
E (ZN Σn) E (ZN -ZN)= I S |B[ZN =0
(S.31)
Combining S..29
(..3
(..3).
| (m))
- + T Tnggk,n A knx
(8.32)
By Assumption (A3) and Lemma G.5 we have inf w(s a) Cj and
T-1
Amin (1) 2 Co mmn
50,
TT) 2 0 2
(S.33)
which yields
Amin (I(m)
Amin (S2)
Amin ( (
2
Go
12 (m)
(S.34)
F
Combing
S.32
(s.44
and sub-multiplicatiiity of Frobenius norm, we have
E
G (m) N
GN
IE
(1112 (12mL)
1/2
G
< 1/2 mm) 1/2
=0 (I + T KR am).
(S.35)
By similar arguments in S.27 EK A x am = o(Tn.g) by appropriate m and a, which yields
S.18).
Gaussian approximation Plug n1 = m, 72 = 2m, n2 = 3m, K = N-1/2 into Lemma G.3 with SS.34. we have
KC
(m) N
((()) 0 ( (K (11/4N-1/277 log" n1
(S.36)
Moreover, in the proof of Lemma
G.3 equation (4.23) in Fang (016) yields
E ho.a
(Z(r1) - -ar.
G (m)
]] < Cnncnnnng Kt/A(Ei + n2 + KC (Z(r), (G(()) ]
sup DED
N
by (..3)..
sup
E hooes (m))
ho,es
(G<TT) (m) }] o (K1//A-1/22
1/2 log² N
1 K 1
x log N
DED
E1
(S.37)
12

---

Let = K2K2,, where
K1 =; N 1) 410-1) K
R2 =; No
log log--- N.
(S.38)
Combining (6.16).
(S.17. (S.18) S.19 and S.115
K(ZNN GN) = K + 61
1 1---- ' + NuN 2-20 2g 22
N
+ K N- Tx log (og"")) + K N 3 log" N).
Knn
Therefore, with appropriate 61 and q > have when K o(n2/7--0 for any given 0.
KZN,, GN) Gx)= = (v KiNi ( KK Ka NN "K + K3 N 2 2 log² N) d(1)
(S.39)
Furthermore, combining Lemma G.4 and using the fact K & VN log N, by similar arguments in (E.29) of Shi, Wan, Chernozhukov & Song 2022), one show | E 1-- ||
(1). which yields the validation of the Bootstrap algorithm from the Slutsky's theorem.
G.3 Proof of Proposition 3.2
Proof. It suffices to find C,,, such that as N sc.
P sup T(s) G S Co.N) aa,
(S.40)
where T(s) I(s)/|I(s}] U (s) (s) and G is standard p-dimensional random
vector. Denote manifold
M == (T)) sE
(S.41)
and let Ko the volume of the manifold M, and Co the area of the boundary aM; Let K2 and {) be measures of the curvature of M and 0.M respectively, and mg measures the rotation angles in the regions AMM.
By Proposition 3 in Sun & Loader 1994). for the a in S.40 we have
T((d + 1)/2)
So ((d/2)
P (X2+1 > C2,N) +
P Px3 Ca.N)
a K0
x (1d-1)/2
2 4//2
R2 + C1 + mo r((d - 1)/2)
+
P(x21 > >R,N)
2x
(ld-1)/2
C²
+ +0
Cul- exp
o.N
(S.42)
2
where x2 is the chi-square random variable with the degree of freedom d.
To bound the positive geometric quantities Ko, Co. K2, Ç1. mo appearing in (S.42) we give the follow-
ing formulations for numerical computation. For simplicity, suppose S = [0, 1] and the boundary as consists of those points with exactly one component 0 or 1. The regions where two faces of as Ta(s)) where T, (T())a0r, with (II,..., xxa) and indicator vector e; (e.. ej,a) such that ej.i and E₁ 0
meet are denoted SPS. Denote matrix
(Ti
k + j. By (3.2) and (3.3) in Sun & Loader 1994). the Ko and X2 can be computed as
Kg
det 1/2 (/ A)dss
(S.43)
( ((s)
d(d-1) 1)
R2
(AA AAdds
(S.44)
2
where
d
8(s) - 2 Σ
Jaja (s) Ta (s) (on.
1=2 k=1
13

---

(s, - =ok (ATA) -1 OA T
I- A -ATAT)))ATA))
dzy
For C1, So measuring the boundary 0.M. by (3.4) in Sun & Loader (994) and the second and third equation on Page 1335 of Sun & Loader 1994).
Co / Jss det (AT A.)ds,
(S.45)
51 / (i(s) det (2A (et) A,)ds,
(S.46)
where indicator vector of (eg,1 -ja-1)) such that ei = 1 and Cj,a = 0 1: + j.
d-1
OA Ua(s);
(1()) Σ (e)) A,
Øs;
j=1
Ua(s) x (I - -A(AAAA.) A. = (T)()).... Ta-1
(S.47)
on the face s E OS at which sy is maximized, with similar definitions for (i(s), (s),,, A.. on other faces where sj maximized. Moreover, by the fifth and seventh equations on Page 1335 of Sun & Loader (194)).
mu - mo(s) det 2/2 (A A...A
(S.48)
where
m)(s) = cos (Ud-1(s) (U)=)s))UU(s)))) A.. (s), Td=((s)).
(S.49)
at point at the meeting of the faces sd-1 and sa 1, with similar definitions for mu(s) other meetings of the two faces of as.
Denote (s) = (s) and 0. (s) = (s)) then basic calculation yields
(0,)((s)
(())
S (s))) 10,, (1))
10,, (())
(s))22
(8)]1
|x())12
Using the fact det (AT A) tr A) and tr(A A) Σラ= (s) by (S.43). (S.45).
SS448). Note that [s) 2
Amann (A) (s)] 2 no by condition 3.15). Assumptions (A2) and
(A3), there exists constant C1
> 0 such that
Kg < ( A)
d/2
ds d S /
VV(()) d [0())]
Σ (s)
ds S
ds - O(N9),
and similarly, So = ( mg = O(Na)) since (T.T.A,,) S tT A.) S (A For K2, note that matrix A(A (ATA)--AA is idempotent, thus
aA Jsj
dA
(s))(()) <
(A A) AA
lek
Amin (A A)
0s;
By condition (.15). for some constant C2 > 0,
1
0A
K2 S N
ds + O(N9) = ((Nertea)
Amin (
(ATA)
0s;
thus K2 ( and similarly, we have Ç1 O(N*)) for some constant C3 >
To sum up, there exists constant ē > 0 such that
No. So, K2, S1, mo} = O
(S.50)
14

---

By Theorem 6 in Zhang & Zhou
2020), for constants č, C,C,C' 0, the tail bounds of xa
čexp (CCx)
p(x d x) exp (-Cx) vx x d.
(S.51)
Combining (S.50).
(S.42).
(S.51) and the fact a is a fixed value, we have n' exp( exp(-
a. 2 1,
which implies that Ca.N O(v VVooogg/. N.. On the other hand, by condition (3115). there exists constant > 0, such that
Ko /
4/2 (A A)ds
AATAdds 2 No.
Combining (S.42). SS.5I) and the fact a is a fixed value, we have N ""eep(( exp(-CC S 1, which
shows Co N 2 Vlog N. Combining S..4 3.111. and 3.4), we have appropriate Ca,N = log N
such that 3.16 holds.
References
Bates, S., Hastie, T. & Tibshirani, R. (2024), 'Cross-validation: what does it estimate and how well
does it do Journal of the American Statistical Association 119(546), 1434-1445.
Chen, X. (2007), 'Large sample sieve estimation of semi-nonparametric models', Handbook of Econometrics 6, 5539-5632.
Chen, X. & Christensen, T. M. (2015), 'Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions', Journal of Econometrics 188(2). 447-465.
Daubechies, I. (1988), 'Orthonormal bases of compactly supported wavelets', Communications pure and applied mathematics 41(7), 909-996.
Daubechies. I. (1992), Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics.
Deák, I. (1990), 'Random number generators and simulation', MMathematical methods of operation research.
Fang, X. (2016), 'A Multivariate CLT for Bounded Decomposable Random Vectors with the Best Known Rate', Journal of Theoretical Probability 29(4), 1510-1523.
Fang, X. & Koike, Y. (2024), 'Large-dimensional central limit theorem with bounds convex sets and balls', The Annals of Applied Probability 34(2), 2065 2106.
error
Haar, (1910), 'Zur theorie der orthogonalen funktionensystemee, Mathematische Annalen 69, 331 371.
Hanna, P., Stone, & Niekum, S. (2018), 'Bootstrapping with models: Confidence intervals for off policy evaluation'.
Hanna, J., Stone, P. & Niekum, S. (2017), Bootstrapping with models: Confidence intervals for off- oof-colicy evaluation, in 'Proceedings of the AAAI Conference on Artificial Intelligence', Vol. 31.
Hansen, B. E. (2014), 'Nomparametric sieve regression: Least squares, averaging least squares, and cross-validation..
Hasselt, H. (2010), 'Double q-learning', Advances in neural information processing systems 23.
Jiang, N. & Li, L. (2016), Doubly robust off- policolic value evaluation for reinforcement learning, in "Internaiooal conference on machine learning'. PMLR, pp. 652-661.
Johnson, A., Pollard, T. & Mark III R. (2016), 'Mimic-iii clinical database (version 1.4). physionet. 2016' Available from:
Levine, S., Kumar, A., Tucker; G. Fu, (2020), 'Offline reinforcement learning: Tutorial, review, and perspectives open problems', arXiv preprint arXiv:2005.01643.
15

---

Quan, M. & Lin, Z. (2024). 'Optimal one- pass nonparametric estimation under memory constraint', Journal of the American Statistical Association 119(545), 285-296.
Rodemund, N., Kokoefer, A., Wernly, & Cozowicz, C. (2023), 'Salzburg intensive care database
(sicdb), freely accessible intensive care database', PhysioNet htips://ooi. org/10. 100.00122226//8/////
Rüschendorf, L. de Valk, V. (1993), 'On regression representations of stochastic processes', Stochastic Processes and their Applications 46(2), 183-198.
Shi, C., Wan, R., Chemoozukov, V. & Song, (2021), Deeply-dcbiased off-policy interval estimation, in "International conference on machine learning", PMLR, 9580-9591.
Shi, C.. Zhang, S., Lu, W. & Song, R. (2021). 'Statistical inference of the value function for reinforcement learning in infinite-horizon settings', Journal of the Royal Statistical Sociery Series B: Statistical Methodology 84(3), 765-793.
Sun, J. & Loader, C. R. (1994), 'Simultaneous Confidence Bands for Linear Regression and Smooth- ing', The Annals of Statistics 22(3), 1328 1345.
Timan, A. F. (2014), Theory of approximation of functions of real variable, Elsevier.
Van Der Laan, M. J. & Dudoit, S. (2003), 'Unified cross-validation methodology for selection among estimators and a general cross-validated adaptive epsilon-net estimator: Finite sample oracle inequalities and examples'.
Wiener, N. (1958), Nonlinear Problems in Random Theory, MIT Press, Cambridge, MA.
Wu, W. B. (2005), 'Nonlinear system theory: Another look at dependence', Proceedings of the National Academy of Sciences 102(40), 14150-14154.
WU, W. B. & Mielniczuk, J. (2010), A new look at measuring dependence, in 'Dependence in probability and statistics', Springer, pp. 122-142.
Yves, M. (1989), Ondelettes opérateurs.. I. Ondelettes / Yves Meyer, Actualités mathématiques, Hermann, Paris.
Zhang, A. R. & Zhou, Y. (2020), 'On the non-asymptotic and sharp lower tail bounds of random variables', Stat (1), e314.
16


Reviews:
[{"review_id": "zaphUrVxuR", "reviewer": "Reviewer_MvGP", "summary": "This paper proposes a strategy to calculate simultaneous confidence regions (SCRs) using a boostrapping-based algorithm for use within off-policy evaluation (OPE). The paper first presents the simultaneous inference problem, which can be used to describe developing confidence intervals within OPE, and then provides a theoretical justification that includes a description of linear approximations to Q-learning, and an approximation technique using Gaussian approximation theory. Then, the paper proposes an algorithm that efficiently calculates SCRs using bootstrapping, and finds that across numerical and real-world experiments the identified algorithm improves upon SAVE (Shi et. al).", "questions": "- It would be helpful to have a comprehensive list of assumptions detailed at the beginning of Section 3. For example, including information about the assumed stationarity of the policies etc. better contextualizes the theoretical result. \n- As mentioned in the Weaknesses component earlier in the review, more comprehensive evaluation on the synthetic setting and an altered metric for the OhioT1DM dataset would be more convincing.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper’s strengths and weaknesses are summarized below. \n\nStrengths:\n- This is a very interesting approach to calculating confidence intervals for OPE, in the sense that most existing literature calculates confidence intervals for a point estimate of the value of the target policy. The proposed method appears to be well-justified based on simultaneous inference literature. \n- The paper contains both theoretical analysis supporting the proposed method and the numerical results from the synthetic simulator seem convincing. \n\nWeaknesses:\n- The proposed algorithm's justification is primarily theoretical, and while this is interesting, a more convincing set of empirical results would be helpful. In particular, it would be nice to see Table 1 results across a larger magnitude of possible $n$ and $T$ values, as well as an empirical analysis of the asymptotic nature of the results. For example, as $n$ and $T$ increase, how do SCR and SAVE compare in terms of the empirical coverage probability?\n- The evaluation of the OhioT1DM dataset is somewhat confusing. Is the assumption that $\\pi^{opt}$ should have a higher value than ${b}$ because it is obtained using a double FQI algorithm as opposed to a random forest? Because the reward function is known, a more convincing metric would be the error of the algorithm's estimated $\\hat{V}(\\pi;s)$ for a random sample of states and the length of the confidence interval in comparison to SAVE, or other baselines. \n- I believe there should be more baselines other than SAVE. For example, an IS- or DM- based bootstrapping baseline should be included.", "was_revised": true, "final_justification": "My main concern was regarding the lack of empirical results in the paper, and the insufficient comparison between the proposed approaches and baseline methodologies for the diabetes dataset. However, in the rebuttal, the authors presented more convincing empirical results and clarified that the empirical results reported in the paper for the diabetes dataset. I will increase my score to a 4 to reflect the insights from these new results.", "rebuttal": "We sincerely appreciate your review of our manuscript and your insightful feedback. We are grateful for your recognition of the interest and soundness of our work. In the following, we will address the questions you raised.\n\nQ1. **Regarding consolidating assumptions in Section 3**\n\nWe appreciate your helpful suggestion for improving our paper’s readability. We agree that presenting a consolidated list of assumptions at the beginning of Section 3, which includes policy stationarity and other key conditions, would significantly improve the paper's clarity and help contextualize the theoretical results. While we cannot modify the current NeurIPS submission due to venue constraints, we will incorporate this change in the camera-ready version if accepted, and in any extended/publicly archived versions of the work.\n\nQ2a&W1,W3. **Regarding more comprehensive evaluation on synthetic settings.**\n\nThank you for your question. Following your comments in weakness 1 and 3, We have conducted a broader range of additional simulations based on the settings in Table 1, and the results are shown in Table below. In addition to the original SAVE method with Bonferroni correction compared in the paper, we also present results using SAVE with Sidak correction [1]. The results indicate that the coverage of our method remains around 0.95. moreover, as $n \\times T$ increases, the averaged length (AL) generally shows a decreasing trend.\n\n**Table R1: Results for Scenario 2 across 500 repetitions. Format: ECP(AL).**\n|  n  |  T  | Legendre SCR | Legendre SAVE (Bonferroni) | Legendre SAVE (Sidak) | Spline SCR | Spline SAVE (Bonferroni) | Spline SAVE (Sidak) |\n|-----|-----|--------------|-------------------------------|--------------------------|------------|-----------------------------|------------------------|\n| 30  | 50  | 0.926(8.472) | 0.982(9.445)                  | 0.980(9.431)             | 0.936(9.452) | 0.978(9.793)               | 0.976(9.778)           |\n| 50  | 30  | 0.946(9.553) | 0.970(10.492)                 | 0.968(10.476)            | 0.922(11.101) | 0.942(11.131)              | 0.942(11.115)          |\n| 40  | 50  | 0.924(7.225) | 0.976(8.193)                  | 0.976(8.181)             | 0.938(8.087) | 0.976(8.606)               | 0.974(8.593)           |\n| 50  | 40  | 0.944(7.138) | 0.990(8.136)                  | 0.990(8.124)             | 0.930(7.247) | 0.984(7.753)               | 0.984(7.741)           |\n| 50  | 50  | 0.942(7.299) | 0.978(8.249)                  | 0.978(8.237)             | 0.930(8.156) | 0.966(8.630)               | 0.966(8.617)           |\n| 50  | 150 | 0.952(6.985) | 0.966(7.402)                  | 0.974(7.393)             | 0.934(5.771) | 0.978(6.080)               | 0.972(6.075)           |\n| 50  | 200 | 0.944(5.978) | 0.978(6.436)                  | 0.974(6.435)             | 0.950(7.733) | 0.960(7.418)               | 0.966(7.385)           |\n| 50  | 250 | 0.942(5.957) | 0.968(6.234)                  | 0.952(6.223)             | 0.930(5.177) | 0.960(5.446)               | 0.980(5.431)           |\n| 200 | 70  | 0.934(4.924) | 0.988(5.370)                  | 0.972(5.360)             | 0.926(4.766) | 0.968(5.045)               | 0.978(5.050)           |\n| 250 | 70  | 0.936(4.374) | 0.986(4.800)                  | 0.972(4.804)             | 0.906(4.245) | 0.968(4.521)               | 0.958(4.503)           |\n| 300 | 70  | 0.934(4.430) | 0.974(4.737)                  | 0.986(4.726)             | 0.936(5.029) | 0.978(5.031)               | 0.972(5.020)           |\n   \nQ2b&W2. **Regarding the real data application on OhioT1DM dataset.**\n\nThank you for your question. To clarify, we do not pre-assume that double FQI policy ($\\pi^{opt}$)  outperforms random forest policy ($b$). Instead, our key contribution lies in proposing a simultaneous framework to compare $\\pi^{opt}$ and $b$ across all different initial states, whereas prior works (e.g., [3]) could only assess them pointwisely. Figure 2 in our paper illustrates that, when using $\\pi^{opt}$, the number of states where the value function is significantly non-zero, which indicates that glucose levels outside the normal range (either hypoglycemic or hyperglycemic) are fewer, suggesting that $\\pi^{opt}$ performs better. This conclusion cannot be drawn from pointwise comparison.\n\nWe sincerely thank you for your insightful suggestion to compare the lengths in real data analysis. The true values of the value function under both policies are unknown; however, we can report the difference in average length of the confidence band at the same significance level. Taking $V(\\pi^{opt}, S_0)$ as an example, our method yields an averaged length of 27.0, while SAVE with Bonferroni correction produces an AL of 32.4, which is 20\\% larger than ours. We will include these results in the revised version of the paper. This reply also addresses the weakness 2.\n\n Q2c. **Regarding more baselines other than SAVE.**\n\n Thank you for your question. We would like to clarify that SAVE is a direct method, and as such, we focus on comparing our method with IS. To illustrate, we performed additional simulations by modifying Scenario 2. Specifically, we set $S_0 = (-2 + 0.2i, -2 + 0.2j)^\\top$, where $0 \\leq i, j \\leq 20$. For each combination of $i$ and $j$, we generated 10 trajectories, each of length 10. All other settings remained unchanged. For bootstrapping IS, we used the bootstrap algorithm from [2], which provides confidence intervals for each $V(\\pi, S_0)$. We then applied the Bonferroni correction to obtain the SCB. We present the results in figure (similar to Figure 1 in the main article), and it will be included in the revised version. The results show that the ECP of IS is mostly conservative, although some coverage rates fall below 0.95. As for AL, it is on average longer than our method. These results demonstrate that our approach offers certain advantages over IS. This reply also addresses the weakness 3. \n\n\n\n[1] Abdi, Hervé. Bonferroni and Šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics 3.01 (2007): 2007.\n\n[2] Hanna, Josiah, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.\n\n[3] Shi, C., Zhang, S., Lu, W., \\& Song, R. (2022). Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3), 765-793."}, {"review_id": "vYpA1o7wpB", "reviewer": "Reviewer_HyHA", "summary": "The authors develop a method to provide confidence regions covering the entire state space,\nimproving upon prior work that was restricted to pointwise confidence intervals.", "questions": "- Q1. Can the authors comment on the computational cost involved in constructing their confidence regions?", "limitations": "Methodical limitations are addressed, potential negative societal impact is not.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- The method closes a significant gap in the literature.¹\n- The paper, apart from some typos (see below), is well-written and can be followed easily.\n- All theoretical claims and assumptions are clearly stated and proven.\n- The method is evaluated both on toy data as well as a medical data set.\n \n\n## Weaknesses\n- The method is limited to offline RL (acknowledged by the authors).\n- The authors withhold a public implementation, making it impossible to evaluate the \nquality of the implementation and its usefulness. \nThe claim that withholding the code is justified to avoid it being appropriated is rather \nweak, as the same reasoning applies to the theoretical part of the paper as well.\nIf a reviewer violates confidentiality for code, they will also violate it for the main paper.\n- A discussion of computational cost and other technical constraints is missing.\n\n\n### Minor\n\n#### Typos\n- l88 fun**c**tion\n- l142 sa~~i~~tisfies\n- l184 appro**p**riate\n- l210 implementa**t**ion\n- l254 \\ldots instead of \\cdots\n- l277 method**s**\n \n\n_____\n¹Please note that my familiarity with this area of research is limited. \nTake any statements regarding it with a grain of salt.", "was_revised": true, "final_justification": "I keep my score given the strengths of the paper as outlined above.", "rebuttal": "We sincerely appreciate your careful review of our manuscript and your thoughtful feedback on our work. Thank you for remarking on the clarity of our presentation, theoretical rigor, and experimental validation, as well as for your kind recognition of the novelty of our method in addressing a key gap in the literature. We are also grateful for your attention to detail in identifying typos. Your acknowledgment of the soundness of our methodology and the relevance of our empirical results is greatly encouraging.\n\nW1. **Regarding offline RL limitation.**\n\nWe acknowledge your observation that our method is currently limited to offline RL. This focus aligns with our goal of addressing critical challenges in off-policy evaluation, particularly in safety-sensitive domains where online exploration is costly or impractical (e.g., healthcare, autonomous systems).\n\nW2. **Regarding code for implementation.**\n\nThank you for your very nice question! We have set up a github repository to make the code public, however, due to the rebuttal regulations, we cannot give out links now. We will put a link to the code in the revised version. The datasets used in our experiments are publicly available and cited in the manuscript. We will also incorporate the data in our code to ensure good reproducibility.\n\n\nW3. **Regarding computational cost and technical constraints.**\n\nThank you for your question. In terms of computation, our procedure performs fast and can be completed on a personal laptop. For example, all simulations in this paper were run on an Apple M1 machine with 16GB of RAM and macOS Sonoma. From a theoretical perspective, the term $\\hat{\\beta}_{\\pi}$ in (3.6) is similar to a least squares estimation, which can be efficiently computed. Additionally, Steps 1 and 4 of the bootstrap procedure are both linear, as we use a linear approximation method. Therefore, the computations are efficient. When higher precision is required, we can choose a larger $B$ (bootstrap size) and employ parallel computing. Specifically, the time complexity of our method is $O(NK^2+BK)$ and the space complexity is $O(N+BK)$. This also addresses Question 1 and will be included in our revised paper.\n\nMinor. **Regarding typos**\n\nThank you and we have corrected all the typos you mentioned in the revision. \n\n[1] Abdi, Hervé. \"Bonferroni and Šidák corrections for multiple comparisons.\" Encyclopedia of measurement and statistics 3.01 (2007): 2007."}, {"review_id": "jJJE2d4CVn", "reviewer": "Reviewer_ZaNN", "summary": "This paper tackles the fundamental problem of quantifying uncertainty in Off-Policy Evaluation (OPE) across entire state spaces, moving beyond the limitations of traditional pointwise confidence intervals. This is crucial for deploying RL policies with assured reliability.", "questions": "Please see the weakness.", "limitations": "Please see the weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The framework offers the first theoretically justified simultaneous inference for OPE, providing valid inference over the entire state space and effectively addressing the long-standing multiple testing problem in continuous or infinite state spaces. Its efficiency is notable, with SCR widths exceeding pointwise CIs by only a logarithmic factor $\\sqrt{\\log(N)}$, indicating near-optimal performance.\n2. The multiplier bootstrap algorithm enhances practicality by obviating the need to estimate complex joint distributions, and the flexible scaling factor $L(s)$ allows for customization to specific application needs.\n\nWeakness:\n1. The constraint on $K$ relative to $N$($K = O(N^{2/7}(\\log(N))^{-1})$) creates a tension. Achieving high approximation accuracy for complex Q-functions often demands a large $K$, which might conflict with the rate required for valid asymptotic inference, especially in high-dimensional settings.\n2. The validity of the derived logarithmic rate for SCR width depends on specific geometric properties of the chosen basis functions (conditions 3.15). Verifying these properties for novel or highly complex basis function choices can be challenging in practice.\n3. The \"asymptotically correct\" nature of the framework implies its validity for very large sample sizes. The paper's empirical results are promising, but exploration of its finite-sample behavior under various data characteristics (e.g., small $N$, high noise, non-Gaussian errors) is not presented in the paper.\n4. The paper primarily compares its novel framework against Bonferroni-corrected pointwise confidence intervals. The absence of empirical comparisons against other advanced simultaneous inference techniques (if any exist or emerge in related fields) limits the understanding of its relative performance within the broader state-of-the-art for global inference.", "was_revised": true, "final_justification": "I believe this paper has justified my concerns. I incline to give this paper accept.", "rebuttal": "We sincerely appreciate your thorough review and insightful feedback on our manuscript. Thank you for recognizing the significance of our work in advancing uncertainty quantification for OPE in RL. We are particularly grateful for your acknowledgment of our framework's contributions regarding its efficiency, practical applicability, and flexibility. We have carefully considered all of your comments. In what follows, we address each of the questions you raised in detail.\n\nW1. **Regarding the constraint on $K$.**\n\nThe condition $K=O(N^{2/7}\\log^{-1}N)$ is imposed to ensure the convex Gaussian approximation error in Theorem 3.1 can attain $o(1)$. \nIn contrast to the pointwise CLT ($K=O(N^{1/2}\\log^{-1}N)$), this condition narrows the range of $K$ but does not lead to conflict in most cases. \n\nTake a simple case where the state space $\\mathcal{S}$ is one-dimensional$(d=1)$ and the true Q-function lies in a H\\\"older space with smoothness $p=2$. On the one hand, the mathematical approximation theorem ensures approximation accuracy for the Q-function is of rate $\\epsilon_K=O(K^{-2})$ (details included in Section D.1 of the appendix). On the other hand, our Proposition 3.1 shows our SCR width is of rate $O(N^{-1/2}\\log^{1/2} N)$. Therefore, we wish the $K$ is sufficiently large so that $\\epsilon_K$ is negligible compared to our SCR, i.e., $\\epsilon_K\\ll N^{-1/2}\\log^{1/2} N$. This yields $K\\gg N^{1/4}\\log^{-1/4} N$ and does not conflict with $K=O(N^{2/7}\\log^{-1}N)$ so that we can find a feasible $K=N^{\\alpha}$ for some $\\alpha\\in (1/4,2/7)$ to support valid asymptotic inference. \n\nFor the high-dimensional setting where $d\\to \\infty$, the curse of dimensionality is involved in the asymptotic rate. To the best of our knowledge, constructing confidence intervals/regions for the high-dimensional data in off-policy evaluation is still an open problem, which could be a future work. \n\n\nW2. **Regarding the geometric properties of the chosen basis functions.**\n\nWe appreciate your insightful observation. Indeed, the validity of the logarithmic rate for SCR width relies on the geometric properties specified in Condition (3.15). While these properties can be rigorously verified for commonly used basis functions (e.g., Legendre polynomials, splines, and Fourier series in Section D of the supplement), extending the verification to arbitrary or highly complex bases may require additional substantial theoretical work.\n\nIn practice, most applications employ such standard bases due to their well-characterized properties and computational tractability. Therefore, for the current analysis, we focus on establishing guarantees for those widely adopted bases that possess the geometric properties, and leave the generalizations to broader function classes as future work.\n\nW3. **Regarding the exploration of finite-sample behaviors under various data characteristics.**\n\nThank you for your question.  Taking Scenario 2 (i.e., Table 1) as an illustrative example, we conducted additional simulations for both smaller and larger values of $N$. We kindly refer you to our **second** response to Reviewer MvGP for the results.  Moreover, for high noise and non-Gaussian errors, we modified the state transition rule, specifically the $z_{0,t}$ in line 268, to be an i.i.d. two-dimensional $t(8)$ random variable, while keeping the rest unchanged. The results are shown in Table R2. These results demonstrate that for noise with heavier tails, our coverage and length remain robust.\n\n**Table R2: Results for Scenario 2 with $t(8)$ noises across 500 repetitions.. Format: ECP(AL).**\n\n| $n$  | $T$  | Legendre SCR | Legendre SAVE (Bonferroni) | Legendre SAVE (Sidak) | Spline SCR | Spline SAVE (Bonferroni) | Spline SAVE (Sidak) |\n|------|------|--------------|------------------------------|--------------------------|------------|----------------------------|------------------------|\n| 30   | 50   | 0.940(4.706) | 0.934(4.431)                 | 0.980(4.424)             | 0.936(3.716) | 0.934(3.967)               | 0.986(3.961)           |\n| 50   | 30   | 0.944(2.721) | 0.932(3.316)                 | 0.990(3.311)             | 0.934(3.694) | 0.932(3.959)               | 0.988(3.953)           |\n| 40   | 50   | 0.946(3.349) | 0.934(3.658)                 | 0.992(3.652)             | 0.952(3.632) | 0.950(3.755)               | 0.990(3.749)           |\n| 50   | 40   | 0.932(2.876) | 0.928(3.279)                 | 0.990(3.274)             | 0.944(3.104) | 0.942(3.393)               | 0.988(3.388)           |\n| 50   | 50   | 0.934(3.488) | 0.926(3.448)                 | 0.986(3.443)             | 0.934(2.776) | 0.932(3.048)               | 0.994(3.044)           |\n| 50   | 150  | 0.948(2.333) | 0.944(2.498)                 | 0.988(2.494)             | 0.948(2.415) | 0.944(2.524)               | 0.988(2.520)           |\n| 50   | 200  | 0.942(2.012) | 0.946(2.168)                 | 0.984(2.165)             | 0.936(2.078) | 0.934(2.184)               | 0.988(2.181)           |\n| 50   | 250  | 0.932(1.962) | 0.934(2.111)                 | 0.994(2.108)             | 0.950(2.330) | 0.944(2.332)               | 0.986(2.329)           |\n| 200  | 70   | 0.946(1.970) | 0.996(2.024)                 | 0.996(2.021)             | 0.944(2.176) | 0.944(2.196)               | 0.980(2.192)           |\n| 250  | 70   | 0.948(1.644) | 0.980(1.784)                 | 0.980(1.780)             | 0.944(1.943) | 0.944(1.968)               | 0.982(1.965)           |\n| 300  | 70   | 0.930(1.495) | 0.986(1.625)                 | 0.986(1.623)             | 0.948(1.770) | 0.944(1.800)               | 0.980(1.797)           |\n\n\n\n\n\nW4. **Regarding comparisons against baseline other than Bonferroni correction.**\n\nThank you for your question. In addition to the Bonferroni correction, we also compared our method with SAVE using Sidak correction [1], and the results are presented in Table R1 (we also paste it below for your convenience), and Table R2 in the previous question. The results show that, while ensuring coverage (ECP), our method achieves the shortest averaged length (AL).\n\n**Table R1: Results for Scenario 2 across 500 repetitions.. Format: ECP(AL).**\n\n| $n$  | $T$  | Legendre SCR | Legendre SAVE (Bonferroni) | Legendre SAVE (Sidak) | Spline SCR | Spline SAVE (Bonferroni) | Spline SAVE (Sidak) |\n|------|------|--------------|------------------------------|--------------------------|------------|----------------------------|------------------------|\n| 30   | 50   | 0.926(8.472) | 0.982(9.445)                 | 0.980(9.431)             | 0.936(9.452) | 0.978(9.793)               | 0.976(9.778)           |\n| 50   | 30   | 0.946(9.553) | 0.970(10.492)                | 0.968(10.476)            | 0.922(11.101) | 0.942(11.131)              | 0.942(11.115)          |\n| 40   | 50   | 0.924(7.225) | 0.976(8.193)                 | 0.976(8.181)             | 0.938(8.087) | 0.976(8.606)               | 0.974(8.593)           |\n| 50   | 40   | 0.944(7.138) | 0.990(8.136)                 | 0.990(8.124)             | 0.930(7.247) | 0.984(7.753)               | 0.984(7.741)           |\n| 50   | 50   | 0.942(7.299) | 0.978(8.249)                 | 0.978(8.237)             | 0.930(8.156) | 0.966(8.630)               | 0.966(8.617)           |\n| 50   | 150  | 0.952(6.985) | 0.966(7.402)                 | 0.974(7.393)             | 0.934(5.771) | 0.978(6.080)               | 0.972(6.075)           |\n| 50   | 200  | 0.944(5.978) | 0.978(6.436)                 | 0.974(6.435)             | 0.950(7.733) | 0.960(7.418)               | 0.966(7.385)           |\n| 50   | 250  | 0.942(5.957) | 0.968(6.234)                 | 0.952(6.223)             | 0.930(5.177) | 0.960(5.446)               | 0.980(5.431)           |\n| 200  | 70   | 0.934(4.924) | 0.988(5.370)                 | 0.972(5.360)             | 0.926(4.766) | 0.968(5.045)               | 0.978(5.050)           |\n| 250  | 70   | 0.936(4.374) | 0.986(4.800)                 | 0.972(4.804)             | 0.906(4.245) | 0.968(4.521)               | 0.958(4.503)           |\n| 300  | 70   | 0.934(4.430) | 0.974(4.737)                 | 0.986(4.726)             | 0.936(5.029) | 0.978(5.031)               | 0.972(5.020)           |\n\n\n[1] Abdi, Hervé. \"Bonferroni and Šidák corrections for multiple comparisons.\" Encyclopedia of measurement and statistics 3.01 (2007): 2007."}, {"review_id": "4nJPJrFXNw", "reviewer": "Reviewer_SiUk", "summary": "The paper proposes methods for simultaneous inference for off-policy evaluation in reinforcement learning. The primary problem addressed is the quantification of global uncertainty across an infinite or continuous initial state space. Existing methods primarily focus on point estimates or pointwise confidence intervals (CIs), which, when applied to multiple states inflate the overall significance level, especially in infinite state spaces. The proposed solution is the construction of asymptotically correct simultaneous confidence regions (SCRs) that cover the true policy value functions over the entire state space at a given significance level. The effectiveness of this approach is demonstrated through numerical simulations and analysis of the real-world OhioT1DM dataset", "questions": "1. Can the authors include a detailed literature on offline RL, highlighting the challenges in evaluation and learning? Especially, how to quantify the goodness of the dataset, when is learning possible etc.?\n2. What is $L(s)$ in (3.1)?\n3. How reasonable is it to assume access to an offline dataset in practice that permits learning and evaluation well enough for reliable confidence set computation?\n4. Can the authors elaborate the derivation of (3.13)? \n5. Can the authors explain the intuition behind Assumptions A1-A3.? How reasonable are the assumptions in the offline setting?\n6. Can the authors provide a discussion of Theorem 3.1? The convergence in space-time to the gaussian process somehow gets absorbed into convergence in space? How are the asymptotic guarantees translating into finite time and space results? \n\n\nMinor:\nAssumption on Eq (2.2) is unclear.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The SCRs developed are good in terms of efficiency, with widths exceeding those of pointwise CIs by only a logarithmic factor.\n2. Empirical studies show that the choice and number of basis functions do not impact method's performance, enhancing its practical applicability. \n\n\nWeaknesses:\n1. Key assumptions are strong and unmotivated.\n2. The offline setting is under explained, especially the challenges in offline RL.\n3. It's unclear how to reconcile asymptotic results with the offline setting. The experimental demonstrations are unclear.", "was_revised": true, "final_justification": "The authors clarified all the concerns.", "rebuttal": "We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful feedback. Below, we address each of your comments in detail.\n\nQ1&W2. **Regarding the offline RL, challenges, and the goodness of dataset.**\n\nWe will include [1]-[5] as literature on offline RL and highlight the challenges in offline RL. The following argument will also apear in the revision. Unlike online environments (e.g. Go, MiniGrid), collecting data with online interaction in many real-world applications (e.g. healthcare, autonomous driving) can be expensive or even dangerous. This limitation prevents the widespread adoption of traditional online RL methods. As an alternative, offline RL has been proposed to effectively utilize pre-collected experience data without requiring any online interaction. \n\nHowever, this offline approach faces several challenges that make it different from online RL. For learning, the key challenges are:\n- Distributional shift: The data generated by the behavior policy (used to collect the dataset) might mismatch the target policy (which is learned), which leads to poor performance or overfitting.\n- Sample inefficiency: The learning must rely entirely on the static dataset, which means 'exploration' (as in online RL’s exploration-exploitation trade-off) is not possible during the learning process. \n  \nFor evaluation, offline RL faces the difficulty of **estimating policy performance without online deployment**, necessitating off-policy evaluation (OPE)—the focus of our work. One of the core challenges in OPE is the gap between estimated off-policy evaluation and real-world performance. To address this, recent attempts, including pointwise CIs (as we concluded in the related work section), have been proposed to quantify the gap between offline assessments and real-world deployment through probabilistic measures. Our work strengthens the statistical tools from pointwise to global scale, providing more reliable guarantees for decision-making.\n\nFor the goodness of the dataset, the quality of an offline dataset depends critically on several factors:\n- State-space coverage: Breadth of visited states\n- Reward signal diversity: Range of observed returns\n- Action variety: Exploration of possible actions\n-  Algorithm compatibility: Match between data distribution and RL method\n\nAs noted in [2], formalizing a non-trivial \"sufficiency\" condition for dataset quality remains an open problem. In our framework, we distill these complex considerations into Assumptions (A1)-(A3) (detailed in response Q5&W1). Under these assumptions, the goodness of the dataset can be roughly quantified by the sample size $N$. This simplification enables tractable theoretical analysis.\n\nQ2. **Regarding the $L(s)$ in (3.1).**\n\n$L(s)$ is a user-chosen scaling factor which shapes our SCR for different states $s\\in\\mathcal{S}$ ($L(s)>0$), especially the width. CIs in OPE consider $L(s)=\\sqrt{U_\\pi(s)^\\top \\Lambda_\\pi U_\\pi(s)}$ since the CLT theorem gaurantees $\\sqrt{N}(\\hat{V}(\\pi;s)-V(\\pi;s))/L(s)\\to_d N(0,1)$ for fixed $s$. \n\nIn our framework, the choice of $L(s)$ can be very flexible ($L(s)=\\sqrt{U_\\pi(s)^\\top \\Lambda_\\pi U_\\pi(s)}$ or simply $L(s)=1$). This is achieved by applying a stronger convex Gaussian approximation theorem (Theorem 3.1) instead of the simple CLT (the detailed explanation is put in response Q6&W3). \n\nQ3. **Regarding the accessibility of the offline dataset for reliable results.**\n\nOur theorems ensure that the SCRs are reliable when the sample size $N$ is sufficiently large. A large offline dataset is accessible in many real-world settings since\n- Offline RL leverages existing datasets (e.g., possibly historical logs, human demonstrations, or prior experiments).\n- Many domains (healthcare, education) have abundant offline data.\n\nAdditionally, our theoretical reliability is built upon our Assumptions, which are mild for most cases and no stronger than those in serial theoretical work on offline RL (detailed in response Q5&W1). Thank you again for your comment, and the discussion on the access to offline data will be included in our revised paper.\n\nQ4. **Regarding the derivation of (3.13).**\n\nOn the one hand, we write the probability in the first line of (3.13) by a more specific formulation:\n$$\n\\mathrm{P}\\left( \\mathbf{Z}\\_\\pi\\in \\left\\\\{\\theta\\in \\mathbb{R}^{mK}: \\left|\\Phi(s)^{\\top} \\sum_{a \\in \\mathcal{A}} \\pi(a \\mid s) \\theta\\right| \\leq C_{\\alpha, N} L(s), \\forall s \\in \\mathcal{S}\\right\\\\} \\right)\n$$\nOn the other hand, the second line in (3.13) can be formulated as\n$$\n\\mathrm{P}\\left( \\mathbf{Z}\\_\\pi\\in \\left\\\\{ \\theta\\in \\mathbb{R}^{mK}:  \\sup_{s \\in \\mathcal{S}}\\left|\\frac{\\Phi(s)^{\\top} \\sum_{a \\in \\mathcal{A}} \\pi(a \\mid s) \\theta}{L(s)}\\right| \\leq C_{\\alpha, N} \\right\\\\} \\right)\n$$\nNote that $L(s)>0$ and one can verify\n$$\n\\left|\\Phi(s)^{\\top} \\sum_{a \\in \\mathcal{A}} \\pi(a \\mid s) \\theta\\right| \\leq C_{\\alpha, N} L(s), \\forall s \\in \\mathcal{S} \\iff \\sup_{s \\in \\mathcal{S}}\\left|\\frac{\\Phi(s)^{\\top} \\sum_{a \\in \\mathcal{A}} \\pi(a \\mid s) \\theta}{L(s)}\\right| \\leq C_{\\alpha, N},\n$$\nthus, the derivation in (3.13) is correct.\n\nQ5&W1. **Regarding the intuition and rationality for Assumptions A1-A3.**\n\nAssumptions A1-A3 are very mild assumptions and serve as the minimal requirement for the goodness of the offline dataset to support feasible evaluation. \n- Assumption (A1) requires the density $\\mu(s)$ for the invariant distribution to be uniformly bounded away from $0$ and $\\infty$ so that the Markov chain would not be trapped in a small subset of the entire space. The density $\\nu_0(s)$ for the initial state should also follow the boundary condition so that every state is possible to be the initial state.\n- Assumption (A2) relaxes the condition on sample size. Previous work (e.g., [5]) requires the number of trajectories $n\\to\\infty$. (A2) additionally allows fixed $n$, but length $T\\to \\infty$ when the action variety is sufficiently large on each chain. The geometrical ergodic for the Markov chain is technical for theoretical deduction and commonly assumed for deriving limit theory.\n- Assumption (A3) requires the reward signal diversity. The $\\omega_\\pi(s,a)\\geq c_0^{-1}$ requires that the reward random variable is nondegenerate (not always the same). $P(\\max_{0\\leq t\\leq T-1}|R_{0,t}|\\leq c_0)=1$ means the rewards are bounded. \n\nThese assumptions are the same as those needed to establish pointwise CIs in OPE as discussed in Remark 3.1.\n\nQ6&W3. **Regarding the discussion of Theorem 3.1.**\n\nTheorem 3.1 is our main theorem, which provides a higher convex Gaussian approximation towards $\\hat{\\theta}\\_\\pi=\\hat{\\beta}\\_\\pi-\\hat{\\beta}\\_\\pi\\^*$ in the OPE estimation error $\\hat{V}(\\pi;s)-V(\\pi;s)=\\Phi(s)^\\top \\sum_{a\\in\\mathcal{A}}\\pi(a|s)\\hat{\\theta}\\_\\pi$.\nThis is a different approach from the previous pointwise CI results obtained by the simple CLT. The CLT focuses on deriving a limiting theorem on the **distribution function** of inner product value $\\Phi(s)^\\top \\sum_{a\\in\\mathcal{A}}\\pi(a|s)\\hat{\\theta}\\_\\pi$ for fixed $s\\in\\mathcal{S}$. The generalization from fixed $s\\in\\mathcal{S}$ to arbitrary $s\\in\\mathcal{S}$ is not straightforward, since one needs to control\n$$\n    \\left|\\mathrm{P}\\left(\\left|\\frac{\\Phi(s)^\\top \\sum_{a\\in\\mathcal{A}}\\pi(a|s)\\sqrt{N}\\hat{\\theta}\\_\\pi}{L(s)}\\right|\\leq t,\\forall s\\in\\mathcal{S}\\right) - \\mathrm P\\left(\\left|\\frac{\\Phi(s)^\\top \\sum_{a\\in\\mathcal{A}}\\pi(a|s)\\mathbf{Z}\\_\\pi}{L(s)}\\right|\\leq t,\\forall s\\in\\mathcal{S}\\right)\\right|\n$$\nwhere $\\mathbf Z\\_\\pi$ is a random Gaussian vector.\nNote that for both scenarios of finite collection of $s$  and all $s\\in \\mathcal S$, the above can be rewritten as $\\Delta\\_{\\mathbb O}=\\left|\\mathrm{P}\\left(\\sqrt{N} \\hat{\\theta}\\_\\pi \\in \\mathbb{O}\\right)-\\mathrm{P}\\left(\\mathbf{Z}\\_\\pi \\in \\mathbb{O}\\right)\\right|$ for some covex region $\\mathbb O$ (see Remark 3.2 for details). When considering finite sample properties, we provide a bound concerning sample size $N$ and basis number $K$ from the proof of Theorem 3.1 (details in (S.37), (S.38), and (S.39) in the supplement),\n$$\n\\sup_{\\mathbb{O} \\in \\text{all convex sets} } \\Delta\\_{\\mathbb{O}} \n\\leq C\\left(\\sqrt{K^{\\frac{1}{4}} N^{\\frac{1}{2}} \\pi_N^{1-q} \\xi_{K, N}^q}+K^{\\frac{1}{8}} N^{\\frac{1}{2}} \\pi_N^{1-q} \\xi_{K, N}^q+K^{\\frac{1}{4}} N^{-\\frac{1}{2}} \\pi_N^3 \\log ^2 N\\right).\n$$\nIn the proof of Theorem 3.1, we also verify that the bound can converge to $0$ as $N\\to \\infty$ when $K$ is smaller than $N^{2/7}$. In this way we address the space-time issues in the Gaussian process in Theorem 3.1. Our convex Gaussian approximation focuses on the probability of the random vector $\\hat{\\theta}_\\pi$ **on all convex sets** in $\\mathbb{R}^{mK}$.  We will add a new remark in revision for the explanation.\n\nW3, part II. **Regarding the experimental demonstrations.**\n\nDue to the page limit, please kindly see our Q2bW2 response to Reviewer MvGP for our demonstration on OhioT1DM experiment.\n\nMinor. **Regarding Assumption on Eq (2.2).**\n\nEq (2.2) assumes the conditional mean of the reward $R_{0,t}$ is independent of $(S\\_{0,k},R\\_{0,k},A\\_{0,k})\\_{k<t}$, i.e.\n$$\nE(R\\_{0,t}|S\\_{0,t}=s,A\\_{0,t}=a,(S\\_{0,k},R\\_{0,k},A\\_{0,k})\\_{k<t})=E(R\\_{0,t}|S\\_{0,t}=s,A\\_{0,t}=a)=r(s,a)\n$$\nfor some reward function $r$.\n\n[1] Fujimoto S, Gu S S. A minimalist approach to offline reinforcement learning. \n\n[2] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\n\n[3] Prudencio R F, Maximo M R O A, Colombini E L. A survey on offline reinforcement learning: Taxonomy, review, and open problems.\n\n[4] Shi, C. (2025). Statistical inference in reinforcement learning: A selective survey.\n\n[5] Jiang, N. Li, L. (2016) Doubly robust off-policy value evaluation for reinforcement learning."}]

Author rebuttal:

Meta-review: The paper presents a method for simultaneous inference in off-policy evaluation (OPE) in reinforcement learning. The goal is to quantify uncertainty estimates across an infinite or continuous initial state space, rather than focusing solely on point estimates, pointwise confidence intervals (CIs), or averages with respect to the initial state distribution. The proposed solution is to construct asymptotically correct simultaneous confidence regions (SCRs) that cover the true policy value functions over the entire state space at a prescribed significance level. Results are illustrated through numerical experiments on both synthetic and real-world datasets.

The paper is novel and interesting, as acknowledged by all reviewers. However, the manuscript can be improved. We recommend incorporating the following points raised during the rebuttal phase:

Presentation: The preliminaries are standard and unnecessarily long. Section 2 and part of Section 3.1 should be streamlined.

Numerical experiments: Include additional baselines, as suggested for instance by Reviewer MvGP.

Computational complexity: Provide comments on the computational cost of the proposed method.

Finite-time analysis: Discuss possible extensions of the results to finite-time settings and add references to the corresponding related work.

Related work: This section is currently too limited and should be substantially expanded. There exists a large body of work on OPE, in particular on constructing confidence intervals for value functions averaged over initial states. For example, see Duan & Wang (ICML 2020), Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation. The paper should clarify what is missing in such results to extend them technically to the simultaneous inference framework. A discussion of finite-time guarantees is also needed.

Overall, while the paper makes a promising and original contribution, its presentation and positioning within the broader OPE literature must be improved to strengthen its impact.
Decision: Accept (poster)