A Unifying View of Coverage in Linear Off-policy Evaluation
Authors: Philip Amortila
Audrey Huang
Akshay Krishnamurthy
Nan Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: off-policy evaluation
reinforcement learning
coverage
Primary area: learning theory

Abstract: Off-policy evaluation (OPE) is a fundamental task in reinforcement learning (RL). In the classic setting of \emph{linear OPE}, finite-sample guarantees often take the form
$$
\textrm{Prediction error} \le \textrm{poly}(C^\pi, d, 1/n, log(1/\delta)),
$$
where $d$ is the dimension of the features, and $C^\pi$ is a **_feature coverage parameter_** that characterizes the degree to which the visited features lie in the span of the data distribution. While such guarantees are well-understood for several popular algorithms under stronger assumptions (e.g. Bellman completeness), the understanding is lacking and fragmented in the minimal setting where the target value function is linearly realizable in the features. Despite recent interest in tight characterizations of the statistical rate in this setting, the right notion of coverage remains unclear, and candidate definitions from prior analyses have undesirable properties and are starkly disconnected from more standard definitions in the literature.  

We provide a novel finite-sample analysis of a canonical algorithm for this setting, LSTDQ. Inspired by an instrumental-variable view, we develop error bounds that depend on a novel coverage parameter, the **feature-dynamics coverage**, which can be interpreted as linear coverage in an induced dynamical system for feature evolution. With further assumptions---such as Bellman-completeness---our definition successfully recovers the coverage parameters specialized to those settings, finally yielding a unified understanding for coverage in linear OPE.

Published as a conference paper at ICLR 2026
A UNIFYING VIEW OF COVERAGE IN LINEAR OFF- POLICY EVALUATION
Philip Amortila
Audrey Huang
University of California, Berkeley p. samorttla@berreleyy.coo
University of Illinois Urbana- Chaamapgn audreyh5@i ilinois.
Akshay Krishnamurthy Microsoft Research, NYC akshaykremiccrooof..cm
Nan Jiang
University of Illinois Urbana-Champaign
monjiangeillinon edu
ABSTRACT
Off-policy evaluation (OPE) is a fundamental task in reinforcement learning (RL). In the classic setting of linear OPE, finite-sample guarantees often take the form
Evaluation
S poly(C", dd, 1/n, log(1/6)).
where d is the dimension of the features and C" is a coverage parameter that characterizes the degree to which the visited features lie in the span of the data distribution. While such guarantces are well-understood for several popular algo- rithms under stronger assumptions (e.g. Bellman completeness), the understanding is lacking and fragmented in the minimal setting where the target value function is linearly realizable in the features. Despite recent interest tight characterizations of the statistical rate in this setting, the right notion of coverage remains unclear, and candidate definitions from prior analyses have undesirable properties and are starkly disconnected from more standard definitions in the literature.
We provide novel sample analysis of canonical algorithm for this setting, LSTDQ. Inspired by an instrumental-variatee that depend which can be interpreted as linear coverage in an induced dynamical system for feature evolution. With further assumptions- -ushonsssssshhhh---- as cellman-completnna our definition successfully recovers the coverage parameters specialized to those settings, finally yielding unified understanding for coverage in linear OPE.
view, we develop error bounds novel coverage parameter, the feature-dynamics coverage,
INTRODUCTION
Coverage foundational concept in reinforcement learning (RL) theory. In off-policy evaluation (OPE), the task of evaluating target policy based on data collected from different behavior policy, coverage characterizes the degree to which the data distribution contains relevant information about the target policy. The relevance of coverage extends beyond OPE, and the concept plays important roles offline policy learning (Jin et al.) 2021] Xie et al. (002). online RL (Xie et al. 2023 Amortila
al. 2024blc). even cwwiitiicll-cmpuaattiinin trade-offs in LLMs (Foster et al.) 2025). Coverage
provides a mathematical characterization of distribution shift which is a central challenge in RL.
Mathematically, coverage is manifested coverage parameters in finite-sample guarantees: for example, standard OPE guarantees often take the form
Evaluation error < poly(C"", dd, 1/n, log(1/6)).
(I)
where n is sample size, ò the failure probability, is the statistical dimension of the function class, 8 is the failure probability, and C is the coverage parameter. The definition of C can take many different forms depending on the algorithm and the assumptions, as well as how the proof handles error propagation through the dynamics of the MDP (Farahmand al.] 2010). The most naïve definition is ||_T" // bl the boundedness of density ratio between the target policy's 's discounted occupancy f² and the data distribution D More refined definitions often take advantage of the

---

Published as a conference paper at ICLR 2026
structure of the underlying MDP or the funetion-approxxine definitions offer connections and unified understanding across different learning settings, such as policy evaluation vs. optimization (Duan & Wang/ 2020 Jin et al. (202I), offline online (Xie et al.
scheme. Comparisons between these
2023), tabular vs. function approximation (Yin & Wang 2021). ). Markovian vs. partially observed
(Zhang & Jiang 2024), and single-agent vs. multi-agent RI (Cui & Du 2022 Zhang et al. 2023).
While a unified understanding of RL through the lens of coverage is emerging, one of the most fundamental Seerings-where the target value function is linearly realizable in given feature map- eludes such understanding and remains starkly disconnected from the rest of the literature. The most natural algorithm for this setting arguably LSTD(Q) (Boyan 1999] Lagoudakis & Parr 2003). Despite recent statistical results for this method (Duan et al. 2021; Mou et al. 2022a [Perdomo et al. 2023), the error bounds often come with obscure conditions and are hard to interpret, with little or no understanding of which quantities play the role of coverage and how they connect to coverage parameters related settings and algorithms.
On the other hand, simpler analyses do provide more interpretable candidates, such as 1//minn/Aaaaaa/ (A) with A = En (,,s, ()(((, a) 70(s', ④) being the key matrix estimated in LSTDQ (ectton 2.1 Given that LSTDQ approximates Q = o by solving a linear equation in the form of Ae = b. 1 1/0min (A) is a very natural candidate as it determines the invertibility of A and the solution's numerical stability. While bounds in the form of Eq (1) can be established with C* 1//0min (A), the quantity m/m (A) is unsatisfactory in many aspects as a coverage parameter:
1. Lacking scale invariance. The value of min (A) can change arbitrarily if simply redefine the features as Onew = co [ While seemingly unrelated, this issue is mathematically tied to the fact that miin (A) as a coverage parameter has concern over the initial state distribution of the MDP which should play an important role in the definition of coverage.
2. Lacking off- off-policy characterization. Coverage parameters provide important understanding for when data contains relevant information about the target policy. For 1/ (A), however, the only known characterization to our knowledge is its boundedness in the restrictive on-policy case, and its magnitude is hard to interpret for general off-policy distributions.
3. Lacking unification with other analyses. State abstractions are special case of linear function approximation, under which LSTDQ coincides with the model-based solution. Prior works have established aggregated concentrability (Jia et al., 2024) as the appropriate coverage parameter for this setting, which cannot be recovered by specializing 1 1/ Moreover, both concepts differ significantly from standard definitions of coverage in linear OPE when analyzed under the Bellman- Bellman-completenee completeness assumption: standard definitions measure coverage by analyzing how errors propagate under the groundtruth dynamics, whereas aggregated concentrability does so under the compressed dynamics determined by the abstraction scheme.
In this paper, we provide a novel finite- finite-sampll sample analysis of LSTDQ inspired by an instrument-varian (IV) view (Bradtke & Barto 1996 Chen et al. 2022), which comes with new coverage parameter that we call feature-dynamte elegantly addresses the above problems. Furthermore, it corresponds to feature coverage in linear dynamical system induced by the features (first studied by Parr et al. (2008)). The system the transition dynamics of the true MDP compressed through the given features, and naturally subsumes the x² version of aggregated concentrability as a special case. Furthermore, given Bellman- completeness as an additional assumption, Ceature-dynamio coverage recovers the standard notion of linear coverage, successfully unifying the previously fragmented understanding.
coverage, C Feature-dynamics coverage replaces m/m (A) and
2 PRELIMINARIES
Markoy Decision Process (MDP). We consider the groundtruth environment modeled as an infinite-horizon discounted MDP A, P. R, 7, Ho), where S is the state space, A is the action space, P : S x A - ASS) is the transition dynamics (A(.) is the probability simplex), R S x A - A ([0, Rmax]) the reward function, 7 E [0, 1) the discount factor, and po A(S) is the initial state distribution. We assume S, A are finite, but their cardinalities can be prohibitively large
Perdomo et al. 2003)) provide a bound that depends term related to our coverage parameter and is scale invariant. We will compare and connect their results in Section 4
2

---

Published as a conference paper at ICLR 2026
and thus should not appear in sample-complexity guarantees. A policy S S A(A) induces a distribution over random trajectories, generated as So ~ Mo. at ~ T(|(st) (or simply ay ~ TT, rr R(s,, az), St+1 ~ P(-1s,, az). Let P+[ and E, denote the probability and expectation under such a distribution. The expected return of policy is J(x) == E,z 12==0 7're]. which falls in the range of (0, Vmax] with Vmax= = Rmax/ (1 7). The discounted occupancy of a is defined
n² (s) = 7) E120 12 ui (s, a) = (1 -)) Er / 8, Or a).
(2)
Value Function and Bellman Operator The Q-function Q" E RSXA is the fixed point of Bellman operator T SxA + RSXA i.e., Q T=Q", where vʃ E RSSA, (T*f)(s, a)
R(s, + t. (f(ss,))((((ss,s)))))))) (s', =)]. Here f(s', T) is shorthand for Ex' f E Sxx as an approximation of Q", we can induce an estimate of J(TT as:
.')]. Given any
(T(T) == PP -= (So, an)];
since ((x] Jo (z). Here we assume μi is known for simplicity, and our results extend straight- forwardly to the case where 140 is unknown and needs to be estimated from separate dataset.
Linear Off-policy Evaluation (OPE). OPE is the task of estimating the performance of given target policy т based on an offline dataset D sampled from a behavior policy Th. As standard simplification, We assume that D consists of i.i.d. tuples (s,a,r, 8', a') generated
(8,a) ~ pr.rr . R(s,a), 8' (-1ss,)), ))),a' ~ + ('')
We use E, [-] denote the expectation of functions of as,,,,,'', a') under the data distribution, and Ep[:] denotes the empirical approximation from D. For most of the paper we are concerned with return estimation via linear function approximation, i.e., estimating the scalar [(x) Jā- (x)
where (s, a) o(s, a) T for some given feature map : S A Rd. We make the following standard assumptions throughout the paper:
Assumption 1 (Feature boundedness and realizability). We assume that there exists 0 E Id such that Q* a) = 6(8 a) 0*, Furthermore, assume that [ a)ll2 S Bo, Vs, a.
Mathematical Notation. We use minn ()) and Xmnn (-) to denote the smallest singular value of an asymmetric matrix and the smallest eigenvalue of a symmetric matrix, respectively. Let p(-) denote the spectral radius of a matrix. For functions over S x A such Q= and d', we also view them interchangeably as vectors in R|Sx whenever convenient. We use S. b shorthand for a = e- b for some absolute constant c, and use big-Oh notation (e.g., o(1/n) and O(1 (n)) to highlight dependence on while ignoring other problem-dependent quantities. Given two square and possibly asymmetric matrices Σ and Σ', Σ < E' means ( E')o S 0 for all v. We let |/ll|s = VuT Ev denote the Mahalanobis norm, and [K] = fl...., K}.
2.1 LSTDQ
The LSTDQ algorithm estimates the following moments from data: == (o(s,, a))(s,)())))))))
Ver = E ((() Epo((((,
A == -
Throughout the paper, assume:
Assumption (Invertibility). Σ and A are invertible.
These moments satisfy
A0 - b = E,olo(s, () ()(((()(())))))))))))) -r o(s - =E, (s. - (T*Q*)
00
which implies 0 = - if A invertible, and LSTDQ is simply the plug-in estimate of this inverse: let E, Se,, A, be the empirical estimates from D and
bbb
0 = A-1ō, Quta a) = 0(s, a) T 0maa.
"When estimating E.c. either taking the average of ((s, (00,((s',)) a²)T a)8,s) is fine and our results apply to both versions.
3

---

Published as a conference paper at ICLR 2026
We do not explicitly assume the empirical matrices E and À are invertible, with the understanding that algorithms based on these quantities have degenerate behaviors when they are singular, and our guarantees hold under such convention
RELATED WORKS
Coverage Parameters in Offliin RL. Early research in offline RL identifies the boundedness of
density ratios, such as | "////Pllo lloo. as a coverage parameter (Munos 2007 Munos & Szepesvári 2008 Antos et al. 2008). Later works point out that they can be lightened by leveraging the structure
(E.+ ff.= f1}" In
of the function class F. used to approximate the value function, such as supjer [_-
our setting, the linear feature á induces a linear Fe = T 0 E R. and these parameters often have simplified forms. Among them, the tightest known coverage parameter for off-policy return estimation is (Zanette et al. [021] Yin et al. 2022; Gabbianelli et al. 2024 Jiang & Xie (024):
C = ---))~-- where d=
E (s.a))pp (((s,)))
In words, C only requires the mean feature vector under the discounted occupancy T (a",, cf. lie in the span of data features. Looser alternatives have also been proposed, such as E pm y-1 require coverage of the distribution Φ(s, a). (s, a) μ²" in point-wise manner; see Jiang & Xie 2024) for further discussions. The majority of the study on coverage, however, crucially
Eq.
(2)
relies on the following assumption on F which is substantially stronger than realizability (Q" E
and we do not assume it in our main results unless stated otherwise explicitly.
Assumption 3 Beellman-compleeeness).. Let F C (SxA R) be functio class for approximating Q². We say that it is Bellman-complete if
TIEEVEER.
Indeed, a major potential of LSTDQ is that it is one of the few algorithms who enjoy theoretical guarantees under only realizability Q F, which further enables its role in the challenging problem of offline model selection (Xie & Jiang 2020a Liu et al., 2025). Unfortunately, coverage in the absence of Bellman-commleteen
is poorly understood in the linear setting, discussed in which we address in this paper.
Section I
LSTD(Q) Analyses. LSTD methods are initially derived as the fixed point solution of TD methods (Sutton & Barto 2018), and its closed-form nature separates from typical dynamical-prorramming style RL algorithms that often suffer from divergence. While we focus on LSTDQ, our results naturally extend to other variants such LSTD (which uses state-feature to approximate V*) or off-policy LSTD (up to handling importance sampling via concentration inequalities) (Nedić & Bertsekas 2003 Bertsekas & Yu 2009 Dann al. 2014).
Early finite-sample analyses of LSTD are mostly in the on-policy setting and depend on quantities
like mmin (A) (Bertsekas) 2007 Lazaric al. 2010 2012). There is recent surge of interest in
providing tight statistical characterizations of LSTD. including in the off-policy setting (Amortila
al. 22020] Mou al.) 2022b Amortila al., 2023 Perdomo et al. (202)). These results, however, offer very little discussion on coverage, and sometimes require additional regularity assumptions.
Perdomo al. (2023) recently provides sharp analysis that largely subsumes the earlier results.
which we will compare to in Section 5.1
4 FINITE-SAMPLY ANALYSIS OF LSTDQ
In this section we will present the finite-sample analysis of LSTDQ, which depends on our proposed coverage parameter.
Thhat is, either the high-prohayility event guarantees that the empirical matrix is invertible, or such matrices appears in the bound and make the guarantee vacuous (as in e.g., g.. [Fq. (12)
4

---

Published as a conference paper at ICLR 2026
Special case of 0. It is instructive to start with the special case of 0, where tight guarantees and the definition of coverage are well understood. Recall that A#* = b can be rewritten as:
E[ZX
100 = E[ZY,,
(6)
where Z =
=ds, ((s,a) E TR. x ,0(s', a') E IR. = IR, and the expectation E[]
under J.P Below we will go back and forth between the linear regression (LR) notation system (X, Y, EE [XXXX""]
and the RL notation system (Φ. r. E....,, where we obtain concentration bounds
from the LR literature and meaningful guarantees for OPE in the RL setting, respectively.
When O, we essentially face a contextual bandit problem with linear reward, and Eq. (6) becomes
E[X 10 [[XY],
which is a classic linear regression problem, with A Σ ] In this special case, LSTDQ simply performs LR to fit the parameter 0*, While parameter identification guarantees in LR (i.c.,
error bounds on | 0* (]) inevitably have to depend on (A) Amin ( D). the key is in
how we Pld to form the final estimation in OPE:
JQuad (T) E =Es0 =000p0000===0(9 ag) T 0 000ds
where
Φ Poo (0(0, on)]-
(8)
That is, for the purpose of estimating Jmx), we only need Olsad to be accurate in the direction of dn. and high-probability bound can be established if do is well covered by the distribution of X 0(8, a) observed in data ((s, a) ~ P): informally, with probability at least 1 5, 5,
(1/0) Vmaa-
@ (T) - [(T] = 10b (orad 0*) S |1ool|--1
(9)
Here | plays the role of coverage, characterizing how well the expected feature under the target policy æ is covered by the random features observed in the data (which determines E) Similar bounds with the population version of coverage ||/bol x- also hold under additional regularity assumptions (Hsu et al. 2011). The quantity is also consistent with the standard notion of linear coverage in MDPs (Eq. (5)) under Bellman completeness [Assumption 3), which is equivalent to
realizability in the bandit setting (n my=00 0).
For readers familiar with using L.R bounds calculate uncertainty bonuses in RL., this result might come as surprise as dimension-free. whereas the usual bound has / (1/)))) which depends on (Abbasi-Yadii) Yadkori et al. [011 Jin et al.) 2020 (202]) Roughly speaking, this is because the latter bound holds for all possible Φu IR" simultancously, but OPE we only care about single given do, and leveraging this saves the d factor, we call this kind of results directional bounds and provide more detailed discussions Appendix
Extending to 7 0 with Instrumental-Vartie Variable inspiration. Given that the = 0 case is well-understood and does not suffer the issues mentioned the introduction, we therefore seek to extend the above framework to y O. When y > however, have Z * X, and (Bq (6) is form of Instrumental Variable (IV) problem induced by
issues: it is known that
R(s, a) = 0u a) 8*, with (s, == Φ(s, a) 7 Ey, [o(s', "as"","))) a')ls,a],
that is, the expected leverages to recover 8* However, in the data we do not observe the expected TD feature but its random realization, X = Φ(8, a) - 70(s' a"), and X Aa a) is zero-mean (conditioned on (s, a)) noise. Given such "error in the variable". X * E[XY]. so straightforward linear regression from to Y does not work. LSTDQ solves this problem by introducing Z = o(s,, a) as instrumental variable (Bradtke & Barto 1996 Chen et al. 2022). which is independent of the noise - (s, (s, a) given (s, a) and thus helps marginalizes out the said noise. Based on this view, extend the LR analysis of 0 to the 0 case by consulting the IV literature (c.g., Xia et al. 2024; Della Vecchia & Basu 2025). which leads to our main maite-sample error bounds (see Appendix B for the proof).
difference feature. dea can linearly predict reward. which LSTDQ
5

---

Published as a conference paper at ICLR 2026
Theorem (Population Coverage Bound). There exists no such that when 2 no, w.p. 2 8,
V maxx
C log(1/6)
J@maa (T) [(TT]
o( 1/1),
(10)
n
where
=1 E 10
(11)
and no and the o(1/vm) term may depend on d and 1/0min mA)..
Theorem 2 (Empirical Coverage Bound). Under Anndeeeissmptionss 7 and with probability at least
8,
T (d log(1/8))
Viax
JQua (T) - ((T) 2
(12)
n
where
0% :1- TAAAA-AAAA
20.
(13)
We treat Oz = +o0 if not invertible.
Coverage Parameter and Tightness. Our main results consist of two guarantees where Cg and C play the role of the coverage parameters. The (1 - > are normalization constants, whose roles will become clear in Section Eq. provides bound that largely recovers that of linear regression in
Eq. (9); when 7 = O, we have A = E, and therefore C = %0 Σ 6o
| 1
which is the (often more desirable) population version of the | term in Eq. (9) Since Eq I9)is is well-established for linear regression (and already tighter than commonly used LR bound in terms of d dependence), this demonstrates the tightness of our bound in the 0 regime up the
lower-order term. When 7 > 0, our bound also tight when compared to existing OPE guarantees for general function approximation analyzed under Bellman completeness as will see in /Section 5.3)
caveat of Eq. (10) is that it still depends on quantities like miin albeit in the lower-order term (o(1/ /n)) and the burn-in condition (n > Ro), raising the suspicion that m (A) is still relevant to coverage. To address this, we provide Eq. (12) that depends on the empirical coverage parameter Öz similar to Eq. (9), with no lower-order terms whatsoever, showing that Jmin (A) can be completely removed from the bound. [Eq. (12) is looser than E.q. (10) in d dependence, as it provides guarantee that holds for all possible initial distribution 140 simultaneously which can be used to establish
guarantees (Appendix D): see the discussion of directional bounds below Eq. and in Appendix A In [Appendix EJ we provide yet another result that eliminates the dependence on 1/Omiia (A) in the population C bound (Theorem 1}. in exchange for dependence on standard quantities relevant to the analysis of LR under random design (namely 1/ Amii (E), which
we expect can further sharpened to leverage-score-eee conditions (Hsu et al. 201 Perdomo
al. 2023
UNDERSTANDING THE COVERAGE PARAMETER
In this section we provide interpretations of C coverage parameter and discuss how it addresses the issues mentioned in Section 1 First, it is clear that Cg is invariant feature rescaling, thanks to the introduction of Φb. That said, the expression C = (1 2) A Φ does not lend itself to easy intuition, let alone how it connects to and unifies existing results.
Warm-up: the tabular case. We start with the tabular setting and show that C becomes something familiar, offering some basic intuitions as well as assurance that C a quantity that falls out of the IV concentration analyses, holds meaningful interpretations in RL. The key is rewrite C 7) as
80 A áu 0o (I ----B=*' - B*) - '0n
where B ==:= :====-1 ( (14)

---

Published as a conference paper at ICLR 2026
The tabular setting can be viewed special case of linear function approximation with d = |SSx Al, and ((s, a) = Ca.a is the unit vector with the (s, a) coordinate being and all other coordinates being O. In this case, On is simply the vector representation of the initial state-action distribution MD- where (8o, an) (er))))
Ho + &D ~ pa, dg ~ π; B" is an IS x A| x |S x Al matrix with P(s'ls, (as,))a(aa(s))) (a')s"), i.e., the transition kernel of the Markov chain over S x A induced by policy ग Put together, we have the textbook identity
P a alIss, a)
(1 - +) B 6 = μe,
where we recall the definition of the discounted occupancy n² from Eq. (2) Plugging it into C
T -
Ee,a Ee, (s, (s/ (s, = E (u /n2)²),
which is the divereegee between fr and S.P up constant shift and has appeared tight coverage parameter (especially when compared to || /p²) 1o Xie & Jiang 2020b when coverage measured based on density ratios.
5.1 GENERAL INTERPRETATION
We now offer the interpretation for the general setting. Note that B" can be viewed as the multi-variate rimeas-regression solution of the regression problem Φ(s, a) + obs', thus
Ey-P((-b,a))de [6(s', T)] s D" (ss
(15)
In general, the above relationship is only approximate (in [Section 5.3 we will see that it becomes exact under an additional assumption), although 8 is the best linear predictor. This leads to the following interpretation of Cg (see Appendix C.T for the proof):
Proposition 1. Define a deterministic linear dynamical system {u 112,0220 with 18,0 -o. and Vt 0,
1o = =B" P2,1-
When p(B") < 1/7 1/77deefne the feature occupancy in B as pT = (1- (1 7' Mo then
T = (~=))
The proposition rewrites C in a form that closely resembles the standard notion of linear coverage in the literature (Eq. (.). where we see the expected feature occupancy under the target policy (n% here) measured under the data-covariance norm Σ-1; see Section 5.3 Accordingly. we call C the feature-dynamics coverage. The difference is that here the feature occupancy defined in a deterministic dynamical system 1= instead of the true MDP. Furthermore, while the latter, o" := (s,a))=== [ø(s, a)], is always bounded, 142, on the other hand, may not be bounded in general and 1120 may actually diverge. The connection between LSTD and the linear dynamical system B" was first identified by Parr et al. (2008) (see also Duan & Wang (0220), though they focused on the algebraic equivalence between LSTD and the model-based solution B=, and did not perform finite-sample analyses or connect this to the notion of coverage.
Comparison to Perdomo et al. 2023) and subsuming known tractable conditions. Our bound shows that linear OPE under realizability is tractable as long as C is small, which sharpens and generalizes existing understanding of the tractability of linear OPE. In particular, we compare to Perdomo et al. (2023), whose analysis was shown to be sharp and subsume many prior conditions known in the literature, including on- on-policy sampling (Tsitsiklis & Van Roy 1997). Bellman com- pleteness (c.f. (Section 5.3), low distribution shift (Wang et (021). symmetric stability (Mou et al.) 2022a), and contractivity (Kolter, 201T) (see the discussion in Perdomo et al.) for formal definitions). leverage the interpretation of our coverage coefficient provide novel and weaker on-policy-type condition that enables tractability.
Furthermore,
Proposition 2. Assume & contains a bias term, that is, there exists dg E IRd such that a) = 1 Vs,a. a. If pl BB7) < 1/y and BED (0(s, a)) = F,, (o(s', =)] = 6o, then C < 1.
'For cleanliness notation, we are taking the convention that current state is on the column, and next state on the row (as standard in stochastic processes). In RI it is often the other way around. RRcaal that p denotes the spectral radius of matrix.
7

---

Published as a conference paper at ICLR 2026
Unlike previous on on-policy-type type conditions that relate the distributions of (s, a) and (s', a') (see Appendix C.2 for more discussion), Proposition Prrpossinn 2 only requires their means to be identical. The intuition that while 8 may not be accurate in next-feature predictions population prediction for " [obs', ( a')] can he correct when & contains a bias term. As consequence, when (s, a) and (s', a') share the same feature Φ0, we can establish that all future P.. are identical to [ a)}, leading to coverage in the compressed feature dynamics.
individual (s, a), its
feature F [o(s, a)] and this coincides with the starting
In terms of quantitative rates, PPerdomo et al. establish that, under some regularity assump- tions, || 0) E 1/omiu(l-77)" 1/9x,, 2:1/) Estat, for some Estat which is polynomial in d, 1/n, (1/6). _/)o.aand and spectral properties of Σ. While they only show
guar- antee on μ (c.f. Appendix D). this intermediate result immediately implies a return-estimation guarantee comparable to ours:
1 E-1 miin (I - y 1/2 1/22)
Tmm (T) J(T)
Estat-
As we have already shown that our statistical rate is tight, suffices to compare our Cg to their multiplicative factor in front of Estat- In partieular, establish the following relationship (see Appendix C.3 for the proof).
Proposition 3. We have that:
1 x-1
= -1- 112-
$ (1-7) 0 m
- 7 1/2 E 1/2)
Furthermore, the gap between the LHS and the RHS may be arbitrarily large.
This demonstrates that our coverage parameter, addition subsuming known tractability conditions for linear OPE, may lead to return-estimation guarantees that are sharper than those of Perdomo et al. (2023) (in an restance-dependee the spectrum of the matrix (I -/11/2 11/2).
sense). This is due to our sensitivity on do and its relationship to
5.2 RECOVERING AGGREGATED CONCENTRABILIA
State abstractions are special case linear function approximation, where each state is mapped to one of the K abstract states, (s) E [ := {1...., effectively treating states with the same (s(s) as aggregated and equivalent reduce the size of the state space. Under the abstraction scheme, the natural model-base solution coincides with LSTDQ with o(s,a) a) @s(a) aa When we only assume realizability (Assumption 1), the abstraction Q" -irreeellannt (Li et al. 2006), and has been analyzed by Xie & Jiang (2020a); Zhang & Jiang (2021); Jia et al. (2024) with the following notion of aggregated concentrability as its coverage parameter (sec Apppeddix C.4 details):
Definition 1 (Aggregated concentrability). Given vp > [K], define the abstract MDP My- G (S,, A, Pe. Røs 7, v/)) where Sy [K], the initial distribution is (l) Emv(s)) Mu(s), and
1 (s) Exp(sr (Eorwian=N P(
ak)
Σ P (s,
Consider ग whose action distribution depends on s only through (sss,, with a slight abuse of notation write <(-ls) T--lk),, vk. (s)). Aggregated concentrability refers the size of / / either measured in || lbo or x2, where Vhs, is discounted occupancy in MDP Mg. and 6D a) EsSo)) P a) (o,,)(s
In this definition, Pe. is the dynamics over the abstract state space, and easy to see that the transition kernel of abstract-state pairs under n, P3 (k', a' [kk a) Pv (k' a)= (a')k') coincides with the definition of D= (see Proposition 1 specialized to this setting, and Σ = diag(oP). As result, C recovers the x² version of aggregated concentrability (see Apeenddp C.5 for the proof):
"The definition of Rg irrelevant here and thus omitted.
8

---

Published as a conference paper at ICLR 2026
t-0
t=1
1=2
1=3
State distrriutiin
Ho
P12
P3
space 6/5
p
p
p
p
E
B
B
B
Feature expectation space Rd H6.0
8,1 6,2
Figure 1: Illustration of the evolution of occupancies under the true dynamics Pd (top row) and that of features under the compressed dynamics 8" (bottom row). Under Bellman completeness, the dashed blue arrows hold and two routes (+ ... vs. ++ ... -+) yield the same expected feature vectors, but they are generally different without such an assumption.
Proposition 4. When ((s, a) = aa is induced by state abstraction &o and depends only through (8), recall that ΦD , (0) is a distribution over (k, a) pairs, and have
C ((k (1 / /p)
5.3 RECOVERING STANDARD LINEAR COVERAGE UNDER
Prior results on abstractions leave an intriguing question open: they measure coverage by analyzing error propagation in Ms. which a -wwer-dimensional and approximate model compressed from M by 1) as evidenced by M3z., in the definition of aggregated concentrability; this is also consistent with our results in Section 5.1 where occupancy is measured in the compressed linear dynamical system B On the other hand, the mainstream notion of coverage in linear OPE, obtained under the Bellman-come commaa-ncopleteness assumption, is C = (07)T>> ot (q. (5)). which concerned with error propagation in the true dynamics M since o= is defined w.r.t. the occupancy μ¹" in M. While anecdotally it has been the general perception from the community that error propagation in compressed models (as in Q iirelevvant abstractions)
exception to the typical error propagation
analyses, our results below show that results that are seemingly disconnected with each other can be elegantly unified through the following proposition:
Proposition 5. Let F == To A E Rd) be the space of functions linear in 6. Assume F& satisfies dellman-complette (Assumption 3). Then, (1) T becomes an exact model for next-feature prediction, i.e., Pg ((8 ạt)
B ((8, a), (2) ug σ², (3) pl (B* < 1, and (4)
C = = (6 ( 1 10.
The essence of the proposition is illustrated in Figure 1, showing that the expected features produced by the groundtruth dynamics (d*) and the compressed dynamics (2 (1 - ?) E. (2,,)) coincide under Bellman-come compleenecm thus demonstrating that error propagation through true dynamics a special case of and thus unified with error propagation in the compressed dynamics.
Comparison to Yin & Wang 2020); Duan Wang (2020) Tabular and Linear MDPs. the special case of tabular MDPs, LSTDQ coincides with model-based OPE, and Bellman-complltenes, holds automatically. In this setting, our Eq. (12) matches the dimension-free guarantees (in 1/ yn term) given by Yin & Wang (2020) Duan & Wang (2020) also provide similar dinension-fie guarantees for FQE in linear MDPs. In comparison, our analyses hold in arbitrary MDPs beyond these structural models.
Connection to Bellman Residual Minimization (BRM). Many (if not most) algorithms for learning Q with general function approximation coincide with LSTDQ under linear function approximation (Antos et al.) 2008 Xie et al. [221]] Uehara et al. 2020), and this fact allows to compare our bound to the more general maalyses in the literature. Among those algorithms, BRM
'They analyze the mnite-horizon setting but the 1/vn terms in the bounds match under standard translation. Also, they use the variance of value function w.r.t. transition randomness in place of our Vaax, which we could also incorporate and choose to for readability.

---

Published as a conference paper at ICLR 2026
is well-investigateer example, which approximates Q" by solving the following minimax problem (Antos et al.) 2008):
arg min sup fEF J'EF
-
ED
(16)
whose sinite-sample guarantee can be established under Bellman completeness Assumption 3 Antos
jtt al. (2008); Xie et al. (202T) show that when JF is linear, the solution coincides with LSTDQ. so we
can compare the guarantee of BRM under linear JF with our Theorem 2
Jiang & Xie (0044) show
that BRM's error bound is (see their Eq. (19))
Jz. (T) - Vmax
C 1og([
1-7 V
(17)
n
In the linear setting, their C" is Gin (see their Eq. (23)), and log [F] x d based on a standard
covering-number argument. Under such translation, the main O(n 2) term in our our (10) matches Eq. (17) in the coverage and horizon dependence, and is sharper than the latter as our bound is "directional" and does not depend on log [F] d. In contrast, the concentration event in BRM's analysis is that Eq. (16) concentrates to E [(f - - f)²]. which is independent of the initial state. This makes the nature of Eg. (17) function-estimation guarantee (see [Appendix A), similar to our Theorem (which does depend on d). Whether [Eg. (17) can be improved to directional bound (i.e..
no log FD) for general non-linear JF is an interesting open problem.
5.4
UNIFICATION WITH MARGINALIZED IMPORTANCE SAMPLING
we mentioned that many algorithms designed for general function approximation reduce LSTDQ when linear classes are used. Another example is Minimax Weight Learning (MWL: Uehara et al., 2020), representative method for marginalized importance sampling, whose key idea is illustrated by the following inequality: given such that Q F, Vw : A IR,
In Section 5.3
1
1-7 E T (wr a)r) J(=) S sup Jy( +- R,,- B. a) (+f(d,,) - [(), (18) JEF
so learning w from some W class that minimizes (the empirical estimate of) the RHS to 0 ensures that 1 E, (w(ssa)) is good estimation of J(r). Theoretically, if some w* E W sets the RHS of Eq. (18) to 0, finite-sample guarantees can be established, where coverage is reflected by the magnitude of wt As an example, w (s, a) = 1=" (s, a)/pp/(/ a) always sets the RHS to 0, and we pay the size of w² as the coverage parameter (e.g., ||," P/ lloo) through concentration inequalities; see[iee & Jiang 2020b, Section 6.2) for further discussions on this.
When both W and F are linear, Uehara et al. (2020) show that the MWL algorithm is equivalent to LSTDQ. We now show that their coverage parameters and guarantees, when improved with insights from follow-up works, coincide with our analyses in the linear setting. In particular, (Zhang & Jiang (2024) point out that the w* that minimizes the population objective [Eq. (a8)s a different form the linear case: w* (s, a) (1 11--) 7) A 1 a). An immediate implication is that
(x, = Că-
That is, the second moment of w= on data is precisely our coverage parameter. While Uehara et al. (2020) measures the size of w" by ||w" due to the use of Hoeffding's ' inequality, replacing it with Bernstein's 's will improve ||w= E,, [ww a) in the main O(n 1/2) term, which matches our bound in Eq (10) except that we do not have d dependence; see Appendix C.7 for further details.
6 CONCLUSION AND DISCUSSION
We tackled the fundamental problem of linear off-policy evaluation under the minimal assumption of realizability. We re-analyzed canonical algorithm for this setting, LSTDQ, and developed error bounds that introduced the feature-dynamics coverage, a new notion of coverage that tightens and sharpens our understanding of this setting. This parameter admits a natural interpretation as coverage in a feature-induced dynamical system, while simultaneously generalizing special cases such as aggregated concentrability with state abstraction features and linear coverage with Bellman-commp features. Altogether, our results serve as clearer and more unified foundation for the theory of linear OPE.
10

---

Published as a conference paper at ICLR 2026
DISCLOSURE OF LLM USAGE
In the initial phase of the project, the authors had a vague conjecture and rough road-map of the main results the paper, and used an LLM execute the plan further to verify the feasibility of the project. We also subsequently used LLMs to help with literature review and proofs with some elementary linear- algebaal lemmas.
REFERENCES
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, 2011.
Philip Amortila, Nan Jiang, and Tengyang Xie. A variant the sang-foster-kalee lower bound for the discounted setting. In arXiv preprint arXiv:201..01075, 2020.
Philip Amortila, Nan Jiang, and Csaba Szepesvári. The optimal approximation factors misspecified off-policy value function estimation. In International Conference Machine Learning, 2023.
Philip Amortila, Dylan J Foster, Nan Jiang, Akshay Krishnamurthy, and Zak Mhammedi. Reinforce- ment learning under latent dynamics: Toward statistical and algorithmic modularity. In Advances in Neural Information Processing Systems, 2024a.
Philip Amortila, Dylan J Foster, Nan Jiang, Ayush Sekhari, and Tengyang Xie. Harnessing density ratios for online reinforcement learning. In International Conference on Learning Representations,
2024b.
Philip Amortila, Dylan Foster, and Akshay Krishnamurthy. Scalable online exploration via coverability. In International Conference on Machine Learning, 2024c.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and a single sample path. Machine Learning. Springer, 2008
Dimitri Bertsekas. Dynamic programming and optimal control, volume II. Athena scientific, 2007.
Dimitri P Bertsekas and Huizhen Yu. Projected equation methods for approximate solution of large linear systems. Journal of Computational and Applied Mathematics, 227(1):27-50, 2009.
Justin A Boyan. Least-squares temporal difference learning. In ICML, pp. 49-56, 1999.
Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33-57, 1996.
Yutian Chen, Liyuan Xu, Caglar Gulechre, Tom Le Paine, Arthur Gretton, Nando De Freitas, and Arnaud Doucet. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23(302):1 2022.
Qiwen Cui and Simon S Du. Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus. Advances in Neural Information Processing Systems, 35:11739-1175, 2022.
Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences: survey and comparison. The Journal of Machine Learning Research, ((1):809-883, 2014.
Riccardo Della Vecchia and Debabrota Basu. Stochastic online instrumental variable regression: Regrets for endogeneity and bandit feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 16190-16198, 2025.
Yaqi Duan and Mengdi Wang. Minimax-optimal Off-Policy Eyaluation with Linear Function Approximation. International Conference on Machine Learning, 2020.
Yaqi Duan, Mengdi Wang, and Martin Wainwright. Optimal policy evaluation using kernel-based temporal difference methods. In arXiv preprint arXiv:2109. /2002, 2021.

---

Published as a conference paper at ICLR 2026
Amir-massout Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, 2010.
Dylan Foster, Zakaria Mhammedi, and Dhruv Rohatgi. a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025.
Germano Gabbianelli, Gergely Neu, Matteo Papini, and Nneka M Okolo. Offline primal-dual reinforcement learning for linear mdps. In International Conference on Artificial Intelligence and Statistics, pp. 3 3169-3177, PMLR, 2024.
Daniel Hsu, Sham M Kakade, and Tong Zhang. An analysis of random design linear regression. arXiv preprint arXiv: 1et::200662266 2363, 2011.
Audrey Huang and Nan Jiang. Beyond the returm: Off-policy function estimation under user-specified error- mevsurre distributions. Advances in Neural Information Processing Systems, 35:6292-653 6303, 2022.
Zeyu Jia, Alexander Rakhlin, Ayush Sekhari, and Chen Yu Wei. Offline reinforcement learning: Role of state aggregation and trajectory data. In The Thirty Seventh Annual Conference on Learning Theory, pp. 2644-2719 PMLR, 2024.
Nan Jiang. Notes on state abstractions, https://nanjiang,
note4. 2018. Version: September 28, 2018.
Nan Jiang and Tengyang Xie. Offline reinforcement learning in large state spaces: Algorithms and guarantees. Statistical Science, 2024.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference Learning Theory, 2020.
Ying Jin, Zhuoran Yang, and Zhaoran Wang- Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, 2021.
J Kolter. The fixed points of off-policy td. Advances in Neural Information Processing Systems, 2011.
Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. In Journal of Machine Learning Research, 2003.
Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of lstd. In ICML-27th International Conference on Machine Learning, pp. 615-622, 2010.
Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of least-
squares policy iteration. The Journal of Machine Learning Research, ((1):3041-30744 2012.
Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics, 2006.
Pai Liu, Lingfeng Zhao, Shivangi Agarwal, Jinghan Liu, Audrey Huang, Philip Amortila, and Nan
Jiang. Model selection for off-policy evaluation: New algorithms and experimental protocol. arXiv preprint arXiv:2502.0800// 2025.
Diego Martinez-Taboadaa and Aaditya Ramdas. Empirical bernstein in smooth banach spaces. arXiv preprint arXiv:2409.0606/ 2024.
Stanislav Minsker. On some extensions of bernstein's incquality for self-adjoint operators. Statistics & Probability Letters, 127:111-119, 2017.
Wenlong Mou, Ashwin Pananjady, and Martin J Wainwright. Optimal oracle inequalities for solving projected fixed-point equations. In Mathematics of Operations Research. INFORMS, 2022a.
Wenlong Mou, Martin J. Wainwright, and Peter L. Bartlett. Off-policy estimation of linear functionals: non aom-osympto theory for cemi-parametric efficiency, September 2022b. URL httpp//arxxiv org/abs/2209, 12777
arXiv preprint arXiv:2209. 13075 [es, math, stat].
12

---

Published as a conference paper at ICLR 2026
Rémi Munos. Performance bounds in 1p -nor for approximate value iteration. In SIAM Journal
Control and Optimization. SIAM, 2007.
Rémi Munos and Csaba Szepesvári. Finite-time timte-time bounds for fitted value iteration. In Journal of Machine Learning Research, 2008.
A Nedić and Dimitri P Bertsekas, Least squares policy evaluation algorithms with linear function approximation. Discrete Event Dynamic Systems, 13(1):79-110, 2003.
Ronald Parr, Lihong Li, Gavin Taylor. Christopher Painter- waneeiakkef and Michael L Littman. An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the 25th international conference on Machine learning. pp. 752-759, 2008
Juan C. Perdomo, Akshay Krishnamurthy, Peter Bartlett, and Sham Kakade. A complete characteri- zation of linear estimators for offline policy evaluation. In Journal of Machine Learning Research, 2023.
losif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. In The Annals of Probability, volume 22. Institute of Mathematical Statistics, 1994.
Gilbert W Stewart and Ji-guang Sun. Matrix perturbation theory. (No Title), 1990.
Richard S Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. In Foundations of computational mathematics, volume 12. Springer, 2012.
John N. Tsitsiklis and Benjamin Van Roy. An analysis of cemporal-differenc learning with function approximation. IEEE Transactions on Automatic Control. IEEE, 1997.
Masatoshi Uchara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In International Conference Machine Learning, 2020.
J. I.. Hemmen and T. Ando. An inequality for trace ideals. Communications in Matheratical Physics, 76(2):143-144 1980.
Roman Vershynin. High-dimensional Probability: an Introduction With Applications in Data Science, volume 47, Cambridge University Press, 2018
Martin Wainwright. High-dimensional Statistics: Non-Asymptotic Viewpoint. Cambridge
University Press, 2019.
Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham Kakade. Instabilities of offline rl with pre-trained neural representation. International Conference on Machine Learning, pp. 10948-10960. PMLR, 2021.
Eric Xia, Martin J Wainwright, and Whitney Newey. Instrumental variables: A non-asymptotic viewpoint. arXiv preprint arXiv:2410.02015. 2024.
Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability, 2020a.
Tengyang Xie and Nan Jiang. Qa approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty Artificial Intelligence, 2020b.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-come cellmannconsistent pessimism for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Tengyang Xie, Dylan Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. In International Conference Learning Representations, 2023.
Ming Yin and Yu- Xu-Xiang Wang. Asymptotically efficient off- policy evaluation for tabular reinforce- ment learning. In International Conference on Artificial Intelligence and Statistics, pp. 3948-399 3958. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2026
Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065 4078, 2021.
Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05800 2022.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. In Advances Neural Information Processing Systems, 2021.
Siyuan Zhang and Nan Jiang. Towards hyperparameter- free policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 12875, 2021.
Yuheng Zhang and Nan Jiang. On the curses of future and history in future-dependent value functions for off- off-policy evaluation. Advances in Neural Information Processing Systems, 37:124756-124799, 2024.
Yuheng Zhang, Yu Bai, and Nan Jiang. Offline learning markov games with general function approximation. In International Conference on Machine Learning, pp. 40804-40829, PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Dimension-Fres Guarantee for Contextual Bandits
15
Proofs Section 4
16
B Proof of Theorem
16
B. 2 Proof of Theorem
18
Proofs Section 5
21
Proof of Proposition
21
New On-policy Condition
22
Proof of Proposition
22
On Aggregated Concentrability
23
Proof of Proposition 4.
25
C.6 Proof of Proposition
26
Details on MWL
28
D Function Estimation Guarantees
29
mmin A) Independent Population Bound via Loss Minimization Algorithm
30
Technical Tools
34
A DIMENSION-FRED GUARANTEE FOR CONTEXTUAL BANDITS
In Section 4
we claim that the OPE bound for contextual bandits is dimension-free Eq. (9) i.e.,
independent of the feature dimension d. This may come as a surprise, as similar bounds are frequently used in RL for caleulating uncertainty bonuses (Abbasi- Yadkori et al. 2011 Jin al. 2020 2021 but they take the form of
d + log(1/5) 10olls:11
V mmax.
(19)
72
In comparison, our our do)sses not have the d factor, which we explain here. To start, we first rewrite the error the average of terms across the data points. Here temporarily use subscript to denote the i-th data point (s,, ag r:) (in most of the main text we use subscript to denote the time step of state action, such as in 8t):
T (Outd 0¹) = 1-16--6- A0*) E-
1
oss ): = Σ W.E;
Here 6 := :; Ep- [r] zero-mean reward noise, and w; == of プー ((, az). Since wi only depends 8;, Og and is independent of r,, by treating 8,, a, fixed and only considering the randomness of r; given 8;, 0z we have (see proof techniques from Lemma 1) w.p. 2 1-6, 6,
-i= wf/n log(1/5) Vmax-
W
S
FI
Eq. (9) follows by noticing that Σn wf/n= 11001|
A few remarks are in order:
Such analyses often assume ridge regularization, i.e., adding A7 S.
15

---

Published as a conference paper at ICLR 2026
assume μe (and thus Φo) is known for convenience, and Eq. (9) holds without such an assumption. In fact, the most natural setup for contextual bandits is μo μ and the states observed in the data can be used to approximate Φp- In this case,
Unknown Ma- In Section 2
replace do in the above analysis with Φv == 1 n Ei= ((s,, T). Then, the above bound still holds since du does not depend on the random reward, but Φ 0r * Jxx.. This additional error can be handled as % 0 (mm E ( T 0 (=)), which also enjoys dimension-free concentration.
Return vs. function estimation. A key element in the dimension- free guarantee is that we are stating it for fixed initial distribution (An- In contrast, guarantees like [Eq. (19) hold for all possible Nl simultaneously. since the concentration event in its proof does not depend on juo- In this sense, the two results differ in the form of guarantees they offer: [q. (19) "w.p.
6, Vuuo". while [Bq. (9) is "VHo, w.p. > 1 - 8", and the factor-of-d gap can be explained away by union bounding over all possible du in Eq. (9) In fact, Eq. (19) Theorem is useful for establishing
guarantees; see Appendix D
Comparison to Theorem 1
Our main theorems for LSTDQ Theorems 1 Jadd] 2 do not subsume Eq. (9) as a special case as the above analysis for Eq. (9) cannot be directly extended to the > 0 regime. This is because the counterpart of w; will depend on A when > O, whose definition involves the random next-state s' in the data. On the other hand, the counterpart of e; is Q* Q:) - r; 7Q* (5/ af) (see proof of Lemma 1). and 6, will no longer be zero mean and independent when conditioned on Wie That said, our Theorem I iss still dimension- free in the main 1/ ///n term, but it comes with a lower-order term that depends on d.
also crucially relies on the fact that data is collected in a non-adaptive manner, i.e., later actions are not chosen based on the random reward observed in earlier data points. otherwise {a } will not be zero mean and independent when conditioned on {w,). In contrast, Eq (19) can handle adaptively collect data and is often used in online RL where adaptive data collection is inevitable.
Data adaptivity. Eq. (9)
Related works. While Eq. i9) generally known in the statistics community, to our knowledge it less known to the RL theory community, and we believe the idea worth spreading. There are, however, results that share similar spirits. For example. Duan & Wang (2020) shows that FQE in linear MDPs enjoy a form of guarantee similar to our [Eq (12) where the main 1/v2 term only depends on coverage and not the dimensionality. Yin & Wang (220)) Lemma 3.4) establish OPE guarantee for finite-horizon tabular RI,, where the 1/vn term has no polynomial dependence on the number of states and actions (which corresponds to our d), though their result measures coverage by a reachability parameter which translates to our Amin (E).
B PROOFS OF SECTION
B.1 PROOF OF THEOREM 2
Theorem 2 (Empirical Coverage Bound). Under Assumpiossssssonss 1 (and with probability at least 1 - 8.
Vmax
Cz. (d log(1 (1))
Jāna (T) - -J(T) S
(12)
1-7
n
where
0z :- (1- 7-1EA-1 Φ0.
(13)
We treat Oz = À not invertible.
Note that when A is not invertible the guarantee trivially holds due to Og +00, so below we treat A as invertible. This also implies the invertibility of Σ. Now,
Proof of Theorem 2
Jāma (T) [(r)
[Tr)
E Esg norw Q (s0, on) - Qstaa ao)
E Boo ao) (8 Bssus)
16

---

Published as a conference paper at ICLR 2026
where in the second line have used realizabilitty can continue with simple algebra to find that:
Assumption
and the definition of Qiatd. We
R Booopppoont [6(80, ao) (0r Dista)
]]
100 1
0 -Ohsad)
00 A-1 (A0+ -A0lsaa)
0o 1012--11/55- AAQ* - ABbua)
- -- 600 do0|/|| [//1|5-1///// (000 -00
where in the last line we have used Cauchy- Schwartz, and A0 = 8 (by invertibility). We then note that
///2A- Φ Vo
A
V 1
1
which yields that
Jann (T) ((T)
T
1-1
Σ
1/2
A0
-6)1
Then, Eq. (12) will be obtained by establishing the following concentration lemma.
Lemma 1. Fix any unit vector (i.e., ||u||2 1), with probability at least 5,
log(1/0)
1u 1/2 1/2 b) Vmax
where the guarantee is treated as vacuous if E. is not invertible. As corollary, w.p. 2 5,
d log(1/6)
|| 1 8)ll2 S Vmax V
n
Proof Lemma 1.
We first rewrite Ao- b as the average of terms across the data points. Here we temporarily use subscript i denote the i-th data point (s,, ag (s,
s() (in most of the main text use subscript to denote the time step of a state or action, such as in st):
Â0* - b
a(si, ai) (p((s, T 70(5), aT) T ri)
(ss a,) (Q - - (s)
Thus, et's are independent zero-mean random variables when conditioned the (si, a;) pairs. We will treat the design over $;, a; as fixed and non-random and only consider the case when Y is invertible, since we provide vacuous guarantee otherwise. Note that the invertibility of E (which is determined by the (si, a;) pairs) will not affect the validity of our concentration inequality below due to the fixed-design analysis.
Let I : $-1/2 - b), and
n
T
TT:==uTT-- 1/2
1/2
n
where d; is shorthand for ø(s;, a.). The expression is the average of independently distributed zero-mean random variables. Since c; 's are the only randomness in this lemma and |c|| S 2V V each summand's boundedness is
112 dbl 2V
0:
17

---

Published as a conference paper at ICLR 2026
We now invoke Hoeffding's inequality for unequal ranges (Wainwright 2019), which depends the sum of range squared:
Σ
0
4V max
1-5-1/2
0:00
T 4V2 max -u T 1/-112(n1 1/1211(( 1/2 15-1//2
4V max n.
While individual a, can be large (1 (/5511/20 |l2 can be as large as vn), the sum of their squares is 4V2 m, which is identical to the situation when each individual summand is bounded by 2V maxa and Hoeffding's s inequality does not distinguish between them. So w.p. 2 1 6,
u 1| S V max
log 8
This proves the first statement. For the second statement, note that ||4]2 = sup [ul/==11111 u T,,, so suffices to union bound over {u ||ul|2 = 1} by a covering argument. Let C {u : | = 1) be cover be specified later. By union bound, w.p. 2 1 - 5,
U 8
max W'El
Ju 7 S V
maxx
V n
log
Now for any unit let u' be the closest vector in 21 and
u L S I( + u')
lu (u (u- u') |u ||1||1|/ u/ 1|2
Taking supremum over u on both hands, we obtain
|| ||
sup
1 s max (u 1 + |1/1|/|u - u a'Ell
80
|14]/2 S max,' Ell lu
u u
We now ensure that the multiplicative factor 1/(1 || u' (112) < 2 by building a cover such that ||u 2/112 S 1/2, and this only requires constant resolution, so ( 0(1).. Combining the results so far and the lemma statement follows immediately.
B.2 PROOF OF THEOREM
Theorem (Population Coverage Bound). There exists no such that when n. 2 no. w.p. 2 8,
x Vinax
C log(115)
Jāmn (T)) [(TT]
+ 1
(10)
where
C% = (1 (1=11))) A A- 1o
(11)
and ny and the o(1/vn) term may depend on d and 1/m
Proof of Theorem 1 the cost burn-in and lower-order terms. Note the tollowing lemma, which we prove afterwards.
We obtain this from
Theorem by showing that Öz can be replaced by C
There exists and E, are invertible, and
Lemma 2.
Ba (m)) (A)
2
log(d/8) such that when 2 no. w.p. 2 1 - 8, À
C E5 5 (1 (1-
51/2 d01. O(L/n!/1)
where the O hides factors of
Janin (A)
18

---

Published as a conference paper at ICLR 2026
We also recall Lemma We union bound over the events across these two lemmas, each assigning 6/3 failure probability: note that Lemma contains two statements, and we require the first holds
with u being the normalized vector of 2 A
T-Tol Φo. This way, Lemma guarantees that:
100 1-1--11/55-1110000 - -B)] S 11000 A -- 1/2 1/2 Vmax log(1/o)
(20)
+ log
|5-- - B)1l2 S Vmn
(21)
n
Under the above bigh-probability events, we have and Σ invertible, and
JQuen (T) (T)) J 10 (0 - 0)1 = 100 A 1/ b)]
< (00 -0 5-1/2 - b) + 1 1-1/ - B)
(m)
Term (II) is exactly the LHS of Eq. (20) Noting that | A A
03/(1 - 7), this
immediately gives us the 1/v term
Eq. (10) remains to show that term (I) 0l 1/n).
(1 > 10 A 12/2 of A - 1 1/2 (Āo -Al-
Eq (21) shows that 1/-1/2 (Ag - b)1l2 O(L/V)). and Lemma 2 shows that | - 151/2
0 A y1 2 = ((4/n1/4) Put together, term (I) is O(1/n5/4) which proves the claim.
Proof of Lemma 2. end of this lemma.
We first give matrix concentration lemma, whose proof is postponed until the
B ())))
Lemma 3. When n. 2
log(d/o), with probability at least 1 - 6, we have:
mmin
Amaa (B) + Aunax (E)) log(d/6)
||- S <(E)
(22)
n
and
log log(d/0 (d/)
||A-Al] S <(A) (B² + max max((A)
n
and
A - A A- < m (A)2 IA - -All2 1012=00 -A
133 -0((5)
Taxx (A)
log log(d/0)
(23)
Ominl (A)2
n
As a consequence, À and E are invertible with high probability.
Since À and E are invertible, we can write:
Cg 03 = >)² E 12 A T dollz 111a-1122 -1/1//2 - 20l12
$ (1 7)²
11/2A 22122AT-2 0o0 -1122
$ (1 7) ({(15//2
/0001 + -1111/2-51111 A- A-T00 - (b)
-x
$ (1- Ba (/=V/ | ||A--- - + 1512 - 1!/2 ||A- 112
The second line is the intermediate quantity in the lemma statement and the rest of the analysis always bounds IC Og through the above relaxation, will not separately mention the bound on
12/2 A tu 1//2 T 00 2
19

---

Published as a conference paper at ICLR 2026
Let (eE// || -/1/21/ and s(A-1) ||A- - -A -12112. Note that the above inequalities imply
$ (1 - 7)² Ba (Amaxx (5:1/2) + E( < ((A-))+ )(()//))((AA))))))))))))))))))))
c -5
(24)
mmin
(A)
We conclude by bounding << (2) and <AA The bound on e(A-1) follows from (23) To obtain
a bound on E E(21/22) we can use (22) in combination with the the incquality ||>|// Σ1/2 l S
/
V VIE Ell2 (van Hemmen & Ando 1980
obtain:
(AMAAx (Iaxx())B3) (X (E)
Amax (m)) log(d/8) 1/4
|/ - - S V | Ell2 s v (d) -x
Returning to Eq. (24) and combining everything, we have:
(1 --7))²B BB B3 + Xmaxx( (max(()) (1))
log(d/0) (( ( (2) + Amax (E)) (d/()/)) 1/4
C 0% s
amin (A)²
n
Ammax (2))
(1 - 7) (xmax(E)(( + Amax (E)) log(d/0) 1/4
m (A)
O(1/n1/9)
Proof of Lemma 3
We firstly establish the bound on || Σ 22 To do this, use Matrix Bernstein
(Lema Abbreviate X; == (88, az), and let Z; = X,
Σ be the centered matrices. For the
almost sure bound, we have
|| |X,X, ||2 + || |x.12 + Amax (
B²g + Amaax (E).
For the variance term, have:
| [XX,, 112
)3] 2031
S BB [X - "
< B Amax () + Amax (
This yields
2 2)max (e) + Amax (E)) (2d
2( + Amax (E)) log (2d/2)
JS - E1la
3n
Amax (B3 + Amax (E)) log(d/5)
n
We now establish the bound on || - Alla again via Matrix Bernstein Lemma 8 Define the notation
X; = o(s,, a,) and X = (0*,, a.) - ( a(). Then we let Z, (X;) matrices. For the almost sure norm bound, we have
denote the centered
1 < ||X,|le||X|ll + ||Al|2 < B3 7) Fmax (A) S 2B² + [max((A) (A).
For the variance terms, we have:
Le (xx(xx)) - (X)(x) - A) "11 = || X AXIXT - X A+ AA
< 4B3 E [X,, 11, |AAT|I 112
= 4B3 Aunax (Σ) Fmax (A)2,
See also this answer by user "jlewk" on Math StackFxchange: htaps:://mah.... httpss//nath..sacceekhaaagg stacceechaageeccmm/ 3968174
20

---

Published as a conference paper at ICLR 2026
well
E (x,((x) A) -)0 (XxxXX- -4)]][ -||[((((()) (T xx(X)) - A xxxx x;X, A+ A+ ATAlll
S 4 + |A llz
= 4 x + o max (A)².
With the latter, the variance term and the norm bound are of the same order, which gives
A All (B) mmax(()) log(d/0)
"
Then, to bound | - A-1
2, we note the following lemma.
Lemma
(ttwaart & Sun 1990). Ler A E R with m 2 and let À A - E. Then,
1 5
c(A-1) S
max ||A- A |la
Furthermore, El| min (A)/2, then
||
A 112 S 11A 11A E111
This immediately implies that, for ||A Álle S minn (A)/2, 2, we have
1
B3g + max (A) log(d/8)
-
| A- All2 o
min (A)2
Tmin (emin(A) (A)2
This latter condition is equivalent
BBg max (A)) log(d/0)
Omin (A)
B + maxx(A)
log(d/8),
2
mmin (A)
which we have set as the burn-in time the lemma statement. Lastly, Eq. (23) and the reverse triangle inequality shows that (and thus Σ) are invertible with high probability.
C PROOFS OF SECTION 5
C.I PROOF OF PROPOSITION
Proposition 1. Define a deterministic linear dynamical system {u t2z,0120 with 1%,0 -o. and
Vt 0,
1 = 1B 1o
When p(B") < 1/7 define the feature occupancy in B as pb = (1-1)) Ei20 po.e. then
C = =1=)) T--1 1 1...
Proof of Proposition Recall that we defined Bd = - 1 15(r) T We note that
Σ yE,rr = Σ([ - (B*))((>)T))
Substituting this into C we arrive at the expression:
C (1 7) oo E'AEA- do (1 - 2) 0 (I BBa) - -1"
(25)
Note that when p(B*) 1/7, the matrix (I - Bd -1 has the series expansion:
(B- Σ (BTY).
Thus, we notice that
8
2 (BB)) Σ
(I - (1-BB=) do
Substituting this into Eq. (25) gives the result.
1eRecall that p denotes the spectral radius of matrix.
21

---

Published as a conference paper at ICLR 2026
C.2 NEW ON-POLICY CONDITION
Prior to our work, the only notable discussion of coverage in LSTD(Q) is its benign guarantees in strict on-policy setting, where data distribution μ² is an invariant distribution of p". One can further relax this condition to Err I 3Σ as 3 = 1 when μ is invariant under и.. This condition is handled by Perdomo et (223) and inherited by our results.
Given the dynamical system interpretation in Section 5.1 we are able to establish a novel on-policy result as follows:
Proposition 2. Assume & contains a bias term, that is, there exists Ou E Rdd such that d(s, To = 1 VVs a. If (*) < 1/yy and Epp (ő(s, = E [ō(s', , x)] = Qu, then Cg S 1.
The condition Eer 3 SΣ relates the distributions of (s, a) and (s', .'), where Proposition 2 Oolly requires their means to be identical. We provide proof below.
Proof of Proposition 2) Let (ø(s, a) + Y) any regression problem where (s. nD and Y is an arbitrary real-valued random variable jointed distributed with (s, a). Let oy = E- Elo(s a (where B[]] here is w.r.t. the above joint distribution) the population solution to linear regression. Then, when & contains bias term, it is known that
Llo(s, a) T 0y] = [[[]]
if B|Y|s, a] is not linear in @. This can be proved easily by multiplying & on both sides of x By E[o(s,a)) a)
0 Elo(s, as) 100 0 E(o(s, a){]
a) E[y]
Since B linearly predicts each coordinate of @(s', a') separately, apply the above result and obtain
Ep d')l = BB ((, a)] = B" E a)}.
Let ΦD = E a)]. Then the condition of the proposition tells us that
ΦD B Φn-
Since 13.00 ΦD. we immediately have 2T, = ΦD, Vt. Given p(BT) < 1/7, Proposition holds and nT = 2D. so
=0,) T E0' $
C.3 PROOF OF PROPOSITION
Proposition 3. We have that:
100 x-1
(1- - --
S (1- mmin (T
> 1/25 Σ 1/2)
Furthermore, the gap between the LHS and the RHS may be arbitrarily large.
Proof of Proposition 3
The derivation of the equality is follows:
(1 7) C% = 00 A -EAAAAT do
= 2i - Ece) ('a))''S( " - Eer) Φ
o0 (x/1/21
1/2 1-1/22 -/2 -1
Σ 52112 1177-----12- 1/255 540 5.-1/2 5-111/555551111/1111 -T
do
= = I.- 1/2(1 [I- 3 1/22 112)
E 1/2 1/25,,1' E -1/2)- 1/23-11 x/ 112 1/200
= =I((T ----1/// 1/25__1511///3
1 1-112 - 10
22

---

Published as a conference paper at ICLR 2026
For the inequality, simply take the operator norm the matrix (I - - -----/// Eor 1-13 (12)-T
-T:
(1---E-1/2)) x- 0all2 2 S
( --T---11/2200
[
Exc 5- 1/-1/2)- -
5 5- 1/2
40
1 15--
0 min
72 /oo
The fact that this represents arbitrarily large improvement follows by picking an instance where I - -/--//////- Eer E- 1/2 is wear-singular but Σ 1/11/26 do is in non-singular direction. For example, consider the instance where d = 2, the initial distribution Ho is deterministic on a state with feature e2, the data distribution n² uniform over the feature vectors v2e, and v2ez, and all the states in the data distribution transition to state with feature vector o' (1 =)e1. Then, it is simple to
- the LHS equal to 1 but the RHS is equal to 1/E, which can be made arbitrarily large by taking + 0.
)) and - >Σ - 1/2 1 - 1/2 0 () Thus,
verify that do = e2, Σ = Ce =
C.4 ON AGGREGATED CONCENTRABILITY
Section 5.2 mention that prior works establish aggregated concentrability the coverage parameter for learning with Q² -ireeeeeeeaan abstractions. However, these works focus on the model selection setting where learning under abstraction is used as subroutine, and the notion of aggregated concentrability is either implicit in the analysis or not given in a clean form (Xic & Jiang, 2020b) Zhang & Jiang 2021, Jia et al., 2024). Therefore, here we provide a clean sketch for learning under abstractions based on the spirit of these earlier works.
The key observation is that learning can be viewed as entirely happening in M (Definition 1
1. Data: The data-gent data-generating process is identical that from Ma in the sense that the following two generation process for (k, a, rr,kk k') are identically distributed:
(s, n2 a() P(-1s, a), (0)), k'),,'==1 1568)) (ki p , a)) K' P (-
where we recall that ((s, a) = C in this setting and ø E (s (ō(s, a)] is distribution over {1....,K} K} x A.
Learning algorithm: LSTDQ is equivalent to tabular model- based OPE the empirical estimate of Mg.. That is, Ousta is precisely Q Mr. where Md is the tabular MLE estimation of M2. obtained from data {k, a, k.),
Learning target: Since Q realizable by the abstraction, we can write its "compressed version" " (Q") a) Q" (s. a), Vs (K). [ is precisely the Q-function of (recall that in Section 5.2 we have 7-1ss) = (((((ss)) in My. namely [Q=1e = Q.,, (Jiang] 2018 Amortila
et al. 2024a).
Error measure: The return estimation error guarantees also follow a similar translation: let the in Definitonn be the initial distribution in Me.
((x) Jāma (T)) JMs (TT) J (TT).
Given the equivalence, we can simply invoke any guarantee for JM, J (T) and it will equally apply to (TT) - Jamca (T). However, the former model-based OPE, where a standard coverage parameter is x" of the density ratio w.r.t. My as the groundtruth model and d as the data distribution. This yields the aggregated concentrability in Definition
setbbook-standadd instance of tabular
Definition lommited the definition of reward function Rş. For stochastic rewards, the probability distribu- tion (r k; a) mixture of R(rls, a) similar to the transition case: Ry (r|k,, a) ((
(P(k,))
23

---

Published as a conference paper at ICLR 2026
General state-action aggregation. Definition iss restricted to ा that is consistent with the given abstraction v. Here we show a more general analysis for arbitrary म, which also handles general one-hot & that are not necessarily induced from state abstraction t,, corresponding to an arbitrary size-d partition over S x A. In this case, an abstract MDP analogous to My (with (1...., ) as its state space) is not well defined; nevertheless, the notion of aggregated concentrability to handle this setting, and its equivalence to our Oz still holds, We briefly sketch the analyses here, which includes (1) an analysis of learning under general one-hot Φ, (2) showing that the guarantee depends more general notion of aggregated concentrability, and (3) it is equivalent our C There are two approaches that both achieve the goal.
be extended
Approach 1. This approach directly extends the analysis above and is similar to Jia et al. (024). Instead of an MDP over fI,...,A K} (with K - d/f ) as its state space, we can directly define an abstract Markov reward process (MRP) (Ro. Po) with state space [d], and obtain the similar data-generation equivalence: (for this analysis we will abuse the notation and treat the output of e as its one-hot index, Φ : S x A + (1.....d .d)) let 6D = (s [Ø(s, a)], and we have
(s, a) ~ p
R(-ss,)), P(-s, k 2~ Ro(-))), Po(.))).
o(s, a), = (s')
Similar to before, can also show that the value function in the MRP is equivalent to Q", and LSTDQ is equivalent to the model-based solution, and so on.
Approach 2. An alternative approach to define the "abstract MDP" Ms over the original state space S (Jiang 2018 Xie & Jiang 2020a Zhang & Jiang 2021):
Rs(s,a) Σ (2)): (2(,a))
E(s0))022)==00.0))) (s.
Σ E( ((3 P(-ss,))
Po(-|s,a)
2(3d)) (
Unlike Approach 1 which requires to extend some MDP results to MRPs, this version still defines M3 as an MDP, albeit over the original state space S, and the initial state distribution is still μto. It has piecewise-come constant (w.rt. the partition induced by Φ) reward and transition functions, and within each partition it straightforwardly (and seemingly naïvely) mixes the true reward and transition according to the relative weighting in the data distribution. Similar to before, we have Q =QM,
To connect LSTDQ to this abstract model, we define the empirical projected Bellman operator 7u RRSAA_ S in the same fashion Definition of Xie & Jiang (020a)), and show that &
(1) Qiata is the fixed point of this operator, and (2) fixing any f. T concentrates towards TMg with sample complexity independent of (S]. By union bounding over all piecewise-constant (i.c.. linear-in-o) f. we establish that
E Eggaaa= P [(Qrtt Isdd The, Qusea) (s, a))] S Estat;
where Estat is the statistical error term (see Xie & Jiang 2020a Lemma 9)). Then, error propagation analyzed by invoking simulation lemma in Ma:
Jann (IT) J[(T)= Esgeoo Quaa (s, a)] -M, (x)
1
E, Ble.))aai 1Quad Thr, Qustal-
The last step is error translation from p to d where a naïve translation would yield the size of dār,, /nD which is not the same as our Cg. However, note that Qued and T (.) are both piecewise-constant (i.e., linear in Φ), allowing us to projecting each distribution onto the feature space (d%,, - (e (6(s,a)) 1 - E( (0(s, a)) = @D) before calculating their density ratio, which recovers Czy
24

---

Published as a conference paper at ICLR 2026
C.5 PROOF OF PROPOSITION 4
Proposition 4. When ((8, a) - ( is induced by state abstraction vi and depends on only
through (ss), recall that @D e [o] is distribution over (k, a) pairs, and we have
T B(R,))ao /
Proof of Proposition 4 We compute the A matrix. Below, we define ( s P(s 8, 5.
8'), P(E 8,0) Ex:p(e)== P(s' 8, and
PT 8, a) = P(K)) s, s,a)=(d' k)),
which valid since π is consistent with the state abstraction. To start, the covariance matrix becomes
Σ = B ((8.,))((s.u)))
Σ
Σ
" (s)
ekkakka
Σ a (s(sa))
kia
kE[K],REA
(ES:p(a)=
Σ
Ok
(k. diag(e
kE[K],REA
where we recalled the definition of @ matrix with elements of ΦD on the diagonal. Let's examine the cross-covariance Eer.
(k, a)
E. E H HP (s. a), and diag(ø") is the diagonal
Eer = Eg Eg [0(s, a))(s'
Σ (s a)
Σ
sES,aEA
S'EEC'EA
Σ
Σ
Σ
PT s Σ
RE[K]]aEA
=ES:p(s)=k
Σ
Σ
P(,
Σ
T o' s,a)
CKaa
kE|K]aaEA
sES:p(s)=k
KEjKTT,'Ed
Σ
Σ
Σ
# (s(s,))PT(H,d s,a)
eW
kE|K],EAA
KE(K),d'A
(ES:
M (s (,0)
Σ
eka
Σ
et
La olk
(k),
kEATTaAA
WE(KT,EEA
Σ
Σ
(k, k,a)
Ok,a
kE(K),EA
k'(KK],AAA
(PT))
where we recall P as the transition kernel of policy ı in My:
PT a' k, = Exp(ss uP (8, a) (K a' 8,a)
(K(k, a)
Putting our expressions for E and Eer together, we immediately have
Pg.
Given that the initial state distribution in Md voa we can compute its initial state-action distribution Fesnuo.nnn ([b((s) = ko, a ao]]
vo (
Σ
(o(S))T(aolho)
=o
Exwspgaart amut Ck (18(s), a)] @okko ao),
25

---

Published as a conference paper at ICLR 2026
implying that the initial state-action distribution happens to be do. Finally, coverage coefficient becomes
Cy = (1 - -1-11100111---------111 -B*) 60
(1 - 7) (1 7p - diag(8 - 7Pp) 1
(M5Te) " - Măr. Bssan (1/r0//093
C.6 PROOF OF PROPOSITION 5
Proposition 5, Let F& == To E R be the space functions linear in c. Assume F& satisfies
Dellman-completenee AAssmmtiin 3d. Then, (1) B becomes exact model for next-feature
prediction, i.e., (gsa(()ss.))0 (8( n = B" 6(8, a), (2) p "², (**) S 1, and (4)
C = Cin = (Ø T
Proof of Proposition
Eg P ()) 13 ( a) for all (s, a). First, we show that under Bellman completeness F& is also closed under the transition operator pa := T R. that is, f E F& for all f Fa-
(1) D is exact next-feature predictor:
The linearity of F& together with Bellman completeness immediately imply that the reward function linear, or Fa. Define fo Fs be the function corresponding to the parameter Og, so that fo(8, a) 0 for all (s, a); we have T fo R E F
Next, fix any f Fø and observe that T is also linear, since T" Fø under Bellman completeness. follows that f R pd E Fo because the difference of two functions linear the same features is also linear those features, which proves that Fs is closed under p=, This closure implies that for any Fø. there exists some 0, E Rdd such that
( a) 0, (s,,a) Eg (s(s,,
To prove the stated claim we will utilize choice instantations of such functions and their corresponding parameters. For i [d], define the function f. = (Ø. c;) E Føs and let 0g Rd be such that
bls TO (i(l m),, V(s,a).
Then for all (s, a).
(f (225))
07
0
Ey (0==()))))((s
((s. a).
0
Lastly, we will show that the above system of equations is satisfied by setting
T =
Right-multiplying both sides by o(s, a) then taking the expectation over (s, a) ~ nP, we obtain
E "' Pxx [o(s', xxx a')))s,))) a) D"d (o03)))a0((s,))00ss,)))
26

---

Published as a conference paper at ICLR 2026
Solving for 8 and rearranging gives
"x-
B xx
-
which confirms that 8 (2 (2-1) satisfies for all (s, a) the equivalence
Ey By~P(1ss T)] 13" ((8,))
(2) Showing 1 o= Recall that ot (s (0(s, a)}. Using the Bellman flow equations for μ² we obtain recursive system of equations for the dynamics of o=:
Σ (s, (s,a)
Σ (() (1-7))0($,0)) +1 p (t) 8,8
(1 7)@ + E (Bs- x|]]
(1 7)00 + y [) [B ((s
(1 7) BB,
where we invoke the result from (1) in the second-to-last line. Repeatedly expanding the RHS of the equation with the recursion,
Tt (1 7)do + TB=d"
= (1 7)do + 7B ((1 7) + 7B 6*)
= (1 (do + +BB do + 7B=
7) (B")oo
which is exactly the definition of n from Proposition
(3) Showing p( < 1. The proof of (2) implies that for any (so, un) E A. (B=))((s ag) (x.a))=a77 "0 (0(8, a)). where 14 700000 is the 1-th step state-action distribution under x when the initial state-action pair is the given (5o, oo). Given ([o(s, a2 I Bo, v(s,a) we have
(o")) ao) S Bo, Vt.
Given that E is full-rank, we can always find {(s) (1) ( such that (u, = (1500) a0 )
0o
forms basis of Rdd Then we have (I(B))'z uull S B Vt.
Now we show that || (B")'llop- where dlop is the operator norm, also has finite bound that is independent of t. Recall that operator norm is the largest singular value; let the corresponding singular vector be and ||ul|2 1, and express
Ei= aqu, We have
[I 11(B*)'ull
(BTTy
$
>
Jay BB =.
The key here is that the upper bound v < oo is independent of t. Plugging into the Gelfand's formula, we have
p(B))
lim 1000
(B*) <
lim !/t 1.
1-000
27

---

Published as a conference paper at ICLR 2026
(4) Proving equivalence proof of Proposition when maxx 1**h 1 1/7 We may write
C Cii Recalling Eq. (14) and the definition of pa) following the
c = (1 7) EEA TT do
(1 7)² (T 7 - -- -I-
(1²) Σ ' ug.
Substituting the previously derived identity that 1 = Φ in the last line,
(13) Σーン n ---)=== Cll
(
C.7
DETAILS ON MWL
Here we expand on Section and provide more details about the unification with MWL.
MWL guarantee. MWI. is concerned with the following loss:
L(w. f) = [)(()) - E [ur(s, a)
((s,
(26)
Let L(w, f) be its empirical estimate from data. The algorithm learns
t arg min max [(w,)s wEW
JEF
and produces Ja( = (s, a) r] as an estimation of J (TT). Uchara et al. (000 show that when Q" E the estimation error can be bounded as
(Ja(T) - (xx) min max L(w, f) wEW JEJ
L(w, f) D(w,
WEW.JET
The first term is the realizability error W. The second term is uniform deviation bound for L(w, f):
for finite W, F, Hoeffding with union bound gives: w.p. 2 1-8. 8,
Bw 1
1 ln
W
[[w,, f) DI
max WEW.FEF
8
where Bw == max |wllo and By is defined similarly. Since F models Q", we often assume By S Vmax.
Coverage in MWL. In this analysis, the coverage parameter is manifested rather implicitly in Ow: we need to find w" E W to control the first term, and for simplicity let's focus on w such that m L(w", mentioned in Section 5.2 standard choice is w (8, = (s, a) (s, Then, when w W, we have
then the estimation error will only have the uniform deviation term. As
Cw 2 | |l = || 11.
which is the standard lor norm density ratio. W is properly designed realize w", we may assume Cw S ||w"||oo, and obtain bound that explicitly depends on || PI
Connection to LSTDQ and unification with our results. When W and F are linear classes in the same give feature (with appropriately bounded norms the parameters), Uehaaa al. 2020 Appendix A.3) show that MWL is equivalent LSTDQ when À is invertible, in the sense that
(m) J@nn. (TT)
In terms of analysis, the linear structure the classes provide another choice of u* that satisfies max T(w, f) 0, namely w² (s,a) (1 - --1 a), whose data second-moment is
28

---

Published as a conference paper at ICLR 2026
equal to C. To make this quantity appear in the guarantee, calculate the following variance: for fixed to E W.SE. F,,
Var (umpou(ss,))111666 a)(= - [(8,a))] s Bn ( X E ww(s, B) - 2w
where Bw,z wax B EnD a) E (w-(s a)}² Cz. Now by Bernstein's inequality and a standard covering argument over linear W and F, we have (c.f. Xie Jiang (2020b Theorem 8))
C%. Similar before may assume Bw &
(
103 (d log(1/6)) Bw(d + log(1/6))
Vinx
[LW, L(w, fll 1-7
max WEW
n
n
whose / IIn term matches our Theorem except that we do not depend on d.
D FUNCTION ESTIMATION GUARANTEES
For most of the paper have focused providing return estimation guarantees, i.e., error bounds for estimating J(x)) In some scenarios, it is desirable to obtain stronger function estimation guarantees (Huang & Jiang, 2022 Perdomo et al. (023). that Quad and Q" are close as functions, typically measured by weighted 2-norm. Indeed, our proof of Theorem 2 can be easily adapted to provide the
following guarantee:
Theorem 3 (Function Estimation). Under the same ussumptions Theorem
w.p. - 6, for any
E ASS x A).
T dlog
V ((t.a)u((QQ(sss)) (s, 2
Viax 1-
where Oz = (1 7) B (s0.
(o0. - ||51/22 d(s0, 0
When μ4 is point-mass, the LHS of Theorem coincides with that of Theorem and the guarantees on the RHS are identical, too. Also recall that the naïve analysis based on 1/ I/Emin (A) (Section 1) provides parameter identification (i.e., bounded ||0 - 0- []), which immediately provides 230 dunction-estimation guarantee. This result is directly implied by our Theorem where the coverage parameter be bounded as function of minn (A) and B
Remark on CH Similar to how we can replace O with Cg up to lower-order terms (see Theorem and its proof), Čß. which we denote C It interesting to compare it standard coverage parameters that enable function-estimation guarantees under completeness (Section 3). Note that the term inside C ) simply C but for deterministic initial state-action pair ap). Applying
also obtain variant of Theorem 3 that depends the population version of
Proposition 5 we have
C (sooaa))) [(0 ) 1E-1- d 25..ag
where 20 = E (s,a>~= [ø(s, a)] is the expected feature under the occupancy induced from deterministic 80, dn as the initial state-action pair. In comparison, the standard coverage in the literature is
Cha. ((s.))=u== T (s,, a)).
As can be seen, our C is in between C and C since we partially marginalize out the portion of n that can be attribute to each initial state-action pair, instead of measuring every single (s, a) ~ μt" under --1 in completely point-wise manner.
29

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3 We repeat similar derivation to the proof of Theorem noting that the proof holds when the mitial state-action distribution So ~ lAu, do ~ π changes to an arbitrary distribution v.
( 0n) Quata an) )
(00, (0(5)) ao) (0- (0~-Daaa) Ossua) 27
(M(80: (0) A 15 - -/-
-Oisaa) ))
= =Been,
E1 6(s0. an)
|/15: 1/12-1/ 1/00 - 6) 12)
2]
To conclude, recall the concentration bound from Lemma that w.p. 2 1 8, d log(1/8)
-5 1/-11/2A00
-0)1 s V2 max
n
Plugging this in yields the proof.
E miin ( (A)-ENNEPNN POPULATION BOUND VIA LOSS MINIMIZATION ALGORITHM
Here we provide an alternative analysis to Theorem where we are able to eliminate the dependence on m/m (A). but the rates still depend quantities necessary for the analysis of linear regression under random design (in our case, 1/Amin Σ), though we expect that this can be further tightened to
leverage-score scgge-ccore-tppe conditions Hsu et al. (201T): Perdomo al. (2023)) The analysis also requires
slight change of the LSTDQ algorithm to a misimiziinn form (Liu et al. 2025):
0 arg min 111/ -A)-))l))
(27)
d=
In practice, when Á is near-singular, the inverse solution Å may have very large norm which is clearly problematic, demanding some regularization to control the norm of the solution. The loss-minimization formulation of Ef. (27) is a natural abstraction of this process, where we search for 0 in a pre-defined parameter space with bounded norm. If Ā-1 E θ, is easy to see that the loss mosimiininizttion solution coincides with the inverse solution; when Â-1 & O, O.Eq.. (27) still outputs a bounded solution to ensure generalization and good statistical properties.
We will need the following boundedness assumption on θ. Assumption (Boundedness O). Assume ||0||2 S Be, EO
Additional linear algebraic notation. For symmetric E, let k(E) number, where Amax(-) the largest eigenvalue.
Amaz Amiu
be the condition
Theorem 4. Assume that n Z (og(d/))(())))) A/mi Ariia (E). Under Assummptssssssss 1 Eq. (27) satisfies that
and the estimator
C J(x)) J Jāma (T) S max Bg Be, Rmax
d log( Bend- Amin (E)n
with probability at least 1 - 5.
Proof of Theorem 4. Let {(0) and E()) denote the empirical and population vectors:
ē(0) A0 - b and {()) A0 - b.
Recall that A and thus {(0*) = 0. This also implies || ||--1/2((0))) We establish in the sequel the following concentration lemma.
(00))((()))))) )|1 = 0 by invertibility of E.
30

---

Published as a conference paper at ICLR 2026
Lemma 5. With probability at least 8, we have that for all 0 O:
288d log(864
e/()ll -1-11000---11111101)000---11---- I max{ Rmax )
Amin (2)) n
Estat
We also note the following simple technical lemma. Lemma 6. Let M ---1//EE11 1/2, Then, for all v E IR,, we have
v Σー S Amax A T and v - S Amin (M)
E-
Recall that 2 satisfies arg men - 12-1222 We now show that Lemma and Lemma Simply that ||0))/2((0)|l is small. This follows since:
||- 1 = 12 1/2((()) Estat
K max (M) E --)))((0)1 + Estat
Amax M) E 1/2((00))1 )l2 + Estat
Amax (M)
1 1112((000)))) Estat
Amnin (M)
VE(MIE V + (1+ vM(M))
Estat
(I+ VE(M)
Estar
S 2V (M(MM)estta.
In the sequel, we also show concentration for the condition number of M to constant.
Lemma 7. Let 16 we have:
<<X)
B3 + Amax (2)) log(6d/6), Then, with probability least
Amnta [2)
k( <(M) S
This implies that, under the condition on sample size, have |>-1/20(0|| S V12estat with high-probability. We can now conclude the proof. Under the conditions and events stated above, we have:
E Espr (5u ap) Qista au)
E Esorpoo [o(su. To)T(( -mutaa)) )]
(28)
o (0+ (rste)
oo (A-'5- -O1sa
ao A-1 (1. A0
10 A 1//1///22--2 1/2
A0ieta)
11 ATA-'E' 1--'''1--111 112
AÒiera)
A - -00
|||11=11//
1/2 (Arjetá - -b)
<
11/2 A 00
1-11=-1111 1/2
(Ar - -b)
TT -AAAAAAA 0 - 1/2
1
<
/ /
(29)
31

---

Published as a conference paper at ICLR 2026
as desired. We now establish Lemmas to
Proof of Lemma 5) triangle inequality, M ((00)l2 -/-2)2((0) s ||
Let 0 be fixed for now, and A = i(0) - E(0). Note that by the reverse
= =|1> >-12 (A A)O - E 1/ -
We use Vector Bernstein Lemma 9 to show that this is small. Let X, = o(5;, ai), Y, = o(s a,) y0(5%, df),, and (0) X,,y, 0 ri) (A0 b) denote the centered vectors. Note that
1 max (Ix, 0 r2))l2, | bll2].
j (0))2 S Amin
We have the following bound:
r(s, E p(s (l0(s, - r(r( 1)]]
A0 - b
E B[o(s, a)p(s, a)
8]112 + [s(s a') T') 1 + (
- (1 + 7) max (8, a 1)11211012 + max] (6, a) Rmax
8.0
a,a
- 3B, m Bg Be, R.
(30)
We remark that with a similar derivation, this bound applies just as well to |X, - -r:))la so in fact we have
Σ 1/2 (())l < V Amin (2)
Bo max Be, Rmax).
For the variance bound, we simply use that
6 (Vinn
E Σ (
Bø max m Rmuek)
Then, conclude via Lemma 9 that
32 log (2888-1)
| |01\/2(0))2= s BB max(mag Be, Rmax)
Amiu (E)n
We now apply covering argument over E O. Let Op S θ be an Lo-covering of of size N(e). satisfying for for each 0 E e there exists a covering member p(0) E Oo satisfying | p(0) E. Via a simple triangle inequality:
Σ Σー 1/2 ((0) || - =|>~1/2A((00))
+ +12--1/(010) - -
We bound the latter term as a function of
Σー |-))) ((p(0)) || =
(A- -11--1/2AAAA)A -A-))10-(0011
2 max f maxx (A), Tmax (A) }o.
Aminl (E))
We notice that maxf m (aax(A)), (A), m (A) with:
BB via similar reasoning to Eq.
(30)
This leaves us
32 log(288 0g|6 8 1
2B
M 1- 1-1/2
s Bs max max Bo Bmaxty V
Amin (E)r
Amin (Σ)
32d log (8888611///)
2B2
$ By max ByBe, Rmaxty
E,
Amin (2)
Ammin (
32

---

Published as a conference paper at ICLR 2026
where have applied union bound over the set Og. which is of size most (3Be/e))* by standard covering number bounds (Vershynin 2018), since + C {0 Rd ||0||2 s Be } Picking c = 1/ 1//nn lets conclude that, with probability at least 5, for all 0 E e,
for for (0,
|
|| S B& max max Be, Rmaxc
288d log (864 Ban4
mmi EE)n
as desired.
Proof of Lemma 6.
Define E-1122, s0 that E1/2 Then:
Σ v--u=w- w 12//2/-111112 w M
so we have
w S w M S
1 Amin ( (M)
T Σ 1-
W
0,
mmax (M)
Amax (M)
Amin (M)
which establishes both bounds.
Proof of Lemma 7, We firstly establish that
- 2012 8Amax (B) Amax (E)) log(6d/8)
-
Eop-
(31)
To do this, we use Matrix Bernstein (Lemma 8). Abbreviate X₁ == o(s,, a.). and let Z, = X,X,7 - be the centered matrices. Note that | l2
X 1l2 + |E|| I ||x:112 Amnax (E) S B
Amax (.). For the variance term, we have:
[[[ - >)
= | [X, [3] - >
$ E Σ
I Amxx (2) + Amax (2)
This yields
2 2Amax (B) Xmax log(2d/5) 2(B3. + Amax (2d())))
[ - E1|2
3n
The slow term dominates when is large enough:
2(B3 n N Φ + Amax (2:) log (2d())
2
B²
log(2d/0)
+1
(32)
9Amax (2)
9
Amax E
Note that this is implied by our assumption on n, since Amax (E) 2 Amin E) and x<E) 2 1. Thus, under this condition we have
2Amax (E) + Amax (E) log log(2d/))
Σ - S 22
Eopt
n
as desired. Now, note that this implies
Σ i - = IM Tllop 5
Amin (E) Cop == 6M-
Note that assumption implies that
Kr (
n > 16
(B2 + Amaax (E)) log (2d/6)
Amin (2)
CM Amin (E) V 21 2dmax (B) Amax((E) log log(2d/5) - /2
33

---

Published as a conference paper at ICLR 2026
In partieular, this implies so that Amin (M) 2 eas and Amax (M) + Eas which implies
(1 (1-EMM (L-EMM = M M [1+ EM))
1+
<(M) 1 EM
and thus
1+ V
1 + (K(M) S
2 + (1 11--20 28 =2 2 53
11 EM EM
again using that M 1/v2 and the inequality by algebraic manipulations for < 1/v2.
1+0/1-2 2x, which is easily seen be true
F TECHNICAL TOOLS
Lemma 8 (Matrix Bernstein, Tropp 2012). Let Si.... Sn IR xda be random, independent matrices satisfying E[Sx] max E & S ol | Sklop I o², and ||Skllop L almost surely for all k. Then, with probability at least 5 for any E (0, 1),
2x log (d1 + d2)/6)
2L loggoo(ee (d1 + d2)/6)
Sp
3n
k=1
op
Lemma 9 (Vector Bernstein, Minsker 2017 ). Let ;,,... be independent vectors in Rd such that E E[ux] = 0, B[
Un
[8] S of, and 0k 2 I almost surely for all k. Then, with probability at
least 1 for any á (0, 1),
2x log(28/5)
2L log(28/0)
3n
Lemma 10 (Vector Martingale Bernstein (Pinelis
1994
Martinez-Taboade & Ramdas 2024 R adapted to a filtration (FiJesT, such that
Let (Xi)eST be martingale sequence of vectors in
E, [X, = 0, and [xell2 I B, and E E,-1
| I 0². Then, with probability at least
1 8, we have:
2 $ 20 log(2/8) + B log(2/8).
Σ 1=1
X,
3
34


Reviews:
[{"review_id": "toebdECE71", "reviewer": "Reviewer_i914", "summary": "This paper revisits linear off-policy evaluation under mere realizability by\ncasting LSTDQ in an instrumental-variable (IV) form and introducing the\nfeature-dynamics coverage $C^\\pi_\\phi$ that tightens return-estimation guarantees. In doing so, the analysis addresses three shortcomings of\nprior coverage parameters: (i) Scale Invariance: stability to feature\nrescaling, unlike $1/\\sigma_{\\min}(A)$); (ii) Off-Policy Characterization:\nthe dependence on off-policy error propagation rather than only on strict\non-policy boundedness; and (iii) Unification:\n$C^\\pi_\\phi$ aligns with prior coverage concepts, including Bellman completeness, importance weighting and abstract MDP analysis.", "questions": "1. The construction of variables Z and X in the IV-style formulation is introduced without context or intuition upfront, making the initial analysis\nhard to follow. Only later is this formulation justified through IV theory.\n2. Theorem 3, which avoids the drawbacks of 1/σmin(A), is mentioned in\nthe appendix despite directly addressing the key concerns raised in the\nintroduction.\n3. The three issues raised at the beginning of the paper are all addressed\nthrough the coverage form in Proposition 1. In contrast, Theorem 1 seems\nto be more of an intermediate result, and the main storyline becomes\nunclear.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Use $Z=\\phi(s,a)$ as an instrumental variables to solve the \"error in variables\", which is induced by\n    $X=\\phi(s,a)-\\gamma\\,\\phi(s',a')$, yielding a\n    finite-sample value bound.\n\n- The proposed feature-dynamics coverage resolves key deficiencies of prior metrics, by ensuring scale-invariance and meaningful characterization under general off-policy distributions.\n\n- The new definition of coverage via Proposition 1 is elegant, interpretable, and enables unification of various existing notions.", "weaknesses": "1. The motivation for key constructions appears late, making the early sections harder to follow.\n2. The paper could better distinguish the roles of Theorem 1 and Proposition 1 to clarify the main message.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yoyeAXF9cI", "reviewer": "Reviewer_4tkR", "summary": "This paper proposes a novel finite-sample analysis of LSTDQ, based on a new coverage parameter $C_{\\phi}^{\\pi}$ (known as ``feature-dynamics coverage'') derived from an instrument variable (IV) point of view. It also provides sufficient discussion on how the new coverage parameter relate to the existing parameters by showing a good collection of equivalence results.", "questions": "See weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proofs are checked to be mathematically sound.\n2. The perspective of analysis looks new to me.\n3. Section 5 is appreciated since it delivers very clear messages on how to make sense of the newly defined parameter, as well as providing a good collection of equivalence results with existing parameters.", "weaknesses": "1. The so-called ``IV perspective'' that inspires the new results confuses me a bit.\n    * As far as I'm concerned, in a linear model $Y = X^{\\top} \\theta + \\epsilon$, IV is only necessary when $X$ and $\\epsilon$ are not independent. Speaking of intuitions, I don't see why it should be the case here.\n    * It is also a little confusing to refer to Eq. (7) as the linear regression problem, since linear regression shouldn't come with the $\\mathbb{E}$, but rather, with observable individual data points.\n    * The IV perspective totally disappears in the proofs in the appendix. It appears to me that the analysis, at most, borrows tools from the IV literature rather than providing new interpretations of the LSTDQ estimator.\n2. Empirical/numerical results are missing to support the quality of the analysis.\n     * Is it possible to provide some evidence for the advantage of the new coverage parameter? Specifically, is it helpful to include an example where $C_{\\phi}^{\\pi}$ is much tighter than the existing estimators?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5uNQGnMSr", "reviewer": "Reviewer_Jiag", "summary": "The paper presents a finite-sample analysis of linear off-policy evaluation using LSTDQ under the assumption of linear realizability, framed through an instrumental-variable view of the TD equation. Its main technical contribution is the introduction of a scale-invariant \"feature-dynamics coverage\" term, defined in the feature-compressed MDP induced by the given representation, which leads to a value-error bound with the standard $\\sqrt{d/n}$ dependence. A notable aspect is that this coverage quantity is shown to subsume several existing notions: it recovers the usual $\\chi^2$ / density-ratio concentrability in the tabular case, the aggregated concentrability used in abstraction-based work, and the classical linear-coverage condition when Bellman-completeness is additionally assumed. In this way, the paper provides a unifying perspective on when LSTDQ can be expected to succeed off-policy, without proposing a new algorithm but clarifying the role and interpretation of coverage in this widely used setting.", "questions": "Most of my concerns are already detailed in the cons section. I would be happy to raise the score if the authors can address them convincingly. One remaining question: The definition of the feature–dynamics coverage parameter $C_\\phi^\\pi$ is mathematically elegant but somewhat opaque in intuition. It would be helpful if the authors could provide more explanation or intuition for what this quantity measures—e.g., how it relates to state–action visitation overlap or spectral properties of the induced feature dynamics—and how to interpret large or small values of $C_\\phi^\\pi$ in practice. Can the authors give an concrete, real-world example for this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces the feature-dynamics coverage parameter $C_\\phi^\\pi$, providing a unified perspective on coverage in linear off-policy evaluation (OPE). Derived from an IV view of the LSTDQ algorithm, $C_\\phi^\\pi$ quantifies how well features induced by the behavior policy capture the subspace relevant to the target policy. It interprets coverage as occurring within a feature-compressed MDP, linking the environment’s dynamics with the feature representation and offering a scale-invariant, interpretable notion of coverage.\n- The analysis establishes a finite-sample bound for LSTDQ under the realizability assumption, avoiding the stronger Bellman-completeness requirement. The off-policy value estimation error \n\n$$|J_{\\hat{Q}_{\\text{lstd}}}(\\pi) - J(\\pi)|$$\n\nscales as $O(\\sqrt{C^\\pi_{\\phi}   d \\log(1/\\delta) / n})$, matching the optimal linear regression rate for $\\gamma = 0$. Expressing the bound in terms of $C_\\phi^\\pi$ highlights how coverage, feature dimension, and sample size interact. The IV-based formulation offers a clear statistical interpretation of LSTDQ as a solution to an errors-in-variables problem, yielding transparent and tight guarantees.\n- The proposed framework unifies several prior notions of coverage in reinforcement learning. In the tabular case, $C_\\phi^\\pi$ reduces to the $\\chi^2$-divergence between the target and behavior occupancies; under state abstraction, it coincides with aggregated concentrability; and with Bellman-completeness, it becomes the standard linear coverage. This unified formulation connects previously distinct definitions and provides a coherent understanding of coverage across tabular, abstracted, and linear approximation settings.", "weaknesses": "1. The paper focuses on the linear function approximation setting, assuming $Q_\\pi(s,a) = \\phi(s,a)^\\top \\theta^\\star$. This assumption enables a clean finite-sample analysis of the LSTDQ estimator and the introduction of the coverage parameter in Equation (13). However, the framework relies on the invertibility of $\\Sigma$ and $A$ and applies only to the linear regime. Recent work in off-policy evaluation has advanced toward general function approximation via eluder dimension, where representation error and nonlinearity become crucial and closer to real world settings.\n2. Theorem 1 presents a finite-sample error bound of the form \n\n$$|J_{Q^{\\text{lstd}}}(\\pi) - J(\\pi)| \\lesssim \\frac{V_{\\max}}{1 - \\gamma}   \\sqrt{\\hat C^{\\pi}_{\\phi}   \\frac{d \\log(1/\\delta)}{n}}$$ \n\nWhile this expression is elegant, $\\frac{V_{\\max}}{1-\\gamma}$ and the coverage parameter $\\hat{C^{\\pi}_{\\phi}}$ \n\ncan be extremely large, and the feature dimension $d$ may also be high. In practice, for high-dimensional representations or discount factors $\\gamma$ close to $1$ (i.e., long-horizon problems), the required sample size could become prohibitively large. Furthermore, Corollary 1 introduces a burn-in threshold $n_0$, which depends on spectral quantities such as $1/\\sigma_{\\min}(A)$. This implies that the theoretical guarantee may not hold -- or may severely degrade -- when $n < n_0$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ebhK8yQJma", "reviewer": "Reviewer_jTim", "summary": "This work makes an important contribution, studying offline linear policy evaluation via the linear temporal differences method without assuming Bellman completeness.\n\nMoreover, the current submission shows that the analysis of LSTDQ offers a unification in the linear setting of several previous methods initially proposed for the general function approximation setting.", "questions": "1) See weaknesses.\n\n2) How would a linear Monte carlo approach perform in this setting ?  Intuitively a Monte carlo method seems to me more suitable than temporal difference methods when the Bellman completeness assumption is not imposed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)The bound in Corollary 1 scales with the population version of the coverage parameter, not only on the stochastic version.\n\n2) The results are sound and interesting.\n\n3) I really appreciated Section 5.3 and Proposition 2, which makes the comparison with the existing literature very clear.\n\n4) I think that also the unification with MWL is an important contribution.", "weaknesses": "1) I think that the invertibility assumption on $\\Sigma$ and $A$ is quite strong. Could you avoid this assumption adding a small scalar times the identity in the definition of $A$ and $\\Sigma$?\n\n2) Same comment holds for the empirical estimates $\\hat{\\Sigma}$ and $\\hat{A}$\n\n3) Corollary 1 seems to guarantee good approximation of the performance of the policy $\\pi$ starting from states sampled from the initial distribution. \nCan the Q value at states action pairs which are not visited under the initial distribution be estimated reliably with your method ?\n\n4) In proposition 3, in the definition of $C^\\pi_\\phi$ the indices of $k,a$ do not appear inside the expectation, please fix this typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)