Title: Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear $q^\pi$-Realizability and Concentrability
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

learning theory

offline RL

batch RL
Abstract: We consider offline reinforcement learning (RL) in $H$-horizon Markov decision processes (MDPs) under the linear $q^\pi$-realizability assumption, where the action-value function of every policy is linear with respect to a given $d$-dimensional feature function. The hope in this setting is that learning a good policy will be possible without requiring a sample size that scales with the number of states in the MDP. Foster et al. [2021] have shown this to be impossible even under $\text{\textit{concentrability}}$, a data coverage assumption where a coefficient $C_\text{conc}$ bounds the extent to which the state-action distribution of any policy can veer off the data distribution. However, the data in this previous work was in the form of a sequence of individual transitions. This leaves open the question of whether the negative result mentioned could be overcome if the data was composed of sequences of full trajectories. In this work we answer this question positively by proving that with trajectory data, a dataset of size $\text{poly}(d,H,C_\text{conc})/\epsilon^2$ is sufficient for deriving an $\epsilon$-optimal policy, regardless of the size of the state space. The main tool that makes this result possible is due to Weisz et al. [2023], who demonstrate that linear MDPs can be used to approximate linearly $q^\pi$-realizable MDPs. The connection to trajectory data is that the linear MDP approximation relies on "skipping" over certain states. The associated estimation problems are thus easy when working with trajectory data, while they remain nontrivial when working with individual transitions. The question of computational efficiency under our assumptions remains open.

Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear q π -Realizability and Concentrability

Abstract

We consider offline reinforcement learning (RL) in H -horizon Markov decision processes (MDPs) under the linear q π -realizability assumption, where the actionvalue function of every policy is linear with respect to a given d -dimensional feature function. The hope in this setting is that learning a good policy will be possible without requiring a sample size that scales with the number of states in the MDP. Foster et al. [2021] have shown this to be impossible even under concentrability , a data coverage assumption where a coefficient C conc bounds the extent to which the state-action distribution of any policy can veer off the data distribution. However, the data in this previous work was in the form of a sequence of individual transitions. This leaves open the question of whether the negative result mentioned could be overcome if the data was composed of sequences of full trajectories. In this work we answer this question positively by proving that with trajectory data, a dataset of size poly ( d, H, C conc ) /ϵ 2 is sufficient for deriving an ϵ -optimal policy, regardless of the size of the state space. The main tool that makes this result possible is due to Weisz et al. [2023], who demonstrate that linear MDPs can be used to approximate linearly q π -realizable MDPs. The connection to trajectory data is that the linear MDP approximation relies on 'skipping' over certain states. The associated estimation problems are thus easy when working with trajectory data, while they remain nontrivial when working with individual transitions. The question of computational efficiency under our assumptions remains open.

1 Introduction

We study the offline reinforcement learning (RL) setting, where the objective is to derive a nearoptimal policy for an H -horizon Markov decision process (MDP) using offline data . This contrasts with the online RL paradigm, where learners interact directly with an MDP - or its simulator - to collect new data. Offline RL is especially relevant when acquiring new data guided by the learner is infeasible or ill-advised for safety reasons.

Deriving a near-optimal policy is only possible from offline data that covers the MDP well enough. One way to formalize this as an assumption, which we adopt for this work, is called concentrability . This assumption posits that the offline data sufficiently covers the distribution of state-action pairs that are accessible through running any policy. Challenges also arise in MDPs characterized by large or infinite state spaces. In such scenarios, an efficient learner's data requirements cannot scale

with the state space size. An approach to remove state space dependence is to assume that the stateaction value function of any policy can be linearly represented using a d -dimensional feature map, an assumption known as linear q π -realizability .

While linear q π -realizability facilitates efficient online RL [Weisz et al., 2023], its applicability has been limited in offline contexts. For instance, Foster et al. [2021] proves that no learning algorithm can derive an ϵ -optimal policy under linear q π -realizability and concentrability bounded by C conc , with a poly( d, H, C conc , ϵ -1 ) number of samples. However, their result does not apply to trajectory data , where the offline data contains full sequences of state, action, and reward tuples obtained by following some policy from the initial state to the terminal state. The following problem is left open:

'Does there exist an efficient learner that outputs an ϵ -optimal policy, under the assumptions of linear q π -realizability, concentrability, and trajectory data?'

Our findings affirmatively answer this question in terms of sample complexity, highlighting a notable distinction in the requirements for trajectory data versus general offline data for effective learning. This underscores the practical value of accumulating trajectory data whenever feasible.

2 Related Works

In Table 1 we provide a comparison of our result to the other works in offline RL discussed below.

Lower bounds: As we have already discussed in Section 1, the work of Foster et al. [2021] shows a lower bound that depends on the size of the state space (in the same setting as ours), except they do not assume access to trajectory data. The work by Jia et al. [2024] is perhaps the most relevent to ours. They show an exponential lower bound in the horizon for policy evaluation, under the assumptions of trajectory data, concentrability, and a restricted linear q π -realizability where the value function of only the target policy is linear. While we anticipate that evaluating policies (their focus) is no more difficult than optimizing policies (our focus), our q π realizability is for all memoryless policies (Assumption 1), while theirs is restricted to the target policy. Zanette [2021] shows an exponential lower bound in terms of the feature dimension d , under linear q π -realizability, and various other structural assumptions; however, their setting would result in a concentrability coefficient larger than the size of the state space. Wang et al. [2020], Amortila et al. [2020] show a lower bound that is exponential in the horizon, under linear q π -realizability. However, they use a λ min lower bound condition, which requires a lower bound on the minimum eigenvalue λ min of the expected covariance matrix used for least-squares estimation. This is seen as a weaker condition than ours, as it only posits good coverage in terms of the feature space, not the (possibly much richer) state-action space.

Upper bounds: Chen and Jiang [2019], Munos and Szepesvári [2008] present efficient algorithms under concentrability and Bellman completeness , an assumption that the Bellman optimality operator outputs a linearly realizable function when its input is linearly realizable. As linear q π -realizability does not imply Bellman completeness [Zanette et al., 2020], these results do not transfer

to our setting. Xie and Jiang [2021] show an upper bound under linear q π -realizability, albeit, using a stronger notion of data coverage than concentrability, which we call strong concentrability . The work of Xie et al. [2021, 2022] give data-dependent sample complexity bounds that hold under both Bellman completeness and linear q π -realizability even in the absence of explicit data coverage assumptions. Jin et al. [2021] assume a general function approximation setting and also provide data-dependent bounds, while Duan et al. [2020], Xiong et al. [2022] show upper bounds for linear MDPs (a stronger assumption than linear q π -realizability [Zanette et al., 2020]) with the λ min lower bound condition.

3 Setting

Throughout we fix the integer d ≥ 1 . Let ⃗ 0 ∈ R d be the d -dimensional, all zero vector. For L > 0 , let B ( L ) = { x ∈ R d : ∥ x ∥ 2 ≤ L } denote the d -dimensional Euclidean ball of radius L centered at the origin, where ∥ · ∥ 2 denotes the Euclidean norm. The inner product ⟨ x, y ⟩ for x, y ∈ R d is defined as the dot product x ⊤ y . Let /x31 { B } be the indicator function of a boolean-valued (possibly random) variable B , taking the value 1 if B is true and 0 if false. Let M 1 ( X ) denote the set of probability distributions over the set X . Let E B ∼P denote the expectation of random variable B under distribution P . For integers i, j , let [ i ] = { 1 , 2 , . . . , i } and [ i : j ] = { i, . . . , j } . For a symmetric matrix M ∈ R d × d we write λ min ( M ) and λ max ( M ) for its minimum and maximum eigenvalue.

̸

The environment is modeled by a finite horizon Markov decision process (MDP). Fix the horizon to H . This MDP is defined by a tuple ( S , A , P, R ) . Here, the state space S is finite 1 , and organized by stages: S = ⋃ h ∈ [ H +1] S h , starting from a designated initial state s 1 ( S 1 = { s 1 } ) 2 , and culminating in a designated terminal state s ⊤ ( S H +1 = { s ⊤ } ) 3 . Without loss of generality, we assume S h and S h ′ for h = h ′ are disjoint sets. Define the function stage : S → [ H +1] , such that stage( s ) = h if s ∈ S h . The action space A is finite. The transition kernel is P : ( ⋃ h ∈ [ H ] S h ) ×A → M 1 ( S ) , with the property that transitions occur between successive stages. Specifically, for any h ∈ [ H ] , state s h ∈ S h , and action a ∈ A , P ( s h , a ) ∈ M 1 ( S h +1 ) . The reward kernel is R : S × A → M 1 ([0 , 1]) . So that the terminal state s ⊤ has no influence on the learner we force the reward kernel to deterministically give zero reward for all actions a ∈ A in s ⊤ (i.e. R ( s ⊤ , a )( r ) = /x31 { 0 = r } ). An agent interacts with this environment sequentially across an episode of H +1 stages, by selecting an action a ∈ A in the current state. The environment (except at stage H +1 ) then transitions to a subsequent state according to P and provides a reward in [0 , 1] as specified by R 4 .

We define an agent's interaction with the MDP through a policy π , which assigns a probability distribution over actions based on the history of interactions (including states, actions, and rewards). For this work, we restrict policies to be memoryless , that is, their action distribution depends solely on the most recent state in the history. The set of all memoryless policies is Π = { π : π : S → M 1 ( A ) } . For π ∈ Π , we write π ( a | s ) to denote the probability π ( s ) assigns to action a . For deterministic policies only (i.e., those that for each state place a unit probability mass on some action) we sometimes abuse notation by writing π ( s ) to denote arg max a ∈A π ( a | s ) . Starting from any state s within the MDP and using a policy π induces a probability distribution over trajectories, denoted as P π,s . For any a ∈ A , P π,s,a is the distribution over the trajectories when first action a is used in state s , after which policy π is followed. Specifically, for some h ∈ [ H +1] and ( s, a ) ∈ S h ×A , we write Traj ∼ P π,s,a to denote that Traj = ( S h , A h , R h , . . . , S H +1 , A H +1 , R H +1 ) for a random trajectory that follows the distribution specified by P π,s,a , that is, S h = s , A h = a , A i ∼ π ( S i ) for i ∈ [ h +1 : H +1] , S i +1 ∼ P ( S i , A i ) for i ∈ [ h : H ] , and R i ∼ R ( S i , A i ) for i ∈ [ h : H +1] . Writing Traj ∼ P π,s has an analogous meaning, with the only difference being that A h is not fixed,

and instead A h ∼ π ( S h ) . For h ∈ [ H +1] , we write P h π,s (and P h π,s,a ) for the marginal distribution of ( S h , A h ) (i.e., the state-action pair of stage h ) under the joint distribution of P π,s (and P π,s,a ).

For 1 ≤ t ≤ t ′ ≤ H +1 , we use the notation x t : t ′ = ( x u ) u ∈ [ t : t ′ ] throughout, except when ( x u ) u ∈ [ t : t ′ ] are a sequence of scalar rewards. In that case, for convenience, we write r t : t ′ = ∑ t ′ u = t r u and R t : t ′ = ∑ t ′ u = t R u . The state-value and action-value functions v π and q π are defined as the expected total reward along the rest of the trajectory while π is used:

Let π ⋆ ∈ Π be an optimal policy, satisfying q π ⋆ ( s, a ) = sup π ∈ Π q π ( s, a ) for all ( s, a ) ∈ S ×A . Let q ⋆ ( s, a ) = q π ⋆ ( s, a ) and v ⋆ ( s ) = max a ∈A q ⋆ ( s, a ) for all ( s, a ) ∈ S × A . By definition, we have

3.1 Assumptions and Problem Statement

A feature map is defined as ϕ : S × A → B ( L 1 ) for some L 1 > 0 . The representative power of a feature map for an MDP is described by the following assumption:

Assumption 1 ( ( η, L 2 ) -Approximately Linear q π -Realizable MDP) . For some η ≥ 0 , L 2 > 0 , assume that the MDP (together with a feature map ϕ ) is such that

For any h ∈ [ H +1] , let ψ h : Π →B ( L 2 ) be a mapping from policies to parameter values θ h that attain the min in the above display. For h = H +1 , we restrict this mapping to ψ H +1 ( · ) = ⃗ 0 , which satisfies the above display by definition. We write ψ h : t ( π ) for ( ψ h ( π ) , . . . , ψ t ( π )) .

We also make the following assumptions on the offline data (the relationship to non-trajectory data and the negative result by Foster et al. [2021] is discussed in Appendix B) :

Assumption 2 (Full Length Trajectory Data) . Assume the learner is given a dataset of full length trajectories 5 and corresponding features of size n ≥ 1 :

where for some 'data collection policy' π 0 ∈ Π unknown to the learner, ( traj j ) n j =1 are independent samples from P π 0 ,s 1 where traj j = ( s j t , a j t , r j t ) t ∈ [ H +1] . To simplify notation we write

Definition 1 (Admissible Distribution) . Asequence of H state-action distributions ν = ( ν h ) h ∈ [ H ] ∈ ( M 1 ( S h ×A )) H is admissible for an MDP if there exists a policy π ∈ Π such that

Define the state-action occupancy measure of the data collection policy π 0 as µ = ( µ h ) h ∈ [ H ] such that

Assumption 3 (Concentrability) . Assume there exists a constant C conc ≥ 1 , such that for all admissible distributions ν = ( ν h ) h ∈ [ H ]

Problem 1. Let ϵ > 0 . Under Assumptions 1 to 3, does there exist a learner, with access to only n = poly(1 /ϵ, H, d, C conc , log(1 /δ ) , log(1 /L 1 ) , log(1 /L 2 )) full length trajectories (as defined in Assumption 2), that outputs a policy π such that, with probability at least 1 -δ ,

4 Result

We resolve Problem 1 in the positive by defining a learner (Algorithm 1) that: selects parameters optimistically from modified MDPs that 'skip over' certain states while preserving tight q -value estimation guarantees (achieved by solving Optimization Problem 1); then, outputs a greedy policy π ′ (defined in line 3) over the selected parameters. This result is made formal in following theorem (proof in Section 5):

Theorem 1. Let ϵ ∈ (0 , H ] . Under Assumptions 1 to 3, if the number of full length trajectories n = ˜ Θ( C 4 conc H 7 d 4 /ϵ 2 ) and η = ˜ O ( ϵ 2 / ( C 2 conc H 5 d 2 ) ) 6 , then, with probability at least 1 -δ , the policy π ′ output by our learner (Algorithm 1) satisfies

where ˜ Ω , ˜ O and ˜ Θ are the counterparts of Ω , O and Θ from the big-Oh notation that hide polylogarithmic factors of the problem parameters (1 /ϵ, 1 /δ, H, d, C conc , L 1 , L 2 ) . The following subsections focus on introducing the theory needed to formally present our learner, giving intuition behind our learner, and presenting our learner.

4.1 Background Theory

Our learner relies on the observation due to Weisz et al. [2023] that linearly q π -realizable MDPs are linear MDPs, as long as they contain no low-range states. The range of a state is the largest possible regret from that state, that is, the largest difference in action-value that the choice of action in that state can make (up to misspecification):

where ϕ ( s, a, a ′ ) = ϕ ( s, a ) -ϕ ( s, a ′ ) is a notation we use to denote feature differences. Intuitively, the choice of actions in low-range states are unimportant, as

As a warm-up, consider the example MDPs shown in Fig. 1. We will transform the linearly q π -realizable MDP on the left into a linear MDP on the right by 'skipping over' the red low-range states. Let the features for both MDPs be ϕ ( s 1 , · ) = (1) , ϕ ( s 3 , · ) = (0 . 5) , ϕ ( · , · ) = (0) otherwise. Then the left MDP is (0 , 1) -approximately q π -realizable, with realizability parameter ψ h ( π ) = (1) for all h ∈ [ H +1] , π ∈ Π . However, it is not a linear MDP, since the rewards cannot be represented by a linear function of the features. To see this, notice that there exists no θ such that ⟨ ϕ ( s 1 , a 1 ) , θ ⟩ = r ( s 1 , a 1 ) = 1 and ⟨ ϕ ( s 1 , a 2 ) , θ ⟩ = r ( s 1 , a 2 ) = 0 . 5 , since ϕ ( s 1 , a 1 ) = ϕ ( s 1 , a 2 ) = (1) . We modify this MDP on the left to 'skip over' low-range red states, by automatically taking the first available action at such states, and summing up the rewards along skipped paths. This turns the MDP into the one on the right of Fig. 1, which is a linear MDP.

s

1

s

2

s

3

s

4

1

0.5

0

0

0.5

0.5

0

s

1

s

4

1

1

The key fact about linear MDPs that we will use is that for any function f : S → [0 , H ] (e.g., v -value approximators), and any h ∈ [ H ] , there is some parameter θ h ∈ R d so that for any ( s, a ) ∈ S h ×A , ⟨ ϕ ( s, a ) , θ h ⟩ gives the expectation of the reward plus f 's value on the next state. In our modified

MDP this result transfers to the fact that the expected sum of rewards along a skipped path, plus f 's value on the next state after the skipped path, is linearly realizable. Before making this result formal in Lemma 4.2, we clarify the skipping behavior.

First, we address the fact that we need an approximate, parametric bound on range ( · ) with a parameter count that is independent of |S| . For h ∈ [2 : H ] , let Ψ h = { ψ h ( π ) : π ∈ Π } ⊆ B ( L 2 ) be the (compact) set of parameter values corresponding to all policies. For all h ∈ [2 : H ] , fix a subset ¯ G h ⊂ Ψ h of size | ¯ G h | = d 0 := ⌈ 4 d log log( d ) + 16 ⌉ that is the basis of a near-optimal design for Ψ h (more precisely, satisfying Definition 3). The existence of such a near-optimal design follows from [Todd, 2016, Part (ii) of Lemma 3.9]. Let ¯ G = ¯ G 2: H , which we call the true guess . Now notice that ¯ G ∈ G where

For G ∈ G we will use the notation that G = G 2: H , where G h = ( ϑ i h ) i ∈ [ d 0 ] ∈ B ( L 2 ) d 0 . Any G = G 2: H ∈ G can be used to define an approximate, low parameter-count 'version' of range that is completely specified by ˜ O ( Hd 2 ) parameters:

As shown in Proposition 4.5 of [Weisz et al., 2023], range ¯ G

can be used to bound the true range: ∈ ∈ S ≤ √ · ¯ .

Based on any G ∈ G , we are interested in simulating a modified MDP that 'skips over' states s that have a low range G ( s ) , by taking an action according to π 0 , and presenting as the reward the summed up rewards along paths consisting of skipped states. This 'modified MDP' only serves as intuition, and will not be formally defined or used in our formal arguments. Instead, we define the 'skipping probability' parameter at state s ∈ S , with α > 0 (set later in Eq. (35)), as

The skipping behavior is probabilistic 7 : it never skips for stages 1 and H + 1 (where range G is not defined); it always skips for ranges lower than some threshold, never skips for ranges higher than twice this threshold, and linearly interpolates between the two in between the thresholds. For h ∈ [ H ] , and 1 ≤ l ≤ h , let traj = ( s t , a t , r t ) l ≤ t ≤ H +1 be any fixed trajectory that starts from some stage l . Let τ ∼ F G, traj ,h +1 ∈ M 1 ([ h +1 : H +1]) be the random stopping stage, when starting from state s h and skipping subsequent states with probability ω G ( · ) . Formally, for t ∈ [ h +1 : H +1] let F G, traj ,h +1 ( τ = t ) = (1 -ω G ( s t )) ∏ t -1 u = h +1 ω G ( s u ) . We will often write F j G,h +1 to denote F G, traj j ,h +1 where traj j = ( s j t , a j t , r j t ) t ∈ [ H +1] , j ∈ [ n ] .

Next, we present a key tool derived from results of Weisz et al. [2023]: as long as the skips are informed by the true guess, the resulting MDP is approximately linear (proof in Appendix C):

Lemma 4.2 (Approximate Linear MDP under the true guess) . Let η ≥ 0 , L 2 > 0 . Let M be an ( η, L 2 ) -approximately linear q π -realizable MDP (Assumption 1) with corresponding feature map ϕ . Let ˜ L 2 = L 2 (8 H 2 d 0 /α +1) . Then, for each f : S → [0 , H ] with f ( s ⊤ ) = 0 , policy π ∈ Π , and stage h ∈ [ H ] , there exists a parameter ρ π h ( f ) ∈ B ( ˜ L 2 ) such that for all ( s, a ) ∈ S h ×A , where ˜ η = η (10 H 2 d 0 /α +1) .

4.2 The Benefit of Trajectory Data

Our learner will heavily rely on the result presented in Lemma 4.2. We will need to learn good estimates of the parameters ρ π 0 h ( f ) , for any f : S → [0 , H ] , h ∈ [ H ] . However, to estimate a ρ π 0 h ( f )

parameter well we will require least-squares targets that have bounded noise and expectation equal to ⟨ ϕ ( s, a ) , ρ π h ( f ) ⟩ for all ( s, a ) ∈ S h ×A . Full trajectory data (Assumption 2) makes this possible. Each full length trajectory traj j = ( s j t , a j t , r j t ) t ∈ [ H +1] j ∈ [ n ] can be used to create the following least-squares target (which has the desired properties):

Importantly, it is because we have full length trajectories that we can transform the data available to simulate arbitrary length skipping mechanisms.

4.3 Intuition Behind our Learner

Next, we describe the high-level intuition and ideas behind our learner. Consider the 'modified' MDP where low-range states are skipped. As the learner has access to trajectory data (Assumption 2), it can transform this data accordingly to simulate trajectories from the modified MDP. Any near-optimal policy for the modified MDP is also near-optimal for the original MDP (due to Eq. (3)). Thus, our previous linear realizability property (Lemma 4.2) allows for an offline RL version of the algorithm ELEANOR [Zanette et al., 2020] to statistically efficiently derive a near-optimal policy for the modified MDP. Indeed, the optimization problem underlying ELEANOR serves as a starting point for Optimization Problem 1, which is at the heart of our learner.

The challenge is that the true guess ¯ G that Lemma 4.2 relies upon is not known to the learner. This means that the learner is not given any explicit information of what states to 'skip over'. To overcome this, we design a learner to output the policy π ′ (defined in Algorithm 1) based on Optimization Problem 1, where the optimization problem considers all guesses for the possible values of ¯ G . For each G ∈ G , it considers the MDP that skips over low-range states when the range is calculated according to G . It then calculates sets Θ G,h for each stage h , that are guaranteed (with high probability) to include the parameter ψ h ( π ⋆ G ) realizing q π ⋆ G (where π ⋆ G , defined in Eq. (15), is the optimal policy in the MDP with skipping based on G ). We achieve this by defining Θ G,h backwards for h = H,H -1 , . . . , 1 . By induction, if Θ G,h +1 , . . . , Θ G,H all contain the desired parameter for their stage, then some parameter sequence in the Cartesian product Θ G,h +1 × · · · × Θ G,H allows us to near-perfectly (up to some misspecification error) compute q π ⋆ G -values of stages > h . Therefore, the least-squares parameter based on this sequence will be near the true parameter for stage h . Defining ˆ Θ G,h to be all least-squares predictors for sequences in the aforementioned Cartesian product, and Θ G,h to be unions of the confidence ellipsoids around these predictors ensures the true parameter ψ h ( π ⋆ G ) realizing q π ⋆ G for stage h is included in Θ G,h . This argument is made precise in Lemma D.2.

There are two problems remaining. One is that some values of G considered by Optimization Problem 1 lead to skipping over important large-value states, degrading the performance of the best policy π ⋆ G available under that skipping. The other problem is that at the expense of making sure the true parameters are included in the sets Θ G,h , these sets might become large, in the sense of containing parameters that lead to very different predictions. Avoiding the first problem would make v π ⋆ G ( s 1 ) nearly as large as v ⋆ ( s 1 ) . Avoiding the second problem would lead to tight q -value estimators, and therefore to v π ′ G ( s 1 ) being nearly as large as v π ⋆ G ( s 1 ) , for a policy π ′ G that is greedy with respect to our hypothetically tight q -value estimator. A key idea is to reject from consideration any G ∈ G that leads to q -estimations that are not sufficiently tight (Eq. (14)). The reason we can do this is because for G = ¯ G we can show that this condition passes (with high probability), and therefore we do not reject ¯ G (precise statement in Lemma D.3). We can show this since we have trajectory data (Assumption 2), allowing us to use least-squares targets of the form used in Eq. (12), which we know are linearly realizable when G = ¯ G (discussed in Section 4.2). Finally, we resolve the first problem by selecting among these tight estimators the one that guarantees the highest policy value from s 1 , which can be no worse than the value guaranteed by the choice of G = ¯ G , which itself can be seen to be close to v ⋆ ( s 1 ) .

4.4 Learner

Next, we formally introduce our learner, at the heart of which lies Optimization Problem 1. We define various q and v -value estimators that we use. For x ∈ R , let clip [0 ,H ] x = max { 0 , min { H,x }} .

Then, for h ∈ [ H ] , s ∈ ∪ t ∈ [ h : H +1] S t , a ∈ A , θ ∈ R d , θ h : H +1 = ( θ h , . . . , θ H +1 ) ∈ R d ( H -h +2) , let

Optimization Problem 1.

Let ( G ′ , θ ′ 1: H +1 ) denote the solution to Optimization Problem 1. Notice that unlike ELEANOR, we optimize over all G ∈ G , which can be seen as an optimization over all possible 'modified MDPs' with different skipping mechanisms. Another observation is that apart from h = 1 , the choice of θ ′ h from Θ G ′ ,h made by the optimization is arbitrary. Indeed, unlike ELEANOR, which chooses globally optimistic least-squares predictors for each stage, our optimization does not need to care about (or optimize for) the specific choice of q -value predictors from their respective confidence sets Θ G,h . We can be agnostic to the choice of q -value predictors because all choices lead to similar predictions due to Eq. (14), as will be shown formally, in Lemma 5.1. However, fixing an arbitrary concrete choice in the optimization allows us to define an output policy π ′ that is parametrized only by these vectors, making both the memory and computational requirements of representing and executing π ′ small.

Our learner (Algorithm 1) solves Optimization Problem 1 and uses θ ′ 1: H +1 to output a greedy policy.

Algorithm 1: Learner

1: input: accuracy ϵ > 0 , failure probability δ > 0 , concentrability coefficient C conc < ∞ , trajectories ( traj 1 , . . . , traj n ) , features ( ( ϕ ( s 1 h , a )) h ∈ [ H ] ,a ∈A , . . . , ( ϕ ( s n h , a )) h ∈ [ H ] ,a ∈A ) , norm bounds L 1 , L 2 .

2: G ′ , θ ′ 1: H +1 ← solution to Optimization Problem 1

5 Proof of Theorem 1

Before giving the proof, we formally define the optimal policy in the modified MDP that skips according to range G , for any G ∈ G as

Notice that for states s ∈ S h for some h ∈ [ H ] , π ⋆ G ( ·| s ) in the above definition depends on the value of q π ⋆ G ( s ′ , · ) for some s ′ ∈ S h +1 . We can therefore interpret the above recursive definition as defining π ⋆ G ( ·| s ) for s ∈ S h , first for h = H + 1 , then h = H , etc., down to h = 1 . Every time we define the policy for some stage h in such a way, the policy and q π ⋆ G ( s ′ , · ) are already defined on later stages, making the definition valid, and resolving the recursive nature of Eq. (15).

Proof. v ⋆ ( s 1 ) -v π ′ ( s 1 ) can be decomposed into the following error terms.

The remainder of the proof focuses on bounding these error terms. Following the intuition described in Section 4, showing that terms ( I ) and ( II ) are small can be seen as addressing the first problem of potentially skipping over large-value states, while showing that term ( III ) is small can be seen as addressing the second problem of π ′ being greedy w.r.t. to a potentially inaccurate estimates θ ′ 1: H +1 .

Bounding ( I ) = v ⋆ ( s 1 ) -v π ⋆ ¯ G ( s 1 ) : This term cannot be too large since the range ¯ G function is approximately correct (Lemma 4.1), and we only skip over states with low range ¯ G (Eq. (6)), implying the action we take doesn't affect the value function much (Eq. (3)). In Appendix D.1 we formalize this intuition, and show the following result.

Bounding ( II ) = v π ⋆ ¯ G ( s 1 ) -¯ v θ ′ 1 ( s 1 ) : This term can be bounded by approximately zero due to Optimization Problem 1 being optimistic from the start state. First, note that v π ⋆ ¯ G is approximately equal to ¯ v ψ 1 ( π ⋆ ¯ G ) (Assumption 1). Then, in Lemma D.2 we show that ψ 1 ( π ⋆ ¯ G ) ∈ Θ ¯ G, 1 , and in Lemma D.3 we show that ¯ G is a feasible solution to Optimization Problem 1. Since ( G ′ , θ ′ 1: H +1 ) is the solution to Optimization Problem 1, it holds that ¯ v θ ′ 1 ( s 1 ) ≥ ¯ v θ ( s 1 ) for any θ ∈ Θ G, 1 where G is a feasible solution to Optimization Problem 1. Thus ¯ v θ ′ 1 ( s 1 ) ≥ v ψ 1 ( π ⋆ ¯ G ) ( s 1 ) . In Appendix D.2 we formalize this intuition, and show that with probability at least 1 -δ ,

Bounding ( III ) = ¯ v θ ′ 1 ( s 1 ) -v π ′ ( s 1 ) : To bound term ( III ) we will first show in Lemma 5.1 that value estimates in terms of θ ′ h and ψ h ( π ⋆ G ′ ) are close with high probability for all h ∈ [ H +1] . This lemma allows us to relate ¯ v θ ′ 1 to ¯ v ψ 1 ( π ⋆ G ′ ) and then Assumption 1 relates ¯ v ψ 1 ( π ⋆ G ′ ) to v π ⋆ G ′ . We are then left with relating v π ⋆ G ′ to v π ′ . To do this we claim that π ′ is an approximate policy improvement step w.r.t. v π ⋆ G ′ , which can be seen by recalling that π ′ is greedy w.r.t. ¯ q θ ′ 1: H +1 , and as we mentioned a couple sentences ago, ¯ v θ ′ h and ¯ v ψ h ( π ⋆ G ′ ) are close for all h ∈ [ H +1] .

To formalize this intuition we begin by decomposing ¯ v θ ′ 1 ( s 1 ) -v π ′ ( s 1 ) into the following error terms

To bound ¯ v θ ′ 1 ( s 1 ) -¯ q ψ 1 ( π ⋆ G ′ ) ( s 1 , π ′ ( s 1 )) we introduce a useful lemma (proof in Appendix F).

Lemma 5.1. There is an event E 2 , that occurs with probability at least 1 -δ/ 3 , such that under event E 2 , for all G ∈ G that are feasible solutions to Optimization Problem 1, for all h ∈ [ H ] , for all ( θ s,a ) ( s,a ) ∈S h ×A and ( ˇ θ s,a ) ( s,a ) ∈S h ×A ∈ Θ S h ×A G,h , and for all admissible distributions ν = ( ν t ) t ∈ [ H ] , it holds that

To use Lemma 5.1 we must show that ¯ v θ ′ 1 ( s 1 ) -¯ q ψ 1 ( π ⋆ G ′ ) ( s 1 , π ′ ( s 1 )) satisfies its requirements. First, note that ¯ v θ ′ 1 ( s 1 ) = ¯ q θ ′ 1 ( s 1 , π ′ ( s 1 )) , by definition of π ′ (line 3). Second, G ′ is the solution to Optimization Problem 1, thus, a feasible solution. Third, by Lemma D.2, there is an event E 1 , which occurs with probability at least 1 -δ/ 3 , such that under event E 1 , ψ 1 ( π ⋆ G ′ ) ∈ Θ G ′ , 1 , and by definition θ ′ 1 ∈ Θ G ′ , 1 . Let ν h ( s, a ) = P h π ′ ,s 1 ( s, a ) for all h ∈ [ H ] , ( s, a ) ∈ S h ×A . Clearly ν = ( ν h ) h ∈ [ H ] is an admissible distribution by Definition 1. Thus, under event E 1 ∩ E 2 , by Lemma 5.1,

It is left to bound ¯ q ψ 1 ( π ⋆ G ′ ) ( s 1 , π ′ ( s 1 )) -v π ′ ( s 1 ) in Eq. (18). To do this, first note that

where the inequality holds since we have approximate linear q π -realizability (Assumption 1). To bound q π ⋆ G ′ ( s 1 , π ′ ( s 1 )) -v π ′ ( s 1 ) notice that v π ′ ( s 1 ) = q π ′ ( s 1 , π ′ ( s 1 )) , which implies that

Next, we give a bound on E Traj ∼ P π ′ ,s 1 [ v π ⋆ G ′ ( S 2 ) -v π ′ ( S 2 ) ] (proof in Appendix D.3):

Lemma 5.2. Under event E 1 ∩ E 2 , for any h ∈ [2 : H +1] , it holds that

Intuitively, the above lemma holds since π ′ can be thought of as an approximate policy improvement step w.r.t. v π ⋆ G ′ . To see this, recall that π ′ is greedy w.r.t. ¯ q θ ′ 1: H +1 (line 3). Then, with Lemma 5.1, we can show ¯ v θ ′ h and ¯ v ψ h ( π ⋆ G ′ ) (which is close to v π ⋆ G ′ (Assumption 1)) are close for all h ∈ [ H +1] . The above bounds imply that under event E 1 ∩E 2 , which occurs with probability at least 1 -2 δ/ 3 ,

Combining the Bounds: To finish the proof we combine the bounds on all three terms (Eqs. (16), (17) and (19)), to get that under event E 1 ∩ E 2 ∩ E 3 , which occurs with probability at least 1 -δ ,

To bound the above display by ϵ we set α = ϵ/ (12( H + 1)) < 1 . If n = ˜ Θ ( C 4 conc H 7 d 4 /ϵ 2 ) and η = ˜ O ( α/ √ nH ) (Eq. (26)), we show that ˜ ϵ = ˜ O ( C 2 conc H 5 / 2 d 2 / √ n ) (Eq. (44)). This implies that

6 Limitations and Conclusions

In this work we resolved an open problem in the positive, by presenting the first statistically efficient learner (Section 4.4) that outputs a near optimal policy in the offline RL setting with approximate linear q π -realizability (Assumption 1), trajectory data (Assumption 2), and concentrability (Assumption 3). One limitation of this work is that we are not aware of any computationally efficient implementation of Optimization Problem 1, which is at the heart of our learner. As such, it is left as an open problem whether computationally efficient learning is possible in the setting we considered. Another limitation is that we are not sure if our statistical rate in Theorem 1 is optimal. Showing a matching lower bound or improving the rate is left for future work.

Another limitation of our work originates from our setting underpinning our result (Section 4), namely the three assumptions: approximate linear q π -realizability, trajectory data, and concentrability. Approximate linear q π -realizability requires the value function of all memoryless policies to be linear in a fixed and known d -dimensional feature map. While strictly weaker than the linear MDP assumption [Zanette et al., 2020], this assumption is still strong. Trajectory data requires full sequences of interactions with an environment to be collected by a single policy. For long horizon problems this can be practically challenging. Concentrability requires the state and action spaces to be well-covered. This can be challenging to guarantee since often the state and action spaces are unknown at the time of data collection. Further, since we require the trajectory data to be collected by a single policy, it may be the case that no single policy exists that covers the state and action spaces well, and a mixture of policies must be considered, which our current result does not immediately hold for. Although the assumptions appear strong, a justification for them is that under many variations of weaker assumptions (for instance: general data, or linear q π -realizability of only one policy, or only coverage of the feature space), polynomial statistical rates have been shown to be impossible to achieve by any learner (Table 1).

Since this work is focused on foundational theoretical research it is unlikely to have any direct and immediate societal impacts.

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Amortila, N. Jiang, and T. Xie. A variant of the wang-foster-kakade lower bound for the discounted setting. arXiv preprint arXiv:2011.01075 , 2020.

Cai, Z. Yang, C. Jin, and Z. Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Chen and N. Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Duan, Z. Jia, and M. Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning , pages 2701-2709. PMLR, 2020.

J. Foster, A. Krishnamurthy, D. Simchi-Levi, and Y. Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. arXiv preprint arXiv:2111.10919 , 2021.

Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding , pages 409-426, 1994.

Jia, A. Rakhlin, A. Sekhari, and C.-Y. Wei. Offline reinforcement learning: Role of state aggregation and trajectory data. arXiv preprint arXiv:2403.17091 , 2024.

Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

J. Todd. Minimum-volume ellipsoids: Theory and algorithms . SIAM, 2016.

Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Wang, D. P. Foster, and S. M. Kakade. What are the statistical limits of offline rl with linear function approximation? arXiv preprint arXiv:2010.11895 , 2020.

Weisz, A. György, and C. Szepesvári. Online rl in linearly qpi-realizable mdps is as easy as in linear mdps if you learn what to ignore. arXiv preprint arXiv:2310.07811 , 2023.

Xie and N. Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021.

Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Xie, M. Bhardwaj, N. Jiang, and C.-A. Cheng. Armor: A model-based framework for improving arbitrary baseline policies with offline data. arXiv preprint arXiv:2211.04538 , 2022.

Xiong, H. Zhong, C. Shi, C. Shen, L. Wang, and T. Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 , 2022.

Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning , pages 1228712297. PMLR, 2021.

Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Appendix

A Parameter Settings and Notation

B Trajectory Data vs. Non-Trajectory Data

To compare the two types of data, we first define 'non-trajectory' data, which consist of individual transitions without any guarantees of them coming from complete MDP trajectories.

Assumption 4 (Non-Trajectory Data) . Assume that for each h ∈ [ H ] the learner is given a dataset of transition tuples and corresponding features of size n ≥ 1 :

where ( s j h , a j h ) ∼ µ ′ h , r j h ∼ R ( s j h , a j h ) , ¯ s j h +1 ∼ P ( s j h , a j h ) , for all j ∈ [ n ] , and µ ′ h ∈ M 1 ( S h ×A ) is a 'data collection distribution' unknown to the learner.

There are two differences between the above non-trajectory data and trajectory data as defined in Assumption 2. In particular, if µ ′ h = P h π 0 ,s 1 and ¯ s j h +1 = s j h +1 , then we get back the trajectory data assumption.

Foster et al. [2021] showed a negative result under Assumption 4. Our method addresses the hard instance from [Foster et al., 2021] if the data is given as complete trajectories. Below we explain why the lower bound constructions from [Foster et al., 2021] break down if they need to use trajectory data, and why our algorithm breaks down if it doesnt have trajectory data.

The lower bound constructions in Theorems 1.1 and 1.2 of [Foster et al., 2021] were both made hard because the data collection distributions of individual transition tuples ( s, a, r, ¯ s ) were selected such that they reveal no (or almost no) information about the MDP instance. In both cases, receiving samples from the joint distribution of the entire trajectory P π 0 ,s 1 makes the problem easy. In the case of Theorem 1.1, one would simply observe which states are reachable from the start state (the planted states). For Theorem 1.2, some information on whether any next-state ¯ s is planted or not would be leaking in each trajectory, in the form of being able to observe the next-state transition from exactly ¯ s .

A simpler example showing the root of the problem with non-trajectory data is as follows. Consider the toy problem of learning the value of some policy π after taking action a in state s 1 in a 2stage MDP. The data is given as tuples of the form ( s 1 , a, r 1 1 , ¯ s 1 2 , . . . , s 1 , a, r n 1 , ¯ s n 2 ) for the first stage and ( s 1 2 , a 1 2 , r 1 2 , ¯ s 1 3 , . . . , s n 2 , a n 2 , r n 2 , ¯ s n 3 ) for the second stage. Notice there is no guarantee that s j 2 ∼ P ( s 1 , a ) with j ∈ [ n ] . We cannot infer what the rewards from the second-stage states distributed as P ( s 1 , a ) might look like from the data, making this problem hopelessly hard. In the extreme, the MDP might have infinitely many second-stage states, with the probability of any ¯ s j 2 = s k 2 (for any j and k ) being 0, highlighting that one cannot just connect and importance weight matching nextstates ¯ s j 2 of the first-stage transitions with matching start-states s k 2 of the second-stage transitions. In contrast, if we assume the data is such that ¯ s j 2 = s j 2 , this problem is immediately avoided as samples from P ( s 1 , a ) , along with rewards from those states are directly handed to the learner. The learner can then simply use all of the rewards r j 2 from tuples that contain the action π ( s j 2 ) (which we have on average at least 1 /C conc of, due to concentrability) to estimate the value of policy π after taking s 1 , a (solving the toy problem).

Now consider our algorithm if we do not have trajectory data. In this case we are no longer able to construct least squares targets of the form needed to make use of Lemma 4.2 (discussed in Section 4.2). This means that we would not be able to guarantee that our targets are linear, even under the true skipping mechanism ¯ G , implying that ¯ G might not be a feasible solution to our optimization problem. Then our optimism argument that the output of the optimization problem has a value estimate at least as large as the value estimate based on ¯ G would no longer hold, causing our whole proof strategy to break down.

C Proof of Lemma 4.2

Proof. We follow a proof technique introduced in [Weisz et al., 2023]. We start by quoting their definition of admissible functions and their admissible realizability lemma.

Definition 2 (Definition 4.6 in [Weisz et al., 2023]) . For any h ∈ [ H ] , f : S h → R is α ′ -admissible for some α ′ > 0 if for all s ∈ S h , | f ( s ) | ≤ range ( s ) /α ′ .

Lemma C.1 (Admissible-realizability (Lemma 4.7 in [Weisz et al., 2023])) . If f : S h → R is α ′ -admissible then it is realizable, that is, for all t ∈ [ h -1] and π ∈ Π , there exists some ˜ θ ∈ R d with ∥ ∥ ∥ ˜ θ ∥ ∥ ∥ 2 ≤ 4 d 0 L 2 /α ′ such that for all ( s, a ) ∈ S t ×A ,

∣ ∣ Next, we fix some f : S → [0 , H ] with f ( s ⊤ ) = 0 , and policy π ∈ Π . For 2 ≤ h ≤ H +1 , define ˜ g h : S h → [ -H,H ] as ˜ g H +1 ( · ) = 0 , and

Notice that for any h ∈ [ H ] , ( s, a ) ∈ S h ×A , the function of ( s, a ) we aim to linearly realize can be written as

The second term of the sum, q π ( s, a ) is linearly realizable by Assumption 1 with parameter ψ h ( π ) . The first term needs more work before Lemma C.1 can be applied. To this end, for h ∈ [2 : H ] we define g h : S h → R as

Notice that ˜ g h can be decomposed into a sum of g t functions as for all h ∈ [2 : H ] , s ∈ S h ,

The benefit of decomposing ˜ g h into g t functions is that g t are α ′ -admissible under Definition 2 for α ′ = α/ (2 H ) . To see this, note that for any trajectory and s , -R h : H + f ( s ) -˜ g h +1 ( S h +1 ) ∈ [ -2 H, 2 H ] . g t ( s ) multiplies this by 1 -ω ¯ G ( s ) which by Eq. (6) is between 0 and 1 , and satisfies 1 -ω ¯ G ( s ) = 0 if range ¯ G ( s ) ≤ α/ √ 2 d . By Lemma 4.1, range ( s ) ≤ √ 2 d · range ¯ G ( s ) , so g t ( s ) = 0 for any s with range ( s ) ≤ α . On any other s , | g t ( s ) | ≤ 2 H . Therefore g t is α ′ -admissible. This allows us to use Lemma C.1 to get that for any h ∈ [ H -1] , there exist ˜ θ h +1: H ∈ B (8 Hd 0 L 2 /α ) H -h , such that for any stage t ∈ [ h +1 : H ] , for all ( s, a ) ∈ S h ×A ,

∣ ∣ By combining this with Eq. (37), for all h ∈ [ H ] there exists ˜ θ = ∑ H t = h +1 ˜ θ t with ˜ θ ∈ B (8 H 2 d 0 L 2 /α ) such that for all ( s, a ) ∈ S h ×A ,

∣ ∣ Combined with Eq. (36) and the parameter ψ h ( π ) from Assumption 1, there exists θ = ˜ θ + ψ h ( π ) with θ ∈ B ( L 2 (8 H 2 d 0 /α +1)) such that for all ( s, a ) ∈ S h ×A ,

D Results used in Section 5

D.1 Bounding term ( I )

We begin by defining an alternative policy π † ¯ G as

This policy can only be worse in value than π ⋆ ¯ G :

Lemma D.1. For all s ∈ S ,

Proof. We prove by induction for h = H +1 , H, . . . , 1 that for all s ∈ S h , v π ⋆ ¯ G ( s ) ≥ v π † ¯ G ( s ) . The base case of h = H +1 is immediately true by definition as v -values are 0 on s ⊤ , regardless of the policy. Assuming the inductive hypothesis holds for h + 1 , we continue by proving it for h . Let ( s, a ) ∈ S h ×A be arbitrary. Notice that

where the inequality is due to the inductive hypothesis. Next, for any s ∈ S h , by the above and the definition of the policies,

finishing the induction.

Due to Lemma D.1, v ⋆ ( s 1 ) -v π ⋆ ¯ G ( s 1 ) ≤ v ⋆ ( s 1 ) -v π † ¯ G ( s 1 ) . We continue by bounding v ⋆ ( s 1 ) -v π † ¯ G ( s 1 ) . We first decompose it using the performance difference lemma (Lemma J.3) to get that

̸

where we only have the sum from h = 2 to H , since for h = 1 and h = H + 1 , for any s ∈ S h , ω ¯ G ( s ) = 0 and therefore π † ¯ G ( s ) = π ⋆ ( s ) . By the definition of ω ¯ G (Eq. (6)), we can see that for s ∈ S , ω ¯ G ( s ) = 0 only when range ¯ G ( s ) ≤ 2 α/ √ 2 d Thus, the policies π ⋆ ( s ) and π † ¯ G ( s ) are equal for all s ∈ S that satisfy range ¯ G ( s ) ≥ 2 α/ √ 2 d . Making use of this result in Eq. (38), we get that

By Eq. (3), we know that

Then, we can use Lemma 4.1 to get that for all h ∈ [2 : H ] and ( s, a ) ∈ S h ×A

Putting things together we get the following bound.

D.2 Bounding term ( II )

To bound v π ⋆ ¯ G ( s 1 ) -¯ v θ ′ 1 ( s 1 ) we decompose it into the following error terms

where the inequality holds since we have approximate linear q π -realizability (Assumption 1), which implies that

To help us bound ¯ v ψ 1 ( π ⋆ ¯ G ) ( s 1 ) -¯ v θ ′ 1 ( s 1 ) in Eq. (39) we make use of two lemmas. The first is the following (proof in Appendix E).

Lemma D.2. There is an event E 1 , which occurs with probability at least 1 -δ/ 3 , such that under E 1 , for all G ∈ G and h ∈ [ H +1] , it holds that ψ h ( π ⋆ G ) ∈ Θ G,h .

Lemma D.2 tells us that under event E 1 , ψ 1 ( π ⋆ ¯ G ) ∈ Θ ¯ G, 1 . The second lemma is the following (proof in Appendix G).

Lemma D.3 (Feasibility) . There is an event E 1 ∩ E 2 ∩ E 3 , which occurs with probability at least 1 -δ , such that under event E 1 ∩ E 2 ∩ E 3 , the true guess ¯ G is a feasible solution to Optimization Problem 1.

Notice that since ( G ′ , θ ′ 1: H +1 ) is the solution to Optimization Problem 1, it holds that ¯ v θ ′ 1 ( s 1 ) ≥ ¯ v θ ( s 1 ) for any θ ∈ Θ G, 1 where G is a feasible solution to Optimization Problem 1. Thus, we get that under event E 1 ∩ E 2 ∩ E 3 , since ψ 1 ( π ⋆ ¯ G ) ∈ Θ ¯ G, 1 (by Lemma D.3), and ¯ G is a feasible solution to Optimization Problem 1 (by Lemma D.3), it holds that

which together with Eq. (39), implies that

D.3 Proof of Lemma 5.2

Proof. Recall that ( G ′ , θ ′ 1: H +1 ) is the solution to Optimization Problem 1. We aim to show that under event E 1 ∩ E 2 , for any h ∈ [2 : H +1] , it holds that

To prove Eq. (40) we will use induction. The base case is when h = H +1 , which trivially holds, since v π ( s ⊤ ) = r H +1 ( s, a ) = 0 for all π ∈ Π , s ⊤ ∈ S H +1 , a ∈ A .

Now, we show the inductive step. Let h ∈ [2 : H ] be arbitrary. Assume that Eq. (40) holds for any t ∈ [ h +1 : H +1] . We prove that Eq. (40) also holds for h . For any ( s, a ) ∈ ( S\S 1 ) ×A , let

̸

Then

where the inequality holds since we have approximate linear q π -realizability (Assumption 1). To bound the first and third error terms above notice that under event E 1 , by Lemma D.2, we know that ψ h ( π ⋆ G ′ ) ∈ Θ G ′ ,h . We also know that θ ′ h ∈ Θ G ′ ,h , by definition. Let ˜ ν u ( s u , a u ) = P u ˜ π h ,s 1 ( s u , a u ) and ν ′ u ( s u , a u ) = P u π ′ ,s 1 ( s u , a u ) for all u ∈ [ H ] , ( s u , a u ) ∈ S u ×A . Clearly, ˜ ν = (˜ ν u ) u ∈ [ H ] and ν ′ = ( ν ′ u ) u ∈ [ H ] are admissible distributions, by Definition 1. Notice that we have satisfied all the conditions to make use of Lemma 5.1. Thus, under event E 1 ∩ E 2 it holds that and

The term E Traj ∼ P ˜ π h ,s 1 ¯ q θ ′ h ( S h , A h ) -E Traj ∼ P π ′ ,s 1 ¯ q θ ′ h ( S h , A h ) in Eq. (41) can be bounded by recalling the definition of π ′ ( s ) (line 3), to get that

Under event E 1 ∩ E 2 , after plugging the above bounds into Eq. (41), we have that

where the last inequality holds by the inductive hypothesis for h +1 (Eq. (40)), completing the proof of the claim.

E Proof of Lemma D.2

Proof. We will prove the claim using induction. The base case is when h = H + 1 , for which ψ H +1 ( π ) = ⃗ 0 for all π ∈ Π by definition. Thus, for all G ∈ G , it holds that ψ H +1 ( π ⋆ G ) ∈ Θ G,H +1 = { ⃗ 0 } .

Now, we show the inductive step. Let h ∈ [ H ] be arbitrary. Assume Lemma D.2 holds for any t ∈ [ h +1 : H +1] . We prove that it also holds for h . Define

By the inductive hypothesis we know that for any G ∈ G

We would like to make use of Lemma H.1 to bound ∥ ∥ ∥ ˆ ψ h ( π ⋆ G ) -ψ h ( π ⋆ G ) ∥ ∥ ∥ X h . To do so, we map the terms used in the Lemma H.1 to our terms as follows

Thus, ˆ ψ h ( π ⋆ G ) ∈ ˆ Θ G,h . It is left to show that ∥ ∥ ∥ ˆ ψ h ( π ⋆ G ) -ψ h ( π ⋆ G ) ∥ ∥ ∥ X h ≤ β , which together with the fact that ψ h ( π ⋆ G ) ∈ B ( L 2 ) , implies the desired result, ψ h ( π ⋆ G ) ∈ Θ G,h .

With the definitions as above, applying Lemma H.1 we get

The first term in Eq. (42) can be bounded by H 3 / 2 d by recalling that √ λ = H 3 / 2 d/ ˜ L 2 (Eq. (25)) and ∥ ψ h ( π ⋆ G ) ∥ 2 ≤ L 2 ≤ ˜ L 2 .

The ∥ ∆ ∥ ∞ in the second term in Eq. (42) can be bounded by first decomposing the error, and using a triangle inequality as follows.

For the first term in Eq. (43), notice that

where the last inequality holds since we have approximate linear q π -realizability (Assumption 1). To bound the second term in Eq. (43) notice that by the definition of π ⋆ G (Eq. (15)), arg max a ′ ∈A q π ⋆ G ( S τ , a ′ ) is exactly the action π ⋆ G would take at the stopping stage τ , and that the distribution of Traj under policy π 0 until stopping stage τ is same as the distribution of Traj under policy π ⋆ G until stopping stage τ . This implies that

∥ ∥ where the last inequality holds since we have approximate linear q π -realizability (Assumption 1). Plugging the above bounds into Eq. (43), we get that ∥ ∆ ∥ ∞ ≤ 2 η .

To bound the third term in Eq. (42), let the event E 1 be as defined in the proof of Lemma H.2, which occurs with probability at least 1 -δ/ 3 . Then, under event E 1 , the third term in Eq. (42) can be bounded by ¯ β (Eq. (31)), by applying Lemma H.2 since ψ h +1: H +1 ( π ⋆ G ) ∈ B ( L 2 ) H -h +1 ⊂ B ( ˜ L 2 ) H -h +1 .

Plugging the three bounds above back into Eq. (42) we get that, under event E 1 , it holds that where β is defined in Eq. (32), and the last inequality can be seen to hold by plugging in parameter values according to Appendix A. Thus, under event E 1 , which occurs with probability at least 1 -δ/ 3 , for any G ∈ G and h ∈ [ H ] it holds that ψ h ( π ⋆ G ) ∈ Θ G,h , which completes the proof.

F Proof of Lemma 5.1

Proof. Let G ∈ G be a feasible solution to Optimization Problem 1. By Lemma I.1, there is an event E 2 , that occurs with probability at least 1 -δ/ 3 , such that under event E 2 , for all G ∈ G , and for all h ∈ [ H ] , it holds that where | C G ξ | , L ξ , are defined in Eqs. (33) and (34). Let h ∈ [ H ] . Recall that since G is a feasible solution to Optimization Problem 1 we know that Eq. (14) passed for h . Thus,

Combining the above two results, we have that under event E 2 , for any h ∈ [ H ] , it holds that

Now we will relate the data collecting distribution µ to any admissible distribution ν . Notice that by the definition of max and min , for any ( s, a ) ∈ S × A , it holds that

Thus, we can apply Lemma G.4 to get that under event E 2 , for all h ∈ [ H ] , and admissible distribution ν = ( ν t ) t ∈ [ H ] , it holds that

To conclude, we have that under event E 2 , for any G ∈ G that is a feasible solution to Optimization Problem 1, for any h ∈ [ H ] , for any ( θ s,a ) ( s,a ) ∈S h ×A and ( ˇ θ s,a ) ( s,a ) ∈S h ×A with θ s,a , ˇ θ s,a ∈ Θ G,h for all ( s, a ) ∈ S h ×A , and for any admissible distribution ν = ( ν t ) t ∈ [ H ] , it holds that

The third inequality holds by plugging in the values of | C G ξ | , L ξ , as defined in Eqs. (33) and (34). The last inequality holds by setting ξ -1 = 24 √ 2 d √ nH 2 L 1 α -1 ( 2 √ nL 1 ˜ L 2 / ( H 3 / 2 d ) ) H . The last two equalities hold by plugging in parameter values according to Appendix A.

G Proof of Lemma D.3

Proof. To show that ¯ G is a feasible solution we need to show that Eq. (14) is satisfied for all h ∈ [ H ] . By Lemma I.1, there is an event E 2 , that occurs with probability at least 1 -δ/ 3 , such that under event E 2 , for all h ∈ [ H ] , it holds that where | C G ξ | , L ξ , are defined in Eqs. (33) and (34). Let h ∈ [ H ] be arbitrary for the remainder of the proof. We focus on bounding E ( S,A ) ∼ µ h [ max θ ∈ Θ ¯ G,h ¯ q θ ( S, A ) -min θ ∈ Θ ¯ G,h ¯ q θ ( S, A ) ] for the remainder of the proof. The following lemma will be helpful (proof in Appendix G.1).

Lemma G.1. There is an event E 1 , that occurs with probability at least 1 -δ/ 3 , such that under event E 1 , for any h ∈ [ H ] , ( s, a ) ∈ S h ×A , θ h ∈ Θ ¯ G,h , it holds that

where, for any ( s ′ , a ′ ) ∈ S × A , and g ¯ π is a state-action value function of policy ¯ π (similar to q ¯ π ), except in the alternative MDP that has the same state and action spaces, and transition distributions as the original MDP under consideration, but with a reward function modified as follows. For all ( s ′ , a ′ ) ∈ S × A , the reward in this alternative MDP is deterministically min { 1 , ∥ ϕ ( s ′ , a ′ ) ∥ X -1 h } ( i.e. R ( s ′ , a ′ ) = /x31 { min { 1 , ∥ ϕ ( s ′ , a ′ ) ∥ X -1 h }}) . In particular for any h ′ ∈ [ H ] , ( s ′ , a ′ ) ∈ S h ′ ×A

The recursive definition of ¯ π can be interpreted in the same way as described below Eq. (15) .

Let ¯ π be as defined in Lemma G.1. Then, by Lemma G.1, under event E 1 , it holds that

The inequality used Lemma G.1. As we will show shortly, the first term can be bounded by Lemma G.3, since its expectation is taken w.r.t. the data collecting distribution µ . Thus, the approach we take to bounding the second term is to relate its nested expectations to just be a single

expectation taken w.r.t. the distribution µ (similar to the first term, which we claim we know how to bound). To this end, for any ( s ′ , a ′ ) ∈ S × A , define the policy ˇ π h as

Notice that for any t ∈ [ H ] , u ∈ [ t +1 : H +1] , E ( S t ,A t ) ∼ µ t E ( S u ,A u ) ∼ P u ¯ π,S t ,A t = E ( S u ,A u ) ∼ P u ˇ πt ,s . Let ˇ ν u ( s u , a u ) = P u ˇ π h ,s 1 ( s u , a u ) for all u ∈ [ H ] , ( s u , a u ) ∈ S u ×A . Clearly, ˇ ν = (ˇ ν u ) u ∈ [ H ] is an admissible distribution by Definition 1. Thus, with the definition of ˇ ν and using Lemma G.4, we get that Eq. (49) is

The last inequality used that C conc ≥ 1 . Finally, we can apply Lemma G.3 to bound E ( S t ,A t ) ∼ µ t min { 1 , ∥ ϕ ( S t , A t ) ∥ X -1 t } . In particular, let E 3 be as defined in the proof of Lemma G.3. Then, by Lemma G.3, under event E 3 , it holds that

Putting all of the bound after Eq. (49) together and plugging them into Eq. (49), we have that, under event E 1 ∩ E 2 , it holds that

We are now ready to state the final result. Noting that h ∈ [ H ] was arbitrary, by combining Eqs. (45) and (50), we have that, under event E 1 ∩ E 2 ∩ E 3 , for all h ∈ [ H ] , it holds that

The first equality holds by plugging in the values of | C G ξ | , L ξ , as defined in Eqs. (33) and (34). The second equality holds by setting ξ -1 = 24 √ n √ 2 dH 2 L 1 α -1 ( 2 √ nL 1 ˜ L 2 / ( H 3 / 2 d ) ) H . The last two equalities hold by plugging in parameter values according to Appendix A.

Noticing that this is exactly the condition (Eq. (14)) in Optimization Problem 1 that needs to be satisfied by any feasible solution, we conclude that, under event E 1 ∩ E 2 ∩ E 3 , the true guess ¯ G is a feasible solution to Optimization Problem 1.

G.1 Proof of Lemma G.1

Proof. To prove Eq. (46) we will use induction. The base case is when h = H + 1 , for which ¯ v θ H +1 ( s ) = v π ⋆ ¯ G ( s ) for all s ∈ S H +1 . This holds, since for all ( s, a ) ∈ S H +1 ×A , ¯ q θ H +1 ( s, a ) = 0 for all θ H +1 ∈ Θ ¯ G,H +1 , by the definition of Θ ¯ G,H +1 (Eq. (13)), and v π ⋆ ¯ G ( s ) = 0 , by Eq. (1), since S H +1 = { s ⊤ } .

Now, we show the inductive step. Let h ∈ [ H ] be arbitrary. Assume that Eq. (46) holds for any t ∈ [ h +1 : H +1] . We prove that Eq. (46) holds for h . Let ( s, a ) ∈ S h ×A , θ h ∈ Θ ¯ G,h . Then,

∣ ∣ where the last equality holds since ( ⟨ ϕ ( s, a ) , θ h ⟩ -q π ⋆ ¯ G ( s, a ) ) ∈ [ -H,H ] . We will focus on bounding ∣ ∣ ⟨ ϕ ( s, a ) , θ h ⟩ -q π ⋆ ¯ G ( s, a ) ∣ ∣ . By the definition of the set Θ ¯ G,h (Eq. (13)) we know that there exists a ˆ θ h ∈ ˆ Θ ¯ G,h such that ∥ ∥ ∥ θ h -ˆ θ h ∥ ∥ ∥ X h ≤ β . Thus, by the Cauchy-Schwarz inequality, we have that

This implies that

∣ ∣ ∣ ∣ Since we know ˆ θ h ∈ ˆ Θ ¯ G,h , by the definition of ˆ Θ ¯ G,h (Eq. (12)), there exists a θ h +1: H +1 ∈ Θ ¯ G,h +1 × · · · × Θ ¯ G,H +1 , such that

By Lemma 4.2, we know there exists a parameter ρ π 0 h ( f ) ∈ B ( ˜ L 2 ) , such that for all ( s, a ) ∈ S h ×A ,

∣ ∣ Let ρ π 0 h be as defined above. Next, we will show a bound on ∣ ∣ ∣ 〈 ϕ ( s, a ) , ˆ θ h -ρ π 0 h ( ¯ v θ h +1: H +1 ) 〉∣ ∣ ∣ and ∣ ∣ ∣ 〈 ϕ ( s, a ) , ρ π 0 h ( ¯ v θ h +1: H +1 ) 〉 -q π ⋆ ¯ G ( s, a ) ∣ ∣ ∣ , which together will give us a bound on ∣ ∣ ∣ 〈 ϕ ( s, a ) , ˆ θ h 〉 -q π ⋆ ¯ G ( s, a ) ∣ ∣ ∣ , as desired. The following result gives us a bound on ∥ ∥ ˆ θ h -ρ π 0 h ( ¯ v θ h +1: H +1 ) ∥ ∥ X h .

∥ ∥ Lemma G.2. There is an event E 1 , which occurs with probability at least 1 -δ/ 3 , such that under event E 1 , for all G ∈ G for all h ∈ [ H ] , and for all ˆ θ h ∈ ˆ Θ G,h , it holds that where

and θ ⋆ h ∈ B ( ˜ L 2 ) is such that, for all ( s, a ) ∈ S h ×A , it satisfies

Proof. Fix G ∈ G , h ∈ [ H ] , and ˆ θ h ∈ ˆ Θ G,h , such that

Fix θ ⋆ h ∈ B ( ˜ L 2 ) , such that, for all ( s, a ) ∈ S h ×A , it satisfies

We would like to make use of Lemma H.1 to bound ∥ ∥ ∥ ˆ θ h -θ ⋆ h ∥ ∥ ∥ X h . To do so, we map the terms used in the Lemma H.1 to our terms as follows

With the definitions as above, applying Lemma H.1 we get

The first term can be bounded by H 3 / 2 d , by noting that ∥ θ ⋆ h ∥ 2 ≤ ˜ L 2 , and setting

The second term can be bounded by ˜ η √ n , by using Eq. (55).

To bound the third term let the event E 1 be as defined in the proof of Lemma H.2, which occurs with probability at least 1 -δ/ 3 . Then, under event E 1 , the third term in Eq. (56) can be bounded by ¯ β (Eq. (31)), by applying Lemma H.2, since θ h +1: H +1 ∈ Θ G,h +1 ×··· × Θ G,H +1 ⊂ B ( ˜ L 2 ) H -h +1 . Plugging the three bounds above back into Eq. (56) we get that under event E 1 , it holds that

The last equality holds by setting

and the values of n, ¯ β are set according to Appendix A.

We return back to the proof of Lemma G.1. Let E 1 be as defined in the proof of Lemma G.2. For the remainder of the proof, operate under event E 1 . By Lemma G.2 (with ρ π 0 h ( ¯ v θ h +1: H +1 ) = θ ⋆ h ), we have that

The first inequality used the Cauchy-Schwarz inequality. The second inequality used Eq. (60). Now,

∣ ∣ The equality used the definition of ¯ v (Eq. (10)). To bound the first term in Eq. (62) notice that by the definition of π ⋆ ¯ G (Eq. (15)), arg max a ′ ∈A q π ⋆ ¯ G ( S τ , a ′ ) is exactly the action π ⋆ ¯ G would take at the stopping stage τ , and that the distribution of Traj under policy π 0 until stopping stage τ is same as the distribution of Traj under policy π ⋆ ¯ G until stopping stage τ . This gives that

∣ ∣ To bound the second term in Eq. (62) we can use the inductive hypothesis (Eq. (46)). Defining notation that will be needed for the below display, for any ( s ′ , a ′ ) ∈ S × A we will write Traj ′ ∼ P ˇ π,s,a to have the usual definition Traj ′ = ( s ′ , a ′ , R ′ h , . . . , S ′ H +1 , A ′ H +1 , R ′ H +1 ) , except with a superscript ( · ) ′ added to all of the random elements. Then, by letting a τ = arg max a ′ ∈A ( ¯ q θ stage( Sτ ) ( S τ , a ′ ) -q π ⋆ ¯ G ( S τ , a ′ ) ) , applying the inductive hypothesis, and a triangle inequality, we have that

∣ ∣ ∣ We bound each of the terms in the expectation separately. For the first term, we first recall the definition of g ¯ π (Eq. (48)) and upper bound the term inside the expectation as follows, which will help us relate things to ¯ π as we shall see soon.

Along with the above result, notice that by the definition of ¯ π (Eq. (47)), arg max a ′ ∈A g ¯ π ( S τ , a ′ ) is exactly the action ¯ π would take at the stopping stage τ , and that the distribution of Traj under policy π 0 until stopping stage τ is same as the distribution of Traj under policy ¯ π until stopping stage τ . Thus,

For the second term, since τ ≥ h +1 where τ ∼ F ¯ G, Traj ,h +1 , it holds that

Plugging the above two bounds into Eq. (62), we get that

Combining Eqs. (52), (53), (61) and (63), we get that

The second inequality used that β ≥ H and that min( a, b + c ) ≤ min( a, b ) + c for a, b, c ≥ 0 .

Lemma G.3. There is an event E 3 , that occurs with probability at least 1 -δ/ 3 , such that under event E 3 , for all h ∈ [ H ] , it holds that

where ˇ ϵ is defined in Eq. (28) .

Proof. First we will show two useful results (namely Eq. (64) and Eq. (66)) that are needed in the proof. Let

Notice that any X ∈ X can be written as X = ∑ d i =1 x i x ⊤ i with x i ∈ B (1 /λ ) for all i ∈ [ d ] . By Lemma J.2, we know there exists a set C ξ ⊂ B ( a ) , a, ξ > 0 with | C ξ | = (1 + 2 a/ξ ) d such that for any x ∈ B ( a ) there exists a y ∈ C ξ such that ∥ x -y ∥ 2 ≤ ξ . Define the set

with | Y | = (1+2 / ( λξ )) d 2 . Then, for any X = ∑ d i =1 x i x ⊤ i ∈ X there exists a Y = ∑ d i =1 y i y ⊤ i ∈ Y , such that ∥ x i -y i ∥ 2 ≤ ξ for all i ∈ [ d ] . Let X,Y be as we just defined them. Then, writing ∥·∥ op

for the operator norm,

Then, for any u ∈ B ( L 1 )

which implies that (since for non-negative a, b, √ a + b ≤ √ a + √ b )

Thus, for any u ∈ B ( L 1 )

Eq. (64) is the first useful result that we alluded to at the beginning of the proof.

Now, we will show the second useful result. For any Y ∈ Y and h ∈ [ H ] , define the event

Since min { 1 , ∥ u ∥ Y } ∈ [0 , 1] for all u ∈ B ( L 1 ) , Y ∈ Y , we can use Hoeffding's inequality (Lemma J.1) to get that, for any Y ∈ Y , h ∈ [ H ] , event E Y,h 3 occurs with probability at least 1 -δ/ (3 H | Y | ) . Let

Then, by applying a union bound over Y, h we have that the event E 3 occurs with probability at least 1 -δ/ 3 , and under event E 3 , for all Y ∈ Y , h ∈ [ H ] , it holds that

Eq. (66) is the second useful result that we alluded to at the beginning of the proof.

Now, we turn to proving Lemma G.3. Let h ∈ [ H ] . Let X ∈ X , and select Y ∈ Y such that, for any u ∈ B ( L 1 )

which we know exists by Eq. (64). By using Eq. (67) we get that

To bound the first term on the RHS in Eq. (68) we can use Eq. (66), to get that under event E 3

Then, by Jensen's inequality we have that

Putting Eqs. (68) to (71) together and noting that X,h were arbitrary, we get that, under event E 3 , for any X ∈ X , h ∈ [ H ] , it holds that

We can now introduce X h and make use of the above result. Notice that for any h ∈ [ H ] , X -1 h = ( λI + ∑ j ∈ [ n ] ϕ j h ( ϕ j h ) ⊤ ) -1 is such that λ max ( X -1 h ) ≤ 1 /λ , since λ min ( X h ) ≥ λ . Thus, X -1 h ∈ X . For any t ∈ [ n ] , h ∈ [ H ] , define X t,h = λI + ∑ j ∈ [ t ] ϕ j h ( ϕ j h ) ⊤ , and notice that X -1 n,h = X -1 h , and that X -1 t,h -X -1 n,h is positive semidefinite. This implies that, for all h ∈ [ H ]

Now, we can use the elliptical potential lemma (Lemma J.4), to conclude that, for all h ∈ [ H ]

Putting everything together, we get that, under event E 3 , for all h ∈ [ H ] , it holds that

The second inequality used that | Y | = (1 + 2 / ( λξ )) d 2 . The first equality holds by setting ξ -1 = 8 ˜ L 2 L 2 1 n . The last equality holds by plugging in parameter values according to Appendix A.

Lemma G.4. If Assumption 3 holds, then for any non-negative function f : S × A → [0 , ∞ ) , for any admissible distribution ν = ( ν t ) t ∈ [ H ] , and for any h ∈ [ H ] , it holds that

Proof. Let f : S × A → [0 , ∞ ) be any non-negative function. Let h ∈ [ H ] be any stage. Then,

where the inequality holds by applying Assumption 3, and noting that f is non-negative. This implies the desired result, since f and h were arbitrary.

H Lemmas Related to Least-squares

Lemma H.1 (Least-squares Error Decomposition) . Let λ > 0 , θ ⋆ ∈ R d and n ∈ N + , For all k ∈ [ n ] , let

Then,

Proof. We begin by decomposing the targets used in ˆ θ as follows

Then, subtracting θ ⋆ from both sides and taking the matrix V weighted norm of both sides gives us that where the second inequality used that ∥ ∥ V -1 ∥ ∥ ≤ λ max ( V -1 ) = 1 /λ min ( V ) and Lemma J.5 to bound ∥ ∑ n t =1 A t ∆ t ∥ V -1 .

Lemma H.2 (Least-squares Noise Bound) . There is an event E 1 , which occurs with probability at least 1 -δ/ 3 , such that under event E 1 , for all h ∈ [ H ] , for all G ∈ G , and θ h +1: H +1 ∈ B ( ˜ L 2 ) H -h +1 , it holds that

Proof. We begin the proof by showing two useful results (namely Eq. (74) and Eq. (75)), which will be needed later in the proof.

By Lemma J.2, we know there exists a set C ξ ⊂ B ( a ) , a, ξ > 0 with | C ξ | = (1 + 2 a/ξ ) d such that for any x ∈ B ( a ) there exists a y ∈ C ξ such that ∥ x -y ∥ 2 ≤ ξ . Define the set C Θ ξ =

× h ∈ [2: H +1] C ξ ⊂ B ( ˜ L 2 ) H with | C Θ ξ | = (1 + 2 ˜ L 2 /ξ )) dH . Then, for any θ 2: H +1 ∈ B ( ˜ L 2 ) H , there exists a ˜ θ 2: H +1 ∈ C Θ ξ such that

Which implies that for any h ∈ [ H ] , t ∈ [ h +1 : H +1] , s ∈ S t and θ h +1: H +1 , θ ∼ h +1: H +1 as defined above

Let ξ > 0 . Combining Eq. (73) with Lemma I.4, we get that there exists a set C G ξ × C Θ ξ ⊂ G × B ( ˜ L 2 ) H with | C G ξ × C Θ ξ | ≤ (1+2 L 2 ˜ L 2 /ξ )) dH ( d 0 +1) such that, for any ( G,θ 2: H +1 ) ∈ G ×B ( ˜ L 2 ) H , there exists a ( ˜ G,θ ∼ 2: H +1 ) ∈ C G ξ × C Θ ξ such that, for any h ∈ [ H ] , for any u ∈ [ h ] and trajectory traj = ( s t , a t , r t ) t ∈ [ u,H +1] it holds that

Eq. (74) is the first useful result we alluded to at the beginning of the proof.

Now, we show the second useful result, which is a bound under a high probability event. For any ( ˜ G,θ ∼ 2: H +1 ) ∈ C G ξ × C Θ ξ and h ∈ [ H ] define the event

Notice that ι ˜ G,θ ∼ ,h is H -subgaussian. Thus, we can use Theorem 1 from [Abbasi-Yadkori et al., 2011] to get that the event E ˜ G,θ ∼ ,h 1 occurs with probability at least 1 -δ/ (3 H | C G ξ × C Θ ξ | ) . Define

Then, by applying a union bound over ˜ G,θ ∼ , h we have that the event E 1 occurs with probability at least 1 -δ/ 3 , and under event E 1 , for all ( ˜ G,θ ∼ ) ∈ C G ξ × C Θ ξ , h ∈ [ H ] , it holds that

Now with all of the above results in hand, we turn to finally proving Lemma H.2. Let G ∈ G as in the lemma statement. Then, by Eq. (74) we know that there exists a ( ˜ G,θ ∼ 2: H +1 ) ∈ C G ξ × C Θ ξ such that, for any h ∈ [ H ] , for any u ∈ [ h ] and trajectory traj = ( s t , a t , r t ) t ∈ [ u : H +1] , it holds that

Let ˜ G,θ ∼ 2: H +1 be as defined above. Then, for any h ∈ [ H ] , by using the triangle inequality we can write

The first term can be bounded by Eq. (75), if we are under event E 1 . For the second and third term we make use of Lemma J.5, which ensures that for any sequence ( b j ) j ∈ [ n ] such that | b j | ≤ c ∈ R the following holds

For the second term and third term the respective b j terms can be bounded by using Eq. (76), giving us c = 7 √ 2 dH 2 L 1 ξ/α .

Putting the above three bounds together we have that under event E 1 , which occurs with probability at least 1 -δ/ 3 , for all h ∈ [ H ] , for all G ∈ G , and θ h +1: H +1 ∈ B ( ˜ L 2 ) H -h +1 , it holds that where the last inequality used the Determinant-Trace Inequality (see Lemma 10 in [Abbasi-Yadkori et al., 2011]). Setting ξ -1 = 14 √ n √ 2 dH 2 L 1 α -1 , we get that the above display is

I Lemmas Related to Covering G

Lemma I.1. There is an event E 2 , that occurs with probability at least 1 -δ/ 3 , such that under event E 2 , for all G ∈ G , and for all h ∈ [ H ] , it holds that where | C G ξ | , L ξ , are defined in Eqs. (33) and (34) .

Proof. Let G ∈ G be a feasible solution to Optimization Problem 1. By the first result in Lemma I.2, there exists a set C G ξ ⊂ G such that, there exists a ˜ G ∈ C G ξ such that, for any h ∈ [ H ] , ( s, a ) ∈ S h ×A , it holds that

To bound the second term in the absolute value of Eq. (79) to its empirical mean, we can use the second result in Lemma I.2, which gives us that under event E 2

We can relate the ˜ G in the second term of the absolute value in Eq. (80) back to G , by once again using Eq. (78), to get that

Putting together Eqs. (79) to (81), and noting that h was arbitrary, gives the desired result.

Lemma I.2. Let ξ > 0 . There exists a set C G ξ ⊂ G such that, for any G ∈ G , there exists a ˜ G ∈ C G ξ such that, for any h ∈ [ H ] , ( s, a ) ∈ S h ×A , it holds that

∣ ∣ where | C G ξ | , L ξ , are defined in Eqs. (33) and (34) . Furthermore, there is an event E 2 , which occurs with probability at least 1 -δ/ 3 , such that under event E 2 , for any ˜ G ∈ C G ξ , and h ∈ [ H ] , it holds that

Proof. Let ξ > 0 , κ t ≥ 0 , ∀ t ∈ [2 : H + 1] . By Lemma I.4, there exists a set C G ξ ⊂ G with | C G ξ | ≤ (1 + 2 L 2 /ξ )) dHd 0 such that, for any G ∈ G , there exists a ˜ G ∈ C G ξ such that, for any h ∈ [ H ] , for any θ h +1: H +1 , θ ∼ h +1: H +1 ∈ B ( ˜ L 2 ) H -h +1 , such that for all t ∈ [ h +1 : H +1] , s ∈ S t , ∣ ∣ ¯ v θ t ( s ) -¯ v θ ∼ t ( s ) ∣ ∣ ≤ κ t , and for any j ∈ [ n ] , it holds that

For the remainder of the proof let G, ˜ G be as described above.

We first show the following intermediate result.

Lemma I.3. For all h ∈ [ H ] , it holds that:

where,

Proof. Proof of result 1 . : To show Eq. (83) we will use induction. The base case is when h = H , for which

Thus, for any ˆ θ H ∈ ˆ Θ G,H , θ H ∈ Θ G,H , select ˆ θ ∼ H = ˆ θ H ∈ ˆ Θ ˜ G,H , θ ∼ H = θ H ∈ Θ ˜ G,H . Then

Now, we show the inductive step. Let h ∈ [ H -1] be arbitrary. Assume Eq. (83) holds for any t ∈ [ h +1 : H ] . We prove that Eq. (83) also holds for h . Let ˆ θ h ∈ ˆ Θ G,h be arbitrary. Notice that ˆ θ h must have the following form.

which exists by the inductive hypothesis (Eq. (83)) and since Θ G,H +1 = Θ ˜ G,H +1 = { ⃗ 0 } . Define

Recall that we aim to bound ∥ ∥ ∥ ˆ θ h -ˆ θ ∼ h ∥ ∥ ∥ X h . Plugging in the expressions for ˆ θ h , ˆ θ ∼ h , as defined above, we get that where in the second equality we let b j = E τ ∼ F j G,h +1 [ r j h : τ -1 + ¯ v θ h +1: H +1 ( s j τ ) ] -E τ ∼ F j ˜ G,h +1 [ r j h : τ -1 + ¯ v θ ∼ h +1: H +1 ( s j τ ) ] . To bound the above term we can make use of Lemma J.5, which ensures that for any sequence ( b j ) j ∈ [ n ] such that | b j | ≤ a ∈ R the following holds

Thus, we are left to bound | b j | . To do so, we can make use of Eq. (82), which requires us to bound ∣ ∣ ¯ v θ t ( s ) -¯ v θ ∼ t ( s ) ∣ for all t ∈ [ h +1 : H ] , s ∈ S t , which can be done as follows.

The second inequality uses the Cauchy-Schwarz inequality. The third inequality used Eq. (22), that λ max ( X -1 h ) ≤ 1 /λ (by definition of X h Eq. (11)), and Eq. (86). The last equality used that √ λ = H 3 / 2 d/ ˜ L 2 (Eq. (25)). Plugging Eq. (87) into Eq. (82) we get that for any j ∈ [ n ]

Thus,

To see why the second last equality is true, let x = 6 √ n √ 2 dH 2 L 1 ξα -1 and y = √ nL 1 ˜ L 2 / ( H 3 / 2 d ) . Then, for any t ∈ [ h : H ] , c ξ t = x (1 + y ) H -t (by Eq. (85)) and

Notice that the sum can be rewritten as a finite geometric series.

Thus,

This proves the first result in Eq. (83).

Next, we show the second result in Eq. (83). Let θ h ∈ Θ G,h . By the definition of the set Θ G,h , there exists a ˆ θ h ∈ ˆ Θ G,h such that ∥ ∥ ∥ θ h -ˆ θ h ∥ ∥ ∥ X h ≤ β . Then, by the first result in Eq. (83) (which we have shown holds for h above), there exists a ˆ θ ∼ h ∈ ˆ Θ ˜ G,h , such that ∥ ∥ ∥ ˆ θ h -ˆ θ ∼ h ∥ ∥ ∥ X h ≤ c ξ h . Let ˆ θ h , ˆ θ ∼ h be as defined above, and select θ ∼ h = θ h -ˆ θ h + ˆ θ ∼ h , which is an element of Θ ˜ G,h since

Then,

This completes the proof of the second result in Eq. (83).

Proof of result 2 . : The proof is identical to that of the proof of result 1 . , except swapping the roles of ˆ θ h , θ h , G and ˆ θ ∼ h , θ ∼ h , ˜ G .

Now, with the results of Eqs. (83) and (84) in hand, we return to proving Lemma I.2. For any h ∈ [ H ] let θ h = arg max θ ∈ Θ G,h ⟨ ϕ ( s, a ) , θ ⟩ then, by Eq. (83), there exists a θ ∼ h ∈ Θ ˜ G,h such that

The second inequality used Eq. (22), that λ max ( X -1 h ) ≤ 1 /λ (by definition of X h Eq. (11)), and Eq. (86). The last equality used that √ λ = H 3 / 2 d/ ˜ L 2 (Eq. (25)). Now, for the other direction, using similar steps as above, for any h ∈ [ H ] , let θ ∼ h = arg max θ ∼ ∈ Θ ˜ G,h ⟨ ϕ ( s, a ) , θ ∼ ⟩ then, by Eq. (84), there exists a θ h ∈ Θ G,h such that ∥ θ h -θ ∼ h ∥ X h ≤ c ξ h . This gives that, for all h ∈ [ H ] , ( s, a ) ∈ S h ×A

The above two results together imply that, for all h ∈ [ H ] , ( s, a ) ∈ S h ×A

Following the same steps as above for min we can get that, for all h ∈ [ H ] , ( s, a ) ∈ S h ×A

The above two results together imply that, for all h ∈ [ H ] , ( s, a ) ∈ S h ×A ,

∣ ∣ ∣ ∣ ( max θ ∈ Θ G,h ¯ q θ ( s, a ) -min θ ∈ Θ G,h ¯ q θ ( s, a ) ) -( max θ ∼ ∈ Θ ˜ G,h ¯ q θ ∼ ( s, a ) -min θ ∼ ∈ Θ ˜ G,h ¯ q θ ∼ ( s, a ) )∣ ∣ ∣ ∣ = L ξ , where (by recalling Eq. (85)), L ξ = 12 √ 2 dH 2 L 1 ξα -1 ( 2 √ nL 1 ˜ L 2 / ( H 3 / 2 d ) ) H ≥ 2 L 1 ˜ L 2 c ξ h / ( H 3 / 2 d ) . (88) This concludes the proof of the first result in Lemma I.2.

Now we prove the second result in Lemma I.2. For any ˜ G ∈ C G ξ , h ∈ [ H ] , define the event

Then, since max θ ∼ ∈ Θ ˜ G,h ¯ q θ ∼ ( s, a ) -min θ ∼ ∈ Θ ˜ G,h ¯ q θ ∼ ( s, a ) ∈ [0 , H ] for all ( s, a ) ∈ S h × A , by Hoeffding's inequality (Lemma J.1), we have that, for any ˜ G ∈ C G ξ , h ∈ [ H ] , event E ˜ G,h 2 occurs with probability at least 1 -δ/ (3 H | C G ξ | ) . Let

Then, by applying a union bound over ˜ G,h we have that the event E 2 occurs with probability at least 1 -δ/ 3 .

LemmaI.4. Let ξ > 0 , κ t ≥ 0 , ∀ t ∈ [2 : H +1] . Then, there exists a set C G ξ ⊂ G with | C G ξ | ≤ (1+ 2 L 2 /ξ )) dHd 0 such that, for any G ∈ G , there exists a ˜ G ∈ C G ξ such that, for any h ∈ [ H ] , for any u ∈ [ h ] and trajectory traj = ( s t , a t , r t ) t ∈ [ u,H +1] , and for any θ h +1: H +1 , θ ∼ h +1: H +1 ∈ B ( ˜ L 2 ) H -h +1 that are close in predictions, that is, such that for all t ∈ [ h +1 : H +1] , s ∈ S t , ∣ ∣ ¯ v θ t ( s ) -¯ v θ ∼ t ( s ) ∣ ∣ ≤ κ t , it holds that

Proof. Recall that

G = { ( G h ) h ∈ [2: H ] = ( ϑ i h ) h ∈ [2: H ] ,i ∈ [ d 0 ] : for all h ∈ [2 : H ] , i ∈ [ d 0 ] , ϑ i h ∈ B ( L 2 ) } . By Lemma J.2, we know there exists a set C ξ ⊂ B ( a ) , a, ξ > 0 with | C ξ | = (1 + 2 a/ξ ) d such that for any x ∈ B ( a ) there exists a y ∈ C ξ such that ∥ x -y ∥ 2 ≤ ξ . Define the set C G ξ = × h ∈ [2: H ] ,i ∈ [ d 0 ] C ξ ⊂ G with | C G ξ | ≤ (1 + 2 L 2 /ξ )) dHd 0 . Then, for any G = ( ϑ i h ) h ∈ [2: H ] ,i ∈ [ d 0 ] ∈ G , there exists a ˜ G = ( ˜ ϑ i h ) h ∈ [2: H ] ,i ∈ [ d 0 ] ∈ C G ξ such that

Let G, ˜ G be as defined above. Then, for all s ∈ S \ S 1

and, since we have by definition (Eq. (6)) that ω G is a smooth function in terms of range G , we get that

For all h ∈ [ H ] , let θ h +1: H +1 , θ ∼ h +1: H +1 ∈ B ( ˜ L 2 ) H -h +1 , such that for all t ∈ [ h + 1 : H + 1] , s ∈ S t , ∣ ∣ ¯ v θ t ( s ) -¯ v θ ∼ t ( s ) ∣ ∣ ≤ κ t . Then, for any h ∈ [ H ] , u ∈ [ h ] , and trajectory traj = ( s t , a t , r t ) t ∈ [ u : H +1] , it holds that

Using similar steps to above it can be shown that

We claim that for any h ∈ [ H ] , u ∈ [ h ] , and trajectory traj = ( s t , a t , r t ) t ∈ [ u : H +1]

To show Eq. (93) we will use induction on h . The base case is when h = H , for which

where the first equality holds since Θ G,H +1 = Θ ˜ G,H +1 = { ⃗ 0 } (defined in Eq. (13)). The second equality holds since for any u ∈ [ H +1] , and trajectory traj = ( s t , a t , r t ) t ∈ [ u : H +1] , F G, traj ,H +1 ( τ = H +1) = F ˜ G, traj ,H +1 ( τ = H +1) = 1 .

Now, we show the inductive step. Let h ∈ [ H -1] be arbitrary. Assume Eq. (93) holds for any t ∈ [ h +1 , H ] . We prove that Eq. (93) also holds for h . Fix any u ∈ [ H +1] , and trajectory traj = ( s t , a t , r t ) t ∈ [ u : H +1] . To shorten notation let E G,θ,h = E τ ∼ F G, traj ,h +1 [ r h : τ -1 + ¯ v θ h +1: H +1 ( s τ ) ] (and similar for E ˜ G,θ ∼ ,h ). Then using the assumption that for all u ∈ [2 : H + 1] , s ∈ S u , ∣ ∣ ¯ v θ u ( s ) -¯ v θ ∼ u ( s ) ∣ ∣ ≤ κ u along with Eqs. (90) to (92) and noting that for any h ∈

where the second last inequality holds by the inductive hypothesis.

J Other Useful Results and Definitions

Definition 3. A finite set G ⊂ R d is the basis of a near-optimal design for a set Θ ⊆ R d , if there exists a probability distribution ρ over elements of G , such that for any θ ∈ Θ ,

where for some matrix X , X † denotes the Moore-Penrose inverse of some, and Ker( X ) its kernel (or null space).

LemmaJ.1 (Hoeffding's Inequality (Theorem 2 in [Hoeffding, 1994])) . Let ( X i ) i ∈ N be independent random variables such that X i ∈ [ a, b ] for some a, b ∈ R , and let S n = 1 n ∑ n i =1 X i . Then, with probability at least 1 -ζ it holds that

Lemma J.2 (Covering number of the Euclidean ball) . Let a > 0 , ϵ > 0 , d ≥ 1 , and B d ( a ) = { x ∈ R d : ∥ x ∥ 2 ≤ a } denote the d -dimensional Euclidean ball of radius a centered at the origin. The covering number of B d ( a ) is upper bounded by ( 1 + 2 a ϵ ) d .

Proof. Same as the proof of Corollary 4.2.13 in [Vershynin, 2018] with B (1) replaced with B ( a ) .

Lemma J.3 (Performance Difference Lemma (Lemma 3.2 in [Cai et al., 2020])) . For any policies π, ¯ π , it holds that

Lemma J.4 (Elliptical Potential Lemma (Lemma 19.4 in [Lattimore and Szepesvári, 2020])) . Let V 0 ∈ R d × d be positive definite and a 1 , . . . , a n ∈ R d be a sequence of vectors with ∥ a t ∥ 2 ≤ L ≤ ∞ for all t ∈ [ n ] , V t = V 0 + ∑ s ≤ t a s a ⊤ s . then,

Lemma J.5 (Projection Bound (Lemma 8 in [Zanette et al., 2020])) . Let ( a i ) i ∈ [ n ] be any sequence of vectors in R d and ( b i ) i in [ n ] be any sequence of scalars such that | b i | ≤ c . For any λ ≥ 0 and k ∈ N we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims explained and shown in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The result is Theorem 1, which lists all the assumptions it uses (which are presented in Section 3.1), and its proof is presented in Section 5.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our knowledge our research does not violate anything in the Code of Ethics upon our careful review.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed at the end of Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks since we do not use any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves that finite horizon offline RL under linear q^\\pi realizability assumption can be solved efficiently (in terms of sample complexity) if the data are trajectories collected by a policy with bounded concentrability coefficient (that is, the density ratio between the state-action distribution induced by any policy and the data distribution is upper bounded). The result highlights the effect of trajectory data by sharply contrasting with the existing impossibility result where the data can be arbitrary state-action distribution with bounded concentrability coefficient.\n\nSTRENGTHS:\nTo the best of my knowledge, this paper solves a long-standing open question of offline RL with linear q^\\pi realizability. The observation about trajectory data versus general offline data is solid and interesting.\n\nWEAKNESSES:\nI don’t think I completely follows the proof sketch shown in Section 4 & 5 given the complexity of the proofs, and some of the proof intuitions could be made more explicit instead of referring to some lemmas in the appendix. For example:\n\n-\tWhat is the reason that $G=\\bar{G}$ passes the condition (14)? Is it because the concentrability assumption plus Lemma 4.2 results in a tight confidence interval for the q-value?\n-\tOne thing I do not follow is how the algorithm eliminates incorrect guesses $G$. If I understand correctly, Lemma 4.2 only shows that linear q^\\pi-realizable MDPs with skips can be approximated by linear MDPs under the true guess. Then for an incorrect guess $G’$, is the modified MDPs still linear? If not, does optimization problem 1 still give some meaningful guarantees so that the algorithm can eliminate these cases? \nIn addition, since the optimization problem 1 is mostly built upon prior except for condition (14), if could be better to emphasize this new condition.\n\nMinor issue:\n\n-\tIn the statement of problem 1, n does not depend on \\delta.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an important theoretical result in offline reinforcement learning (RL) with linear function approximation. The authors show that under the assumptions of linear q-realizability, concentrability, and access to full trajectory data, it is possible to efficiently learn an ε-optimal policy with a sample complexity that scales polynomially in the relevant problem parameters (horizon H, feature dimension d, concentrability coefficient Cconc) and inversely in the desired accuracy ε. This is in contrast to previous negative results showing exponential sample complexity lower bounds when only having access to individual transitions.\n\nSTRENGTHS:\nThis is a strong theoretical contribution that significantly advances our understanding of the statistical complexity of offline RL under linear function approximation. The results are of high interest to the RL theory community.\n\nThe theoretical analysis is rigorous and the proofs seem correct. The assumptions are clearly stated and discussed. The authors also provide a thoughtful discussion of the limitations of their work, including the open question of computational efficiency and the restrictive nature of the linear q-realizability assumption.\n\nWEAKNESSES:\nIt would be beneficial if the authors could add the concrete definition of the previous non-trajectory data to make a direct comparison with full length trajectory data (Assumtpion 2). Also, the authors could add some comments on the hardness results without the trajectory data after the added definition.\n\nIn Assumption 2, the notation $\\phi(s_h^1, \\cdot)$ is not clearly defined. It may lead to the meaning of $\\phi(s_h^1, a)$ for all $a$.\n\nGiven the lengthy proof with complex notations and limited review time, the proof details are challenging to follow. Adding more discussion on algorithm design, including a pseudocode, and explaining the intuitions behind the proof would be helpful.\n\nOne potential limitation is the absence of experimental results, which means the practical relevance of the theoretical findings is not directly demonstrated. However, considering the focus on fundamental statistical limits, the lack of experiments is understandable.\n\nQUESTIONS:\nna\n\nLIMITATIONS:\nLimitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper considers the problem of learning the value ($Q$) function under q^{\pi} realizability and concentration assumption. The major contribution is to use trajectory data instead of independent samples to learn the target function, where negative results have been proven with independent samples.

STRENGTHS:
The problem is definite challenging, in particular given the negative results with independent samples.

WEAKNESSES:
The presentation is a problem of this work. The notation system is not reading friendly, and there is no algorithm block. I suggest to add an algorithm to make the input and output clear.

The high-level idea is not clear. The authors spend too much space on introducing their methods, and there is no explanation about the hard term in learning with independent terms. For example, why should we skip the states with small range(s)? It is hard for readers to verify the result through touching the high-level intuition.

Another concern is about the computational efficiency. Seems there is no evidence the optimization problem could be resolved efficiently.

QUESTIONS:
I do not have any further questions.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-08-02