Title: The Power of Resets in Online Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement learning

learning theory

generative model

simulator

coverability
Abstract: Simulators are a pervasive tool in reinforcement learning, but most existing algorithms cannot efficiently exploit simulator access -- particularly in high-dimensional domains that require general function approximation. We explore the power of simulators through online reinforcement learning with local simulator access (or, local planning), an RL protocol where the agent is allowed to reset to previously observed states and follow their dynamics during training. We use local simulator access to unlock new statistical guarantees that were previously out of reach:
- We show that MDPs with low coverability (Xie et al. 2023) -- a general structural condition that subsumes Block MDPs and Low-Rank MDPs -- can be learned in a sample-efficient fashion with only Q⋆-realizability (realizability of the optimal state-value function); existing online RL algorithms require significantly stronger representation conditions.
- As a consequence, we show that the notorious Exogenous Block MDP problem (Efroni et al. 2022) is tractable under local simulator access.
The results above are achieved through a computationally inefficient algorithm. We complement them with a more computationally efficient algorithm, RVFS (Recursive Value Function Search), which achieves provable sample complexity guarantees under a strengthened statistical assumption known as pushforward coverability. RVFS can be viewed as a principled, provable counterpart to a successful empirical paradigm that combines recursive search (e.g., MCTS) with value function approximation.

The Power of Resets in Online Reinforcement Learning

Abstract

Simulators are a pervasive tool in reinforcement learning, but most existing algorithms cannot efficiently exploit simulator access-particularly in high-dimensional domains that require general function approximation. We explore the power of simulators through online reinforcement learning with local simulator access (or, local planning), an RL protocol where the agent is allowed to reset to previously observed states and follow their dynamics during training. We use local simulator access to unlock new statistical guarantees that were previously out of reach:

We show that MDPs with low coverability [63]-a general structural condition that subsumes Block MDPs and Low-Rank MDPs-can be learned in a sample-efficient fashion with only Q ⋆ -realizability (realizability of the optimal state-value function); existing online RL algorithms require significantly stronger representation conditions.

As a consequence, we show that the notorious Exogenous Block MDP problem [22] is tractable under local simulator access.

The results above are achieved through a computationally-inefficient algorithm. We complement them with a more computationally efficient algorithm, RVFS ( Recursive Value Function Search ), which achieves provable sample complexity guarantees under strengthened statistical assumption known as pushforward coverability . RVFS can be viewed as a principled, provable counterpart to a successful empirical paradigm that combines recursive search (e.g., MCTS) with value function approximation.

1 Introduction

Simulators are a widely used tool in reinforcement learning. Many of the most well-known benchmarks for reinforcement learning research make use of simulators (Atari [9], MuJoCo [55], OpenAI Gym[11], DeepMind Control Suite [53]), and high-quality simulators are available for a wide range of real-world control tasks, including robotic control [45, 2], autonomous vehicles [10, 6], and game playing [51, 52]. Simulators also provide a useful abstraction for planning with a known or learned model, an important building block for many RL techniques [48]. Yet, in spite of the ubiquity of simulators, almost all existing research into algorithm design-empirical and theoretical-has focused on the online reinforcement learning (where only trajectory-based feedback is available), and does not take advantage of the extra information available through the simulator. Relatively little is known about the full power of RL with simulator access, either in terms of algorithmic principles or fundamental limits.

We explore the power of simulators through online reinforcement learning with local simulator access (RLLS for short), also known as local planning [57, 40, 3, 59, 65, 66]. Here, the agent learns by repeatedly executing policies and observing the resulting trajectories (as in online RL), but is allowed to reset to previously observed states and follow their dynamics during training.

Empirically, algorithms based on local simulators have received limited investigation, but with promising results. Notably, the Go-Explore algorithm [19, 20] uses local simulator access to achieve state-of-the-art performance for Montezuma's Revenge (a difficult Atari game that requires systematic

exploration), beating the performance of the best agents trained with online RL [7, 27] by a significant margin that has yet to be closed. The successful line of research on AlphaGo and successors [51, 52, 48] also uses local simulator access, albeit at test time in addition to training time.

These results suggest that developing improved algorithm design principles for RL with local simulator access could have significant practical implications, but current theoretical understanding of local simulators is limited. Recent work has shown that local simulator access has provable benefits for reinforcement learning with various types of linear function approximation [57, 40, 3, 65, 59], but essentially nothing is known for RL problems in large state spaces that demand general, potential neural function approximation. This leads us to ask: Can we develop algorithms for reinforcement learning with general function approximation that provably benefit from local simulator access?

From an algorithm design perspective, perhaps the greatest challenge in using local simulators to speed up learning is to understand which states are 'informative' in the sense that we should prioritize revisiting them. Here, we are faced with a chicken-and-egg problem: to understand which states to prioritize, we must explore and gather information, but it is unclear how to do so efficiently unless we already have a way to understand which states are informative. It is natural to let function approximation guide us; to this end, recent research [40, 65, 59] on linearly-parameterized RL with local simulators makes use of core-sets : small, adaptively chosen sets of informative state-action pairs designed to cover the feature space and enable efficient value function learning. Core-sets facilitate sample complexity guarantees for linear models that are not possible without local simulator access (e.g., [40]). Yet, for general function classes-particularly rich models like neural networks that do not readily support extrapolation-defining a suitable notion of core-set is challenging. Consequently, existing techniques have yet to meaningfully leverage local simulator access beyond the linear regime.

1.1 Contributions

We show that local simulator access unlocks new guarantees for online RL with general value function approximation-statistical and computational-that were previously out of reach.

Sample-efficient learning. We show that MDPs with low coverability [63]-a general structural condition that subsumes Block MDPs and Low-Rank MDPs-can be learned in a sample-efficient fashion with only Q ⋆ -realizability (that is, realizability for the optimal state-action value function). This is achieved through a new algorithm, SimGolf , that augments the principle of global optimism with local simulator access, and improves upon the best existing guarantees for the fully online RL setting, which require significantly stronger representation conditions. As a consequence, we show for the first time that the notoriously challenging Exogenous Block MDP (ExBMDP) problem [22, 21] is tractable in its most general form under local simulator access.

Practical, computationally efficient learning. Our results above are achieved through a computationally inefficient algorithm. We complement them with a practical and computationally efficient algorithm, RVFS ('Recursive Value Function Search'), which achieves sample-efficient learning guarantees with general value function approximation under a strengthened, yet novel, statistical assumption known as pushforward coverability [62]. Assuming either i) realizability of the optimal state-value function V ⋆ and a state-action gap or ii) realizability of V π for all π , RVFS achieves polynomial sample complexity in a computationally efficient fashion, and leads to guarantees for a new class of Exogenous Block MDPs with weakly correlated exogenous noise. RVFS explores by building core-sets with a novel value function-guided scheme, and can be viewed as a principled counterpart to algorithms including MCTS and AlphaZero [51, 52, 19, 20, 66], that combine recursive search with value function approximation. Compared to these approaches, RVFS is designed to provably address stochastic environments and distribution shift.

Paper organization. Section 2 introduces the local simulator framework. Section 3 presents our main sample complexity guarantees, and Section 4 gives computationally efficient algorithms. All proofs are deferred to the appendix.

2 Setup: Reinforcement Learning with Local Simulator Access

We consider an episodic reinforcement learning setting. A Markov Decision Process (MDP) is a tuple M=(X , A , T, R, H ) , where X is a (large/potentially infinite) state space, A is the action space (we abbreviate A = ∣A∣ ), H ∈ N is the horizon, R = { R h } H h = 1 is the reward function (where R h ∶ X ×A → [ 0 , 1 ] ) and T = { T h } H h = 0 is the transition distribution (where T h ∶ X × A → ∆ (X) ), with the convention that T 0 (⋅ ∣ ∅) is the initial state distribution. A policy is a sequence of functions π = { π h ∶ X →

∆ (A)} H h = 1 ; we use Π S to denote the set of all such functions. When a policy is executed, it generates a trajectory ( x 1 , a 1 , r 1 ) , . . . , ( x H , a H , r h ) via the process a h ∼ π h ( x h ) , r h ∼ R h ( x h , a h ) , x h + 1 ∼ T h (⋅ ∣ x h , a h ) , initialized from x 1 ∼ T 0 (⋅ ∣ ∅) (we use x H + 1 to denote a terminal state with zero reward). We write P π [⋅] and E π [⋅] to denote the law and expectation under this process.

For a policy π , J ( π ) ∶= E π [∑ H h = 1 r h ] denotes expected reward, and the value functions are given by V π h ( x ) ∶= E π [∑ H h ′ = h r h ′ ∣ x h = x ] , and Q π h ( x, a ) ∶= E π [∑ H h ′ = h r h ′ ∣ x h = x, a h = a ] . We denote by π ⋆ the optimal deterministic policy that maximizes Q π ⋆ , and write Q ⋆ ∶= Q π ⋆ and V ⋆ ∶= V π ⋆ .

Online reinforcement learning with a local simulator. In the standard online reinforcement learning framework, the learner repeatedly interacts with an (unknown) MDP by executing a policy and observing the resulting trajectory, with the goal of maximizing the total reward. Formally, for each episode τ ∈ [ N episodes ] , the learner selects a policy π ( τ ) = { π ( τ ) h } H h = 1 , executes it in the underlying MDP M ⋆ and observes the trajectory {( x ( τ ) h , a ( τ ) h , r ( τ ) h )} H h = 1 . After all N episodes episodes conclude, the learner produces a policy ̂ π ∈ Π S with the goal of minimizing the risk given by E [ J ( π ⋆ ) -J (̂ π )] .

In online RL with local simulator access, or RLLS, [57, 40, 65, 59, 66], we augment the online RL protocol as follows: At each episode τ ∈ [ N ] , instead of starting from a random initial state x 1 ∼ T 0 (⋅ ∣ ∅) , the agent can reset the MDP to any layer h ∈ [ H ] and any state x h previously encountered, and proceed with a new episode starting from x h . As in the online RL protocol, the goal is to produce a policy ̂ π ∈ Π S such that E [ J ( π ⋆ ) -J (̂ π )] ≤ ε with as few episodes of interaction as possible; our main results take N episodes = poly ( C,ε -1 ) for a suitable problem parameter C .

Executable versus non-executable policies. We focus on learning policies that can be executed without access to a local simulator (in other words, the local simulator used at train time, but not test time). Some recent work using local simulators for RL with linear function approximation [57] considers a more permissive setting where the final policy π produced by the learner can be non-executable ; our function approximation requirements can be slightly relaxed in this case.

Definition 2.1 (Non-executable policy) . We refer to a policy π for which computing π ( x ) ∈ ∆ (A) for any x ∈ X requires n local simulator queries as a non-executable policy with sample complexity n .

Additional notation. For any m,n ∈ N , we denote by [ m..n ] the integer interval { m,...,n } . We also let [ n ] ∶= [ 1 .. n ] . We refer to a scalar c > 0 as an absolute constant to indicate that it is independent of all problem parameters and use ̃ O (⋅) to denote a bound up to factors polylogarithmic in parameters appearing in the expression. We define π unif ∈ Π S as the random policy that selects actions in A uniformly. We define the occupancy measure for policy π via d π h ( x, a ) ∶= P π [ x h = x, a h = a ] . For functions g ∶ X × A → R and f ∶ X → R , we define Bellman backup operators by T h [ g ]( x, a ) = E [ r h + max a ′ ∈A g ( x h + 1 , a ′ ) ∣ x h = x, a h = a ] and P h [ f ]( x, a ) = E [ r h + f ( x h + 1 ) ∣ x h = x, a h = a ] . For a stochastic policy π ∈ Π S , we will occasionally use the bold notation π h ( x ) as shorthand for the random variable a h ∼ π h ( x ) ∈ ∆ (A) . For a function f ∶A→ R , we write a ′ ∈ arg max a ∈A f ( a ) to denote the action that maximizes f . If there are ties, we break them by picking the action with the smallest index; we assume without loss of generality that actions in A are index from 1 , . . . , ∣A∣ .

3 New Sample-Efficient Learning Guarantees via Local Simulators

This section presents our most powerful results for RLLS. We present a new algorithm for learning with local simulator access, SimGolf (Section 3.1), and show that it enables sample-efficient RL for MDPs with low coverability [63] using only Q ⋆ -realizability (Section 3.2). We then give implications for the Exogenous Block MDP problem (Section 3.3).

Function approximation setup and coverability. To achieve sample complexity guarantees for online reinforcement learning that are suitable for large, high-dimensional state spaces, we appeal to value function approximation . We assume access to a function class Q⊂(X ×A×[ H ] → [ 0 , H ]) that contains the optimal state-action value function Q ⋆ ; we define Q h = { Q h ∣ Q ∈ Q} .

Assumption 3.1 ( Q ⋆ -realizability) . For all h ∈ [ H ] , we have Q ⋆ h ∈ Q h .

Q ⋆ -realizability is widely viewed as a minimal representation condition for online RL [61, 17, 16, 39, 58, 56]. The class Q encodes the learner's prior knowledge about the MDP, and can be parameterized by rich function approximators like neural networks. We assume for simplicity of exposition that Q

and Π are finite, and aim for sample complexity guarantees scaling with log ∣Q∣ and log ∣ Π ∣ ; extending our results to infinite classes via standard uniform convergence arguments is straightforward.

Coverability. Beyond representation conditions like realizability, online RL algorithms require structural conditions that limit the extent to which deliberately designed algorithms can be surprised by substantially new state distributions. We focus on a structural condition known as coverability [63], which is inspired by connections between online and offline RL.

Coverability is an intrinsic strutural property of the underlying MDP. Examples of MDP families with low coverability include (Exogenous) Block MDPs, which have C cov ≤ ∣S∣∣A∣ , where S is the latent state space [63], and Low-Rank MDPs, which have C cov ≤ d ∣A∣ , where d is the feature dimension [29]; importantly, these settings exhibit high-dimensional state spaces and require nonlinear function approximation. As in prior work [63, 4], our algorithms require no prior knowledge of the distribution µ h that achieves the minimum in Assumption 3.2.

3.1 Algorithm

Our main algorithm, SimGolf , is displayed in Algorithm 1. The algorithm is a variant of the GOLF method of Jin et al. [32], Xie et al. [63] with novel adaptations to exploit the availability of a local simulator. Like GOLF , SimGolf explores using the principle of global optimisim : At each iteration t ∈ [ N ] , it maintains a confidence set (or, version space) Q ( t ) ⊂ Q of candidate value functions with low squared Bellman error under the data collected so far, and chooses a new exploration policy π ( t ) by picking the most 'optimistic' value function in this set. As the algorithm gathers more data, the confidence set shrinks, leaving only near-optimal policies.

The main novelty in SimGolf arises in the data collection strategy and design of confidence sets. Like GOLF , SimGolf algorithm constructs the confidence set Q ( t ) ⊂ Q such that all value functions g ∈ Q ( t ) have small squared Bellman error:

Due to the presence of the Bellman backup T h [ g h + 1 ] in Eq. (1), naively estimating squared Bellman error leads to the notorious double sampling problem. To avoid this, the approach taken with GOLF and related work [67, 32] is to adapt a certain de-biasing technique to remove double sampling bias, but this requires access to a value function class that satisfies Bellman completeness , a representation significantly more restrictive than realizability (e.g., Foster et al. [26]).

The idea behind SimGolf is to use local simulator access to directly produce high-quality estimates for the Bellman backup function T h [ g h + 1 ] in Eq. (1). In particular, for a given state-action pair ( x, a ) ∈ X × A , we can estimate the Bellman backup T h [ g h + 1 ]( x, a ) for all functions g ∈ Q simultaneously by collecting K next-state transitions ̃ x ( 1 ) h + 1 , . . . , ̃ x ( K ) h + 1 i.i.d. ∼ T h (⋅ ∣ x, a ) and K rewards ̃ r ( 1 ) h , . . . , ̃ r ( K ) h i.i.d. ∼ R h ( x, a ) , then taking the empirical mean: T h [ g h + 1 ]( x, a ) ≈ 1 K ∑ K k = 1 (̃ r ( k ) h + max a ′ ∈A g h + 1 (̃ x ( k ) h + 1 , a ′ )) . Line 8 of SimGolf uses this technique to directly estimate the Bellman residual backup under a trajectory gathered with π ( t ) , sidestepping the double sampling problem and removing the need for Bellman completeness. We suspect this technique (estimation with respect to squared Bellman error using local simulator access) may find broader use.

3.2 Main Result

We now state the main guarantee for SimGolf and discuss some of its implications.

Theorem 3.1 (Main guarantee for SimGolf ) . Let ε, δ ∈ ( 0 , 1 ) be given and suppose Assumption 3.1 ( Q ⋆ -realizability) and Assumption 3.2 (coverability) hold with C cov > 0 . Then the policy ̂ π produced by SimGolf (Q , C cov , ε, δ ) (Algorithm 1) has J ( π ⋆ ) -E [ J (̂ π )] ≤ ε with probability at least 1 -δ . The total sample complexity in the RLLS framework is bounded by ̃ O ( H 5 C 2 cov log (∣Q∣/ δ ) ⋅ ε -4 ) .

This result (whose proof is in Appendix E) shows that under only Q ⋆ -realizability and coverability, SimGolf learns an ε -optimal policy with polynomial sample complexity, significantly relaxing the representation assumptions (Bellman completeness, weight function realizability) required by prior algorithms for coverability [63, 4]. This is the first instance we are aware of where local simulator

Algorithm 1 SimGolf : Global Optimism via Local Simulator Access

1: input: Value function class Q , coverability C cov > 0 , suboptimality ε > 0 , and confidence δ > 0 .

3: initialize: Q ( 1 ) ←Q .

2: Set N ← ̃ Θ ( H 2 C cov β / ε 2 ) , β stat ← 16log ( 2 HN ∣Q∣ δ -1 ) , β ← 2 β stat , and K ← 8 N β stat .

4: for iteration t = 1 , 2 , . . . , N do

6: For each h ∈ [ H ] and x ∈ X , define π ( t ) h ( x ) ∈ arg max a ∈A g ( t ) h ( x, a ) .

5: Select g ( t ) = arg max g ∈Q ( t ) ∑ t -1 s = 1 max a ∈A g 1 ( x ( s ) 1 , a ) .

7: Execute π ( t ) for an episode and observe τ ( t ) ∶= ( x ( t ) 1 , a ( t ) 1 ) , . . . , ( x ( t ) H , a ( t ) H ) .

9: Compute confidence set:

8: For h ∈ [ H ] , draw K independent samples x ( t,k ) h + 1 ∼ T h (⋅ ∣ x ( t ) h , a ( t ) h ) , r ( t,k ) h ∼ R h ( x ( t ) h , a ( t ) h ) .

10: return: ̂ π = unif ( π ( 1 ) , . . . , π ( N ) ) .

access unlocks sample complexity guarantees for reinforcement learning with nonlinear function approximation that were previously out of reach; perhaps the most important technical idea here is our approach to combining global optimism with local simulator access, in contrast to greedy layer-by-layer schemes used in prior work on local simulators (with the exception of Weisz et al. [57]). In particular, we suspect that the idea of performing estimation with respect to squared Bellman error directly using local simulator access may find broader use beyond coverability. Improving the polynomial dependence on problem parameters is an interesting question for future work.

A conjecture. By analogy to results in offline reinforcement learning, where Q ⋆ -realizability and concentrability (the offline counterpart to coverability) alone are known to be insufficient for sample-efficient learning [12, 26], we conjecture that Q ⋆ -realizability and coverability alone are not sufficient for polynomial sample complexity in vanilla online RL. If true, this would imply a new separation between online RL with and without local simulators.

3.3 Implications for Exogenous Block MDPs

We now apply SimGolf and Theorem 3.1 to the Exogenous Block MDP (ExBMDP) problem [22, 21, 38, 30], a challenging rich-observation reinforcement learning setting in which the observed states x h are high-dimensional, while the underlying dynamics of the system are low-dimensional, yet confounded by temporally correlated exogenous noise.

Formally, an Exogenous Block MDP M=(X , S , Ξ , A , H, T, R, g ) is defined by a latent state space and an observation space . We begin with the latent state space. Starting from an initial endogenous state s 1 ∈ S and exogenous state ξ 1 ∈ Ξ , the latent state z h = ( s h , ξ h ) evolves for h ∈ [ H ] via s h + 1 ∼ T endo h (⋅ ∣ s h , a h ) and ξ h + 1 ∼ T exo h (⋅ ∣ ξ h ) , where a h ∈ A is the agent's action at layer h ; we adopt the convention that s 1 ∼ T endo 0 (⋅ ∣ ∅) and ξ 1 ∼ T exo 0 (⋅ ∣ ∅) . Note that only the endogenous state is causally influenced by the action. The latent state is not observed; instead, at each step h , the agent receives an observation x h ∈ X generated via 1 x h = g obs h ( s h , ξ h ) , where g obs h ∶ S × Ξ →X is the emission function . We assume the endogenous latent space S and action space A are finite, and define S ∶= ∣S∣ and A ∶= ∣A∣ . However, the exogenous state space Ξ and observation space X may be arbitrarily large or infinite, with ∣ Ξ ∣ , ∣X∣ ≫ ∣S∣ . 2

The final property of the ExBMDP model is decodability , which asserts the existence of a decoder such that ϕ ⋆ ∶ X → S such that ϕ ⋆ ( x h ) = s h a.s. for all h ∈ [ H ] with x h = g obs h ( s h , ξ h ) , . Informally, decodability ensures the existence of an (unknown to the learner) mapping that allows one to perfectly recover the endogenous latent state from observations. In addition to decodability, we assume the rewards in the ExBMDP are endogenous ; that is, the reward distribution R h ( x h , a h ) only

depends on the observations ( x h ) through the corresponding latent states ( ϕ ⋆ ( x h ) = s h ) . To enable sample-efficient learning, we assume access to a decoder class Φ that contains ϕ ⋆ , as in prior work.

Assumption 3.3 (Decoder realizability) . We have access to a decoder class Φ such that ϕ ⋆ ∈ Φ .

Applying SimGolf and Theorem 3.1. To apply Theorem 3.1 to the ExBMDP problem, we need to verify that Q ⋆ -realizability and coverability hold. Realizability is a straightforward consequence of decodability (Lemma D.1 in Part II of the appendix). For coverability, Xie et al. [63] show that ExBMDPs have C cov ≤ SA under decodability, in spite of the time-correlated exogenous noise process ( ξ h ) and potentially infinite observation space X (interestingly, coverability is essentially the only useful structural property that ExBMDPs are known to satisfy, which is our primary motivation for studying it). This leads to the following corollary of Theorem 3.1.

Corollary 3.1 ( SimGolf for ExBMDPs) . Consider the ExBMDP setting. Suppose that Assumption 3.3 holds, and let Q be constructed as in Lemma D.1 of Part II. Then for any ε, δ ∈ ( 0 , 1 ) , the policy ̂ π = SimGolf (Q , SA, ε, δ ) has J ( π ⋆ ) -J (̂ π ) ≤ ε with probability at least 1 -δ . The total sample complexity in the RLLS framework is N = ̃ O ( H 5 S 3 A 3 log ∣ Φ ∣ ⋅ ε -4 ) .

This shows for the first time that general ExBMDPs are learnable with local simulator access. Prior to this work, online RL algorithms for ExBMDPs required either (i) deterministic latent dynamics [22], or (ii) factored emission structure [21]. Xie et al. [63] observed that ExBMDPs admit low coverability, but their algorithm requires Bellman completeness, which is not satisfied by ExBMDPs (see Islam et al. [30]). See Appendix A for more discussion.

4 Computationally Efficient Learning with Local Simulators

Our result in Section 3 show that local simulator access facilitates sample-efficient learning in MDPs with low coverability, a challenging setting that was previously out of reach. However, our algorithm SimGolf is computationally-inefficient because it relies on global optimism, a drawback found in most prior work on RL with general function approximation [31, 32, 18]. It remains an open question whether any form of global optimism can be implemented efficiently, and some variants have provable barriers to efficient implementation [14].

To address this drawback, in this section we present a new algorithm, RVFS (Recursive Value Function Search; Algorithm 5), which requires stronger versions of the coverability and realizability assumptions in Section 3, but is computationally efficient in the sense that it reduces to convex optimization over the state-value function class V . RVFS makes use of a sophisticated recursive exploration scheme based on core-sets, sidestepping the need for global optimism.

4.1 Function Approximation and Statistical Assumptions

To begin, we require the following strengthening of the coverability assumption in Assumption 3.2.

Assumption 4.1 (Pushforward coverability) . The pushforward coverability coefficient C push > 0 is given by C push = max h ∈[ H ] inf µ h ∈ ∆ (X) sup ( x h -1 ,a h -1 ,x h )∈X h -1 ×A×X T h -1 ( x h ∣ x h -1 ,a h -1 ) µ h ( x h ) .

Pushforward coverability is inspired by the pushforward concentrability condition used in offline RL by [62, 26]. Concrete examples include, (i) Block MDPs with latent space S , which admit C push ≤ ∣S∣ , (ii) Low-Rank MDPs in dimension d , which admit C push ≤ d [62], and (iii) Exogenous Block MDPs for which the exogenous noise process satisfies a weak correlation condition that we introduce in Appendix B. Note that C cov ≤ C push ∣A∣ , but the converse is not true in general.

Instead of state-action value function approximation as in SimGolf , in this section we make use of a state value function class V ⊂ (X × [ H ] → [ 0 , H ]) , but require somewhat stronger representation conditions than in Section 3. We consider two complementary setups:

Setup I: Assumptions 4.2 and 4.3 ( V ⋆ / π ⋆ -realizability) and Assumption 4.4 ( ∆ -gap) hold.

Setup II: Assumption 4.5 ( V π -realizability) and Assumption 4.6 ( π -realizability) hold.

We describe these assumptions in more detail below.

Function approximation setup I. First, instead of Q ⋆ -realizability, we consider the weaker V ⋆ -realizability [31, 57, 3].

Assumption 4.2 ( V ⋆ -realizability) . For all h ∈ [ H ] , we have V ⋆ h ∈ V h .

Under V ⋆ -realizability, our algorithm learns a near-optimal policy, but the policy is non-executable (cf. Definition 2.1); this property is shared by prior work on local simulator access with value function realizability [57] . To produce executable policies, we additionally require access to a policy class Π ⊂ Π S containing π ⋆ ; we define Π h = { π h ∣ π ∈ Π } .

Assumption 4.3 ( π ⋆ -realizability) . The policy class Π contains the optimal policy π ⋆ .

V ⋆ -realizability (Assumption 4.2) and π ⋆ -realizability (Assumption 4.3) are both implied by Q ⋆ -realizability, and hence are weaker. However, we also assume the optimal Q -function admits constant gap (this makes the representation conditions for Setup I incomparable to Assumption 3.1).

Assumption 4.4 ( ∆ -Gap) . The optimal action π ⋆ h ( x ) is unique, and there exists ∆ > 0 such that for all h ∈ [ H ] , x ∈ X , and a ∈ A ∖ { π ⋆ h ( x )} , Q ⋆ h ( x, π ⋆ h ( x )) > Q ⋆ h ( x, a ) + ∆ .

This condition has been used in a many prior works on computationally efficient RL with function approximation [16, 17, 24, 56].

Function approximation setup II. We also provide guarantees under the assumption that the class V satisfies all-policy realizability [59, 65, 60] in the sense that V π ∈ V for all π ∈ Π S .

Assumption 4.5 ( V π -realizability) . The class V = V 1 ∶ H has V π h ∈ V h for all π ∈ Π S and h ∈ [ H ] .

This assumption will be sufficient to learn a non-executable policy, but to learn executable policies we require an analogous strengthening of Assumption 4.5.

Assumption 4.6 ( π -realizability) . For all π ∈ Π S , we have that x ↦ arg max a ∈A P h [ V π h + 1 ](⋅ , a ) ∈ Π .

This assumption has been used by a number of prior works on computationally efficient RL [8, 44]. Assumptions 4.5 and 4.6 are both implied by the slightly simpler-to-state assumption of Q π -realizability [59, 65, 60], which asserts access to a class Q that contains Q π for all π ∈ Π S .

4.2 Algorithm

For ease of exposition, we defer the full version of our algorithm, RVFS (Algorithm 5), to Appendix F and present a simplified version here (Algorithm 2). The algorithms are nearly identical, except that the simplified version assumes that certain quantities of interest (e.g., Bellman backups) can be computed exactly, while the full version (provably) approximates them from samples.

RVFS maintains a value function estimator ̂ V = ̂ V 1 ∶ H that aims to approximate the optimal value function V ⋆ 1 ∶ H , as well as core sets C 1 , . . . , C H of state-action pairs that are used to perform estimation and guide exploration. At a high level, RVFS alternates between (i) fitting the value function ̂ V h for a given layer h ∈ [ H ] based on Monte-Carlo rollouts, and (ii) using the core-sets to test whether the current value function estimates ̂ V h + 1 ∶ H remain accurate as the roll-in policy induced by ̂ V h changes.

In more detail, RVFS is based on recursion across the layers h ∈ [ H ] . When invoked for layer h with value function estimates ̂ V h + 1 ∶ H and core-sets C h , . . . , C H , RVFS h performs two steps:

For each state-action pair ( x h -1 , a h -1 ) ∈ C h , 4 the algorithm gathers N test trajectories by rolling out from ( x h -1 , a h -1 ) with the greedy policy ̂ π ℓ ( x ) ∈ arg max a ∈A P ℓ [ ̂ V ℓ + 1 ]( x, a ) that optimizes the estimated value function; in the full version of RVFS (see Algorithm 5), we estimate the bellman backup P ℓ [ ̂ V ℓ + 1 ]( x, a ) using the local simulator. For all states x ℓ -1 ∈ { x h , . . . , x H -1 } encountered during this process, the algorithm checks whether ∣ E [ ̂ V ℓ ( x ℓ ) -V ⋆ ℓ ( x ℓ ) ∣ x ℓ -1 = x ℓ -1 , a ℓ -1 = a ℓ -1 ]∣ ≲ ε for all a ℓ -1 ∈ A using a test based on (implicitly maintained) confidence sets. If the test fails, this indicates that distribution shift has occurred, and the algorithm adds the pair ( x ℓ -1 , a ℓ -1 ) to C ℓ and recurses on layer ℓ via RVFS ℓ .

If all tests above pass, this means that ̂ V h + 1 , . . . , ̂ V H are accurate, and no distribution shift has occurred. In this case, the algorithm fits ̂ V h by collecting Monte-Carlo rollouts from all state-action pairs in the core-set C h with ̂ π ℓ ( x ) ∈ arg max a ∈A P ℓ [ ̂ V ℓ + 1 ]( x, a ) (cf. Line 16), and returns.

When the tests in Item 1 succeed for all h ∈ [ H ] , the algorithm returns the estimated value functions ̂ V 1 ∶ H ; in this case, the greedy policy ̂ π ℓ ( x ) ∈ arg max a ∈A P ℓ [ ̂ V ℓ + 1 ]( x, a ) is guaranteed to be near

optimal. The full version of RVFS in Algorithm 5 uses local simulator access to estimate the Bellman backups P h [ ̂ V h + 1 ]( x, a ) for different state-action pairs ( x, a ) . These backups are used to (i) compute actions of the greedy policy that maximizes ̂ V 1 ∶ H via (e.g., Eq. (2)); (ii) generate trajectories by rolling out from state-action pairs in the core-sets (Line 6); and (iii) perform the test in Item 1 (Line 8).

RVFS is inspired by the DMQ algorithm [16, 56] originally introduced in the context of online reinforcement learning with linearly realizable Q ⋆ . RVFS incorporates local simulator access (most critically, via core-set construction) to allow for more general nonlinear function approximation without restrictive statistical assumptions. Prior algorithms for RLLS have used core-sets of state-action pairs in a similar fashion [40, 65, 59], but in a way that is tailored to linear function approximation.

In what follows, we discuss various features of the algorithm in greater detail.

Bellman backup policies. Since RVFS works with state value functions instead of state-action value functions, we need a way to extract policies from the former. The most natural way to extract a policy from estimated value functions ̂ V 1 ∶ H ∈ V is as follows: for all h ∈ [ H ] , define ̂ π h ( x ) ∈ arg max a ∈A P h [ ̂ V h + 1 ]( x, a ) . In reality, we do not have access to P h [ ̂ V h + 1 ]( x, a ) directly, so the full version of RVFS (Algorithm 5) estimates this quantity on the fly using the local simulator using the following scheme (Algorithm 7 in Appendix F): Given a state x , for each a , we sample K rewards r h ∼ R h ( x, a ) and next-state transitions x h + 1 ∼ T h (⋅ ∣ x, a ) , then approximate P h [ ̂ V h + 1 ]( x, a ) by the empirical mean. We remark that the use of these Bellman backup policies is actually crucial in the analysis for RVFS ; even if we were to work with estimated state-action value functions ̂ Q 1 ∶ H instead, our analysis would require executing the Bellman backup policies ̂ π h ( x ) ∈ arg max a ∈A T h [ ̂ Q h + 1 ]( x, a ) (instead of naively using ̂ π h ( x ) ∈ arg max a ∈A ̂ Q h ( x, a ) ).

Invoking the algorithm. The base invocation of RVFS takes the form

Whenever this call returns, the greedy policy induced by ̂ V 1 ∶ H is guaranteed to be near-optimal. Naively, the approximate Bellman backup policy induced by ̂ V 1 ∶ H (described above) is non-executable, and must be computed by invoking the local simulator. To provide an end-to-end guarantee to learn an executable policy, we give an outer-level algorithm, RVFS.bc (Algorithm 6, deferred to Appendix F for space), which invokes RVFS 0 , then extracts an executable policy from ̂ V 1 ∶ H using behavior cloning. Subsequent recursive calls to RVFS take the form ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H ) ← RVFS h ( ̂ V h + 1 ∶ H , V ̂ h + 1 ∶ H , C h ∶ H ; V , ε, δ ) . The arguments here are: Importantly, the confidence sets V ̂ h + 1 ∶ H do not need to be explicitly maintained, and can be used implicitly whenever a regression oracle for the value function class is available (discussed below).

Remark 4.1 (Oracle-efficiency) . RVFS is computationally efficient in the sense that it reduces to convex optimization over the value function class V . In particular, the only computationally intensive steps in the algorithm are (i) the regression step in Line 16, and (ii) the test in Line 8 involving the confidence set V ̂ ℓ . For the latter, we do not explicitly need to maintain V ̂ ℓ , as the optimization problem over this set in Line 8 (for the full version of RVFS in Algorithm 5) reduces to solving arg max V ∈V {±∑ n i = 1 V (̃ x ( i ) ) ∣ ∑ n i = 1 ( V ( x ( i ) ) -y ( i ) ) 2 ≤ β 2 } for a dataset {( x ( i ) , ̃ x ( i ) , y ( i ) )} n i = 1 . This is convex optimization problem in function space, and in particular can be implemented in a provably efficient fashion whenever V is linearly parameterized. We expect that the problem can also be reduced to a square loss regression by adapting the techniques in Krishnamurthy et al. [37], Foster et al. [23], but we do not pursue this here.

4.3 Main Result

We present the main guarantee for RVFS under the function approximation assumptions in Section 4.1.

Theorem 4.1 (Main guarantee for RVFS ) . Let ε, δ ∈ ( 0 , 1 ) be given, and suppose that Assumption 4.1 (pushforward coverability) holds with C push > 0 . Further, suppose that one the following holds:

Setup I: Assumptions 4.2 and 4.3 ( V ⋆ / π ⋆ -realizability) and Assumption 4.4 ( ∆ -gap) hold, and ε ≤ 6 H ⋅ ∆ .

Setup II: Assumption 4.5 ( V π -realizability) and Assumption 4.6 ( π -realizability) hold.

Then, RVFS.bc ( Π , V , ε, δ ) (Algorithm 6) returns a policy ̂ π 1 ∶ H such that J ( π ⋆ ) -J (̂ π 1 ∶ H ) ≤ 2 ε with probability at least 1 -δ , and has total sample complexity bounded by

Furthermore, the algorithm makes at most poly ( C push , H, A, ε -1 ) calls to the convex optimization oracle over value function space described in Remark 4.1.

Theorem 4.1 shows for the first time that sample- and computationally-efficient RL with local simulator access is possible under pushforward coverability. In particular, RVFS is the first computationally efficient algorithm for RL with local simulator access that supports nonlinear function approximation. The assumptions in Theorem 4.1, while stronger than those in Section 3, are not known to enable sample-efficient RL without simulator access. Nonetheless, understanding whether RVFS can be strengthened to support general coverability or weaker function approximation is an important open problem. See Appendix H.1 for an overview of the analysis; we remark (Appendix I.1) that the result is actually proven under slightly weaker assumptions than those in Setup I / Setup II .

Connection to empirical algorithms. RVFS bears some similarity to Monte-Carlo Tree Search (MCTS) [13, 35] and AlphaZero [52], which perform planning with local simulator. Informally, MCTScan be viewed as a form of breadth-first search over the state space (where each node represents a state at a given layer), and AlphaZero is a particular instantiation of a MCTS that leverages V -value function approximation to allow for generalization across states. Compared to RVFS , MCTS and AlphaZero perform exploration via simple bandit-style heuristics, and are not explicitly designed to handle distribution shifts that arise in settings where actions have long-term downstream effects. What is more, MCTS requires finite states to iterate over all possible child nodes of each state, making it inapplicable in environments with continuous states. RVFS may be viewed as a provable counterpart

that can handle continuous states and uses function approximation to address distribution shift in a principled fashion (in particular, through the use of confidence sets and the test in Line 14). 5

Applying RVFS to Exogenous Block MDPs. ExBMDPs satisfy coverability (Assumption 3.2), but do not satisfy the pushforward coverability assumption (Assumption 4.1) in general. However, it turns out that ExBMDPs do satisfy pushforward coverability when the exogenous noise process is weakly correlated across time, a new statistical assumption we refer to the weak correlation condition . In Appendix B (Theorem B.1), we give a variant of RVFS for ExBMDPs that succeeds under (i) weak correlation, and (ii) decoder realizability, sidestepping the need for the ∆ -gap or V π -realizability.

5 Discussion and Future Work

In this paper, we demonstrated that resets can substantially expand the range of reinforcement learning (RL) settings that are tractable, both statistically and computationally. Our practical algorithm, RVFS, provides a principled counterpart to MCTS by supporting continuous state spaces and offering provable guarantees, setting it apart from traditional MCTS. Statistically, our results extend to MDPs with a finite Sequential Estimation Coefficient (SEC) [63], capturing a broader class of MDPs beyond those with finite coverability-encompassing low Bellman Eluder MDPs [32] and MDPs with finite bilinear rank [18]. Although not formally developed here, it is possible to generalize push-forward coverability and our analysis to encompass a range of linear function approximation settings [57, 40, 3, 59, 65, 66], thereby recovering known positive results under local access in these settings.

While our focus has been on theoretical contributions-analyzing the sample and computational complexity of RL with access to a local simulator-this work also raises promising empirical questions. We are particularly interested in exploring these questions in future work, aiming to bridge these theoretical advances with empirical validation in practical RL settings.

References

A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646, 2014.

I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

P. Amortila, N. Jiang, D. Madeka, and D. P. Foster. A few expert queries suffices for sampleefficient rl with resets and linear value approximation. Advances in Neural Information Processing Systems , 35:29637-29648, 2022.

P. Amortila, D. J. Foster, N. Jiang, A. Sekhari, and T. Xie. Harnessing density ratios for online reinforcement learning. International Conference on Learning Representations (ICLR) , 2024.

P. Amortila, D. J. Foster, and A. Krishnamurthy. Scalable online exploration via coverability. arXiv preprint arXiv:2403.06571 , 2024.

S. Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems , 23(2):740-759, 2020.

A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning , pages 507-517. PMLR, 2020.

J. Bagnell, S. M. Kakade, J. Schneider, and A. Ng. Policy search by dynamic programming. Advances in neural information processing systems , 16, 2003.

M. Bellemare, J. Veness, and M. Bowling. Investigating contingency awareness using atari 2600 games. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 26, pages 864-871, 2012.

M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

J. Chen and N. Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games , pages 72-83. Springer, 2006.

C. Dann, N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. On oracleefficient PAC RL with rich observations. In Advances in neural information processing systems , pages 1422-1432, 2018.

S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudik, and J. Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

S. S. Du, Y. Luo, R. Wang, and H. Zhang. Provably efficient Q-learning with function approximation via distribution shift error checking oracle. In Advances in Neural Information Processing Systems , pages 8060-8070, 2019.

S. S. Du, S. M. Kakade, R. Wang, and L. F. Yang. Is a good representation sufficient for sample efficient reinforcement learning? In International Conference on Learning Representations , 2020.

S. S. Du, S. M. Kakade, J. D. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in RL. International Conference on Machine Learning , 2021.

A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019.

A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature , 590(7847):580-586, 2021.

Y. Efroni, D. J. Foster, D. Misra, A. Krishnamurthy, and J. Langford. Sample-efficient reinforcement learning in the presence of exogenous information. In Conference on Learning Theory , pages 5062-5127. PMLR, 2022.

Y. Efroni, D. Misra, A. Krishnamurthy, A. Agarwal, and J. Langford. Provably filtering exogenous distractors using multistep inverse dynamics. In International Conference on Learning Representations , 2022.

D. J. Foster, A. Agarwal, M. Dudík, H. Luo, and R. E. Schapire. Practical contextual bandits with regression oracles. International Conference on Machine Learning , 2018.

D. J. Foster, A. Rakhlin, D. Simchi-Levi, and Y. Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. Conference on Learning Theory (COLT) , 2020.

D. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

D. J. Foster, A. Krishnamurthy, D. Simchi-Levi, and Y. Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. In Conference on Learning Theory , pages 3489-3489. PMLR, 2022.

Z. Guo, S. Thakoor, M. Pîslar, B. Avila Pires, F. Altché, C. Tallec, A. Saade, D. Calandriello, J.-B. Grill, Y . Tang, et al. Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems , 35:31855-31870, 2022.

W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

A. Huang, J. Chen, and N. Jiang. Reinforcement learning in low-rank mdps with density features. International Conference on Machine Learning (ICML) , 2023.

R. Islam, M. Tomar, A. Lamb, Y. Efroni, H. Zang, A. Didolkar, D. Misra, X. Li, H. van Seijen, R. T. d. Combes, et al. Agent-controller representations: Principled offline rl with rich exogenous information. International Conference on Machine Learning (ICML) , 2023.

N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning , pages 1704-1713, 2017.

C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Neural Information Processing Systems , 2021.

S. M. Kakade. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

M. Kearns and S. Singh. Finite-sample convergence rates for q-learning and indirect algorithms. Advances in neural information processing systems , 11, 1998.

L. Kocsis and C. Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning , pages 282-293. Springer, 2006.

A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems , pages 1840-1848, 2016.

A. Krishnamurthy, A. Agarwal, T.-K. Huang, H. Daumé III, and J. Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924, 2017.

A. Lamb, R. Islam, Y. Efroni, A. R. Didolkar, D. Misra, D. J. Foster, L. P. Molu, R. Chari, A. Krishnamurthy, and J. Langford. Guaranteed discovery of control-endogenous latent states with multi-step inverse models. Transactions on Machine Learning Research , 2023.

T. Lattimore, C. Szepesvari, and G. Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , pages 5662-5670. PMLR, 2020.

G. Li, Y. Chen, Y. Chi, Y. Gu, and Y. Wei. Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. Advances in Neural Information Processing Systems , 34:16671-16685, 2021.

Q. Liu, P. Netrapalli, C. Szepesvari, and C. Jin. Optimistic mle: A generic model-based algorithm for partially observable sequential decision making. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 363-376, 2023.

Z. Mhammedi, D. J. Foster, and A. Rakhlin. Representation learning with multi-step inverse kinematics: An efficient and optimal approach to rich-observation rl. International Conference on Machine Learning (ICML) , 2023.

D. Misra, M. Henaff, A. Krishnamurthy, and J. Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. arXiv preprint arXiv:1911.05815 , 2019.

D. Misra, M. Henaff, A. Krishnamurthy, and J. Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning , pages 6961-6971. PMLR, 2020.

M. A. Qassem, I. Abuhadrous, and H. Elaydi. Modeling and simulation of 5 dof educational robot arm. In 2010 2nd International Conference on Advanced Computer Control , volume 5, pages 569-574. IEEE, 2010.

S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

T. Salimans and R. Chen. Learning montezuma's revenge from a single demonstration. arXiv preprint arXiv:1812.03381 , 2018.

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

A. Sidford, M. Wang, X. Wu, L. Yang, and Y. Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. Advances in Neural Information Processing Systems , 31, 2018.

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

A. Tavakoli, V. Levdik, R. Islam, C. M. Smith, and P. Kormushev. Exploring restart distributions. arXiv preprint arXiv:1811.11298 , 2018.

E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Y. Wang, R. Wang, and S. M. Kakade. An exponential lower bound for linearly-realizable MDPs with constant suboptimality gap. Neural Information Processing Systems (NeurIPS) , 2021.

G. Weisz, P. Amortila, B. Janzer, Y. Abbasi-Yadkori, N. Jiang, and C. Szepesvári. On queryefficient planning in mdps under linear realizability of the optimal state-value function. In Conference on Learning Theory , pages 4355-4385. PMLR, 2021.

G. Weisz, P. Amortila, and C. Szepesvári. Exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory , pages 1237-1264. PMLR, 2021.

G. Weisz, A. György, T. Kozuno, and C. Szepesvári. Confident approximate policy iteration for efficient local planning in q π -realizable mdps. Advances in Neural Information Processing Systems , 35:25547-25559, 2022.

G. Weisz, A. György, and C. Szepesvári. Online rl in linearly q π -realizable mdps is as easy as in linear mdps if you learn what to ignore. arXiv preprint arXiv:2310.07811 , 2023.

Z. Wen and B. Van Roy. Efficient reinforcement learning in deterministic systems with value function generalization. Mathematics of Operations Research , 42(3):762-782, 2017.

T. Xie and N. Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021.

T. Xie, D. J. Foster, Y. Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

L. Yang and M. Wang. Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

D. Yin, B. Hao, Y. Abbasi-Yadkori, N. Lazi´ c, and C. Szepesvári. Efficient local planning with linear function approximation. In International Conference on Algorithmic Learning Theory , pages 1165-1192. PMLR, 2022.

D. Yin, S. Thiagarajan, N. Lazic, N. Rajaraman, B. Hao, and C. Szepesvari. Sample efficient deep reinforcement learning via local planning. arXiv preprint arXiv:2301.12579 , 2023.

A. Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

X. Zhang, Y. Song, M. Uehara, M. Wang, A. Agarwal, and W. Sun. Efficient reinforcement learning in block mdps: A model-free representation learning approach. In International Conference on Machine Learning , pages 26517-26547. PMLR, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims do accurately reflect the paper's contribution.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, see the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a proof-sketch in the main paper and detailed proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does conform to the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper provides a purely mathematical contribution. As such, it is subject to the standard ethical concerns present for all mathematical papers, but no further ones.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper has only mathematical congtent. There are no experiments in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Contents of Appendix

A Additional Related Work

Local simulators: Theoretical research. RLwith local simulators has received extensive interest in the context of linear function approximation. Most notably, Weisz et al. [57] show that reinforcement learning with linear V ⋆ is tractable with local simulator access, and Li et al. [40] show that RL with linear Q ⋆ and a state-action gap is tractable; online RL is known to be intractable under the same assumptions [57, 56]. Amortila et al. [3] show that the gap assumption can be removed if a small number of expert queries are available. Also of note are the works of Yin et al. [65], Weisz et al. [59], which give computationally efficient algorithms under linear Q π -realizability for all π ; this setting is known to be tractable in the online RL model [60], but computationally efficient algorithms are currently only known for RLLS.

Global simulators -in which the agent can query arbitrary state-action pairs and observe next state transitions-have also received theoretical investigation, but like local simulators, results are largely restricted to tabular reinforcement learning and linear models [34, 33, 50, 17, 64, 39].

Local simulators: Empirical research. The Go-Explore algorithm [19, 20] uses local simulator access to achieve state-of-the-art performance for the Atari games Montezuma's Revenge and Pitfallboth notoriously difficult games that require systematic exploration. To the best of our knowledge, the performance of Go-Explore on these tasks has yet to be matched by online reinforcement learning; the performing agents [7, 27] are roughly a factor of four worse in terms of cumulative reward. Interestingly, like RVFS , Go-Explore makes use of core sets of informative state-action pairs to guide exploration. However, Go-Explore uses an ad-hoc, domain specific approach to designing the core set, and does not use function approximation to drive exploration.

Recent work of Yin et al. [66] provides an empirical framework for online RL with local planning that can take advantage of deep neural function approximation, and is inspired by the theoretical works in Weisz et al. [57], Li et al. [40], Yin et al. [65], Weisz et al. [59]. This approach does not have provable guarantees, but achieves super-human performance at Montezuma's Revenge.

Other notable empirical works that incorporate local simulator access, as highlighted by Yin et al. [66], include Schulman et al. [49], Salimans and Chen [47], Tavakoli et al. [54].

Planning. RL with local simulator access is a convenient abstraction for the problem of planning : Given a known (e.g., learned) model, compute an optimal policy. Planning with a learned model is an important task in theory [25, 41] and practice (e.g., MuZero [48]). Since the model is known, computing an optimal policy is a purely computational problem, not a statistical problem. Nonetheless, for planning problems in large state spaces, where enumerating over all states is undesirable, algorithms for online RL with local simulator access can be directly applied, treating the model as if it were the environment the agent is interacting with. Here, any computationally efficient RLLS algorithm immediately yields an efficient algorithm for planning.

Empirically, Monte-Carlo Tree Search [13, 35] is a successful paradigm for planning, acting as a key component in AlphaGo [51] and AlphaZero [52]. 6 Viewed as a planning algorithm, a potential advantage of RVFS is that it is well suited to stochastic environments, and provides a principled way to use estimated (neural) value function estimates to guide exploration.

Coverability. Xie et al. [63] introduced coverability as a structural parameter for online reinforcement leanring, inspired by connections between online and offline RL. Existing guarantees for the online RL framework based on coverability require either Bellman completeness [63], model-based realizability [5], or weight function realizability [4, 5]), and it is not currently known whether value function realizability is sufficient in this framework.

Exogenous Block MDPs. Our results in Section 3.3 (Corollary 3.1) show that general Exogenous Block MDPs are learnable with local simulator access. Prior work, on learning EXBMDPs in the online RL model requires additional assumptions:

Deterministic ExBMDP [22]. In this setting, the latent transition distribution T endo is assumed to be deterministic. In this case, it suffices to learn open-loop policies (i.e., policies that play a deterministic sequence of actions). This avoids compounding errors due to

learning imperfect decoders that depend on the exogenous noise, making this setting much less challenging than the general ExBMDP setting.

Factored ExMDP [21]. This is an ExBMDP setting with a restrictive structure in which the observation is a d -dimensional vector and the latent state is a k -dimensional subset of the observed coordinates. This structure prevents the setting from subsuming the basic (non-exogenous) Block MDP framework, and makes it possible to learn decoders that act only on the endogenous state, preventing compounding errors.

Bellman completeness . Xie et al. [63] observed that ExBMDPs admit low coverability, but their algorithm requires Bellman completeness, which is not satisfied by ExBMDPs (see Efroni et al. [22], Islam et al. [30]).

Part I

Additional Results

This section of the appendix contains additional results omitted from the main body due to space constraints.

B Applying RVFS to Exogenous Block MDPs

We now apply RVFS to the Exogenous Block MDP (ExBMDP) model introduced in Section 3.3. ExBMDPs satisfy coverability (Assumption 3.2), but do not satisfy the pushforward coverability assumption (Assumption 4.1) required by RVFS in general. However, it turns out that ExBMDPs do satisfy pushforward coverability when the exogenous noise process is weakly correlated across time; we refer to this new statistical assumption as the weak correlation condition .

Assumption B.1 (Weak correlation condition) . For the underlying ExBMDP M , there is a constant C exo ≥ 1 such that for all h ∈ [ H -1 ] and ( ξ, ξ ′ ) ∈ Ξ h -1 × Ξ h , we have 7

The weak correlation property asserts that the joint law for the exogenous noise variables ξ h and ξ h + 1 is at most a multiplicative factor C exo ≥ 1 larger than the corresponding product distribution obtained by sampling ξ h and ξ h + 1 independently from their marginals. This setting strictly generalizes the (non-exogenous) Block MDP model [36, 15, 43, 68, 42], by allowing for arbitrary stochastic dynamics for the endogenous state and an arbitrary emission process, but requires that temporal correlations in the exogenous noise decay over time.

We show that under Assumption B.1, pushforward coverability is satisfied with C push ≤ C exo ⋅ SA (Lemma K.3 in Appendix K.1). In addition, V ⋆ -realizability is implied by decoder realizability (Lemma D.1). Thus, by applying Theorem 4.1 ( Setup I ), we conclude that RVFS efficiently learns a near-optimal policy for any weakly correlated ExBMDP for which the optimal value function has ∆ -gap.

An improved algorithm for ExBMDPs: RVFS exo . At first glance, removing the gap assumption for RVFS in ExBMDPs seems difficult: The V π -realizability assumption required to invoke Theorem 4.1 ( Setup II ) is not satisfied by ExBMDPs, as decoder realizability only implies V π realizability for endogenous policies π . 8 In spite of this, we now show that with a slight modification, RVFS can efficiently learn any weakly correlated ExBMDP under decoder realizability alone (without gap or V π -realizability).

Our new variant of RVFS , RVFS exo , is presented in Algorithm 8 (deferred to Appendix F for space). The algorithm is almost identical to RVFS (Algorithm 5), with the main difference being that we use an additional randomized rounding step to compute the policies ̂ π 1 ∶ H from the learned value functions ̂ V 1 ∶ H . In particular, instead of directly defining the policies ̂ π 1 ∶ H based on the bellman backups P h [ ̂ V h + 1 ] as in Eq. (14), RVFS exo targets a 'rounded' version of the backup given by

where ε ∈ ( 0 , 1 ) is a rounding parameter and ζ 1 , . . . , ζ H are i.i.d. random variables sampled uniformly at random from the interval [ 0 , 1 / 2 ] (at the beginning of the algorithm's execution). Concretely, RVFS exo estimates the bellman backup P h [ ̂ V h + 1 ]( x, a ) in Eq. (3) using the local simulator (as in Eq. (14) of Algorithm 5), and defines its policies via

This rounding scheme, which quantizes the Bellman backup into ε -1 bins with a random offset, is designed to emulate certain properties implied by the ∆ -gap assumption (Assumption 4.4). Specifically, we show that with constant probability over the draw of ζ 1 ∶ H , the policy ̂ π in (4) 'snaps' on to an endogenous policy π . This means that for RVFS exo to succeed (with constant probability), it suffices to pass it a class V that realizes the value functions ( V π h ) for endogenous policies π ∈ Π S . Fortunately, such a function class can be constructed explicitly under decoder realizability (Assumption 3.3).

Lemma B.1 ([21]) . For the ExBMDP setting, under Assumption 3.3, the function class V h ∶= { x ↦ f ( ϕ ( x )) ∶ f ∈ [ 0 , H ] S , ϕ ∈ Φ } is such that V π h ∈ V h for all endogenous policies π . Furthermore, the policy class Π h ∶= { π (⋅) ∈ arg max a ∈A f ( ϕ (⋅) , a ) ∶ f ∈ [ 0 , H ] S × A , ϕ ∈ Φ } contains all endogenous policies.

A small technical challenge with the scheme above is that it is only guaranteed to succeed with constant probability over the draw of the rounding parameters ζ 1 , . . . , ζ H . To address this, we provide an outer-level algorithm, RVFS exo . bc (Algorithm 9, deferred to Appendix F for space), which performs confidence boosting by invoking RVFS exo multiple times independently, and extracts a high-quality executable policy using behavior cloning.

Main result. We now state the main guarantee for RVFS exo (the proof is in Appendix K).

Theorem B.1 (Main guarantee of RVFS exo for EXBMDPs) . Consider the ExBMDP setting. Suppose the decoder class Φ satisfies Assumption 3.3, and that Assumption B.1 holds with C exo > 0 . Let ε, δ ∈ ( 0 , 1 ) be given, and let V h and Π h be as in Lemma B.1. Then RVFS exo . bc ( Π , V 1 ∶ H , ε, ζ 1 ∶ H , δ ) (Algorithm 9) produces a policy ̂ π 1 ∶ H such that J ( π ⋆ )-J (̂ π 1 ∶ H ) ≤ ε , and has total sample complexity

This result shows for the first time that sample- and computationally-efficient learning is possible for ExBMDPs beyond deterministic or factored settings [22, 21].

We mention in passing that our use of randomized rounding to emulate certain consequences of the ∆ -gap assumption leverages the fact that ExBMDPs have a finite number of (endogenous) latent states. It is unclear if this technique can be used when the (latent) state space is large or infinite.

Algorithm 3 RVFS exo h : Recursive Value Function Search for Exogenous Block MDPs

1: parameters: Value function class V , suboptimality ε ∈ ( 0 , 1 ) , seeds ζ 1 ∶ H ∈ ( 0 , 1 ) , confidence δ ∈ ( 0 , 1 ) .

2: input: Level h ∈ [ 0 .. H ] , value function estimates ̂ V h + 1 ∶ H , confidence sets V ̂ h + 1 ∶ H , stateaction collections C h ∶ H , and buffers B h ∶ H , and counters t h ∶ H .

/*

Initialize parameters.

*/

4: Set N test ← 2 8 M 2 Hε -2 log ( 8 M 6 H 8 ε -2 δ -1 ) , N reg ← 2 8 M 2 ε -2 log ( 8 ∣V∣ HM 2 δ -1 ) .

3: Set M ←⌈ 8 ε -2 C exo SAH ⌉ .

5: Set N est ( k ) ← 2 N 2 reg log ( 8 AN reg Hk 3 / δ ) and δ ′ ← δ /( 4 M 7 N 2 test H 8 ∣V∣) .

/* Test the fit for the estimated value functions ̂ V h + 1 ∶ H at future layers. */

8: for ( x h -1 , a h -1 ) ∈ C h do

9: for layer ℓ = H,...,h + 1 do

10: for n = 1 , . . . , N test do

11:

∼

(⋅ ∣

h

h

1

h

1

-

-

Draw

x

T

x

, a

h

-

1

)

, then draw

x

ℓ

-

1

by rolling out with

̂

π

h

+

1

∶

H

, where

ℓ

.

*/

17:

( ̂

ℓ

1

-

τ

V ̂

∣( ̂

τ

H

,ε

2

∶

H

∶

V

,

,δ

,

18: go to line 8 .

/*

h

V

and

Re-fit build

a

new

̂

20: for ( x h -1 , a h -1 ) ∈ C h do

21: Set D h ( x h -1 , a h -1 ) ← ∅ .

22: for i = 1 , . . . , N reg do

25: Update D( x h -1 , a h -1 ) ← D( x h -1 , a h -1 ) ∪ {( x h , ̂ V h ( x h ))} .

24: For each a ∈ A , let ̂ V h ( x h ) be a Monte-Carlo estimate for E ̂ π h ∶ H [∑ H ℓ = h r ℓ ∣ x h ] computed by collecting N est (∣C h ∣) trajectories starting from x h and rolling out with ̂ π h ∶ H .

26: Let ̂ V h ∶= arg min f ∈V h ∑ ( x h -1 ,a h -1 )∈C h ∑ ( x h ,v h )∈D h ( x h -1 ,a h -1 ) ( f ( x h ) -v h ) 2 .

27: Compute value function confidence set

28: return ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) .

confidence

′

C

[ ̂

τ

V

ℓ

∶

H

] - ̂

,

B

τ

∶

H

ℓ

1

-

, t

,ε

τ

∶

set.

2

H

,δ

[

f

′

*/

ℓ

])(

ℓ

-

x

RVFS exo

τ

1

, a

V

ℓ

τ

( ̂

-

+

1

∶

1

)∣ >

H

,

ε

τ

V ̂

2

+

+

ε

∶

H

2

,

⋅

C

β

τ

∶

(

H

1

t

,

ℓ

)

B

then

τ

∶

H

, t

τ

∶

H

;

V

, ε, ζ

∶

1

H

, δ

)

.

C Helper Lemmas

This section of the appendix contains supporting lemmas used within the proofs of our main results.

C.1 Concentration and Probability

Lemma C.1. Let δ ∈ ( 0 , 1 ) and H ≥ 1 be given. If a sequence of events E 1 , . . . , E H satisfies P [E h ∣ E 1 , . . . , E h -1 ] ≥ 1 -δ / H for all h ∈ [ H ] , then

Proof of Lemma C.1. By the chain rule, we have

We make use of the following version of Freedman's inequality, due to Agarwal et al. [1, Lemma 9]:

Lemma C.2. Let R > 0 be given and let w 1 , . . . w n be a sequence of real-valued random variables adapted to filtration H 1 , /uni22EF , H n . Assume that for all t ∈ [ n ] , w i ≤ R and E [ w i ∣H i -1 ] = 0 . Define S n ∶= ∑ n t = 1 w i and V n ∶= ∑ n t = 1 E [ w 2 i ∣ H i -1 ] . Then, for any δ ∈ ( 0 , 1 ) and λ ∈ [ 0 , 1 / R ] , with probability at least 1 -δ ,

We will also use the following lemma, which is a standard consequence of Freedman's inequality.

Lemma C.3 (e.g., Foster et al. [25]) . Let ( w t ) t ≤ T be a sequence of random variables adapted to a filtration (H t ) t ≤ T . If 0 ≤ w t ≤ R almost surely, then with probability at least 1 -δ ,

and

C.2 Regression

Using Lemmas C.2 and C.3, we obtain the following concentration lemma, which will be used to prove guarantees for square loss regression within our algorithms.

Lemma C.4. Let B > 0 and n ∈ N be given, and let Y be an abstract set. Further, let Q⊆{ g ∶ Y → [ 0 , B ]} be a finite function class and y 1 , . . . , y n be a sequence of random variables in Y adapted to filtration a H 1 , /uni22EF , H n . Then, for any δ ∈ ( 0 , 1 ) , with probability at least 1 -δ , we have

where ∥ g ∥ 2 ∶= ∑ i ∈[ n ] E [ g ( y i ) 2 ∣H i -1 ] and ∥ g ∥ 2 n ∶= ∑ n i = 1 g ( y i ) 2 .

Proof of Lemma C.4. Fix g ∈ Q . Applying Lemma C.2 with w i = g ( y i ) 2 -E [ g ( y i ) 2 ∣H i -1 ] , for all i ∈ [ n ] , and ( R,λ ) = ( B 2 , 1 / B 2 ) , we get that with probability at least 1 -δ /( 2 ∣Q∣) :

By substituting λ = B -2 and rearranging, we get

Similarly, applying Lemma C.2 with w i = E [ g ( y i ) 2 ∣H i -1 ] -g ( y i ) 2 , for all i ∈ [ n ] , and ( R,λ ) = ( B 2 , 1 /( 2 B 2 )) , we get that with probability at least 1 -δ /( 2 ∣Q∣) :

By substituting λ = 2 -1 B -2 and rearranging, we get

Combining this with (6) and the union bound, we get the desired result.

With this lemma, we now prove the following key result for square loss regression.

Lemma C.5 (Generic regression guarantee) . Let B > 0 and n ∈ N be given and Y be an abstract set. Further, let F ⊆ { f ∶ Y → [ 0 , B ]} be a finite function class, and suppose that there is a function f ⋆ ∈ F and a sequence of random variables ( y 1 , x 1 ) , . . . , ( y n , x n ) ∈ Y × R such that for all i ∈ [ n ] :

x i = f ⋆ ( y i ) + ε i + b i ;

∣ b i ∣ ≤ ξ ;

ε i ∈ [-B,B ] ; and

E [ ε i ∣ F i ] = 0 , where F i ∶= σ ( y 1 ∶ i , ε 1 ∶ i -1 , x 1 ∶ i -1 , b 1 ∶ i -1 ) .

Then, for ̂ f ∈ arg min f ∈F ∑ n i = 1 ( f ( y i ) -x i ) 2 and any δ ∈ ( 0 , 1 ) , with probability at least 1 -δ / 2 ,

where ∥ ̂ f -f ⋆ ∥ 2 n ∶= ∑ n i = 1 ( ̂ f ( y i ) -f ⋆ ( y i )) 2 .

Proof of Lemma C.5. Fix δ ∈ ( 0 , 1 ) and let ̂ L n ( f ) ∶= ∑ n i = 1 ( f ( y i ) -x i ) 2 , for f ∈ F , and note that since ̂ f ∈ arg min f ∈F ̂ L n ( f ) , we have

where ∇ denotes directional derivative. Rearranging, we get that

Bounding Term I. To bound Term I, we apply Lemma C.2 with w i = ε i ⋅ ( ̂ f ( y i ) -f ⋆ ( y i )) , R = B 2 , λ = 1 /( 8 B 2 ) , and H i = F -i + 1 , and use

the union bound over f ∈ F ; and

to get that with probability at least 1 -δ / 2 ,

By rearranging, we get that with probability at least 1 -δ / 2 ,

Term I

4

B

2

log

(

2

∣F∣/

δ

)

.

Bounding Term II. We now bound the second term in (7). For this, note that since ∥ ̂ f -f ⋆ ∥ ∞ ≤ B , we have

This completes the proof.

C.3 Reinforcement Learning

Lemma C.6 (Performance Difference Lemma (e.g., Kakade [33])) . For any two policies ̂ π, π ∈ Π S and t ∈ [ H ] , we have

In particular, applying this for t = 1 gives

Lemma C.7 (Potential lemma [63]) . Fix h ∈ [ H ] . Suppose we have a sequence of functions g ( 1 ) , . . . , g ( T ) ∈ [ 0 , B ] and policies π ( 1 ) , . . . , π ( T ) such that

for some β ≥ 0 . Then under Assumption 4.1, we have

and consequently

Proof of Lemma C.8. See proof of [63, Theorem 1].

The following result is a variant of the coverability-based potential argument given in Xie et al. [63].

Lemma C.8 (Pushforward coverability potential lemma) . Fix h ∈ [ H ] . Suppose we have a sequence of functions g ( 1 ) , . . . , g ( T ) ∈ [ 0 , B ] and state-action pairs ( x ( 1 ) , a ( 1 ) ) , . . . , ( x ( T ) , a ( T ) ) such that

for some β ≥ 0 . Then under Assumption 4.1, we have

and consequently

Proof of Lemma C.8. Define d ( t ) h ( x ) ∶= P [ x h = x ∣ x h -1 = x ( t ) , a h -1 = a ( t ) ] , and let ̃ d ( t ) h ∶= ∑ i < t d ( i ) Let

We have

From the definition of pushforward coverability, we can bound

. h

For the other term, we bound

Finally, we have

where the last line uses Lemma 4 of Xie et al. [63].

Part II

Proofs for SimGolf (Section 3)

As described in Section 3.1, the main difference between SimGolf and GOLF lies in the construction of the confidence sets. The most important new step in the proof of Theorem 3.1 is to show that the local simulator-based confidence set construction in Line 9 is valid in the sense that the property Eq. (1) holds with high probability. From here, the sample complexity bound follows by adapting the change-of-measure argument based on coverability from Xie et al. [63].

To this end, this part of the appendix is organized as follows. We first state and prove technical lemmas concerning realizability (Lemma D.1) and the confidence set construction (Lemma D.2 and Lemma D.3) in Appendix D. Then, in Appendix E, we prove Theorem 3.1 as a consequence.

D Preliminary Lemmas for Proof of Theorem 3.1

For this section, we define

and

where ( x ( t ) h , a ( t ) h , r ( t,k ) h , x ( t,k ) h + 1 ) are as in Algorithm 1.

Lemma D.1 ([21]) . For the ExBMDP setting, under Assumption 3.3, the function class Q h ∶= {( x, a ) ↦ g ( ϕ ( x ) , a ) ∶ g ∈ [ 0 , H ] SA , ϕ ∈ Φ } satisfies Assumption 3.1 and has log ∣Q h ∣ = log ∣ Π h ∣ = ̃ O ( SA + log ∣ Φ ∣) . 9

Lemma D.2. With probability at least 1 -δ , for all h ∈ [ H ] , t ∈ [ N ] , and g ∈ Q ,

and

Proof of Lemma D.2. Let t ∈ [ N ] and h ∈ [ H ] be fixed. Let us denote z ( t ) h = {( r ( t,k ) h , x ( t,k ) h + 1 )} k ∈[ K ] . Define a filtration

where τ ( i ) is the trajectory generated in the i th iteration of Algorithm 1 (see Line 7). Fix g ∈ Q . Observe that ℓ ( i ) h ( g ) ∈ [ 0 , 4 ] , so Lemma C.3 ensures that with probability at least 1 -δ ,

and

By the AM-GM inequality, for all i < t , we can bound

and

We have

Since

E

[

r

(

i,k

h

+

a

∈A

g

h

+

1

(

x

(

i,k

h

+

)

1

, a

) ∣

x

h

=

x

(

i

h

,

a

h

=

a

(

i

h

] = T

h

[

g

h

+

1

](

x

(

i

h

,

a

(

i

h

)

so that

Combining these bounds with (8) and rearranging thus gives

and

Taking a union bound yields the result.

)

)

)

)

)

Lemma D.3. Define β stat = 16log ( 2 HN ∣Q∣ δ -1 ) . Suppose we set K ≥ 8 N β stat and β ≥ 2 β stat . Then with probability at least 1 -δ , for all t ∈ [ N ] and h ∈ H :

Q ⋆ ( t )

∈ Q . · All g ∈ Q ( t ) satisfy β.

Proof of Lemma D.3. Condition on the event in Lemma D.2. For any fixed t ∈ [ N ] and h ∈ [ H ] , we have that

where the first inequality uses that Q ⋆ h = T h [ Q ⋆ h + 1 ] and the second inequality uses our choice for K . It follows that Q ⋆ ∈ Q ( t ) as long as β ≥ 2 β stat .

To prove the second claim, we note that for all g ∈ Q ( t ) , by construction,

E Proof of Theorem 3.1

Proof of Theorem 3.1. From Lemma D.3, the parameter setting in the theorem statement ensures that with probability at least 1 -δ , for all t ∈ [ 2 .. N ] , Q ⋆ ∈ Q ( t ) , and all g ∈ Q ( t ) satisfy

for all h . Let us condition on this event going forward. First, note that since Q ⋆ ∈ Q ( t ) for all t ∈ [ 2 .. N ] , we have that

On the other hand, we have g ( t ) ∈ arg max g ∈Q ( t ) ∑ s < t max a ∈A g 1 ( x ( s ) 1 , a ) , and so since x ( 1 ) 1 , x ( 2 ) 1 , . . . are i.i.d. and any g ∈ Q ( t ) take values in [ 0 , H ] , we have that by Hoeffding's inequality, there is an event E of probability at least 1 -δ under which

This implies that under E , we have

where in the last inequality we have used (11) with f = g ( t ) . Thus, summing (12) for t = 2 , . . . N and using (10) gives that under E :

and so since J π H

On the other hand, using that g ( t ) H + 1 ≡ 0 , we get

and so, by the potential lemma (Lemma C.7) and (9), we have

Combining this with (13), we obtain that with probability at least 1 -2 δ ,

It follows that if N = ̃ O ( H 2 C cov β / ε 2 ) , then the policy

returned by SimGolf satisfies, with probability at least 1 -δ :

Sample complexity. We now bound the number of episodes. Note that that within an iteration t of SimGolf , the local simulator is called KH times to update the confidence set, where K ≤ N / log ( 2 HN ∣Q∣/ δ )) . Consequently, the total sample complexity is bounded by

Part III

Proofs for RVFS (Section 4)

F Full Version of RVFS

Algorithm 5 displays the full version of RVFS . Algorithm 6 contains an 'outer-level' wrapper for RVFS , RVFS.bc , which invokes RVFS and extracts an executable policy with imitation learning, and Algorithm 7 contains the subroutine used within Algorithm 5 to approximate Bellman backups for value functions using local simulator access. Additionally, we display the variant of RVFS for Exogenous Block MDPs, described in Appendix B, in Algorithms 8 and 9. Before diving into the proof, we first describe how the full version of the algorithm differs from the informal version presented in the main body in greater detail.

Differences between full version (Algorithm 5) and informal version (Algorithm 2) of RVFS . The main difference between Algorithm 2 and its full version in Algorithm 5 is that in the former we simply assume access to quantities involving conditional expectations such as:

The bellman backups P h [ ̂ V h + 1 ] , which are required to evaluate the actions of RVFS 's policies (see (2)), and to perform the tests in Line 8; and

The value functions E ̂ π h + 1 ∶ H [∑ H ℓ = h r τ ∣ x h = x, a h = a ] in Line 15, which are needed in the regression problem in Line 16.

These quantities are not available to the algorithm directly, but they can be estimated using the local simulator. This is reflected in the full version of RVFS in Algorithm 5.

Extracting policies from value functions. Let us briefly comment in more detail on how Algorithm 5 extracts the policy π ( t ) from the optimistic value function f ( t ) ∈ V at iteration t . From the Bellman equation, the ideal choice would be to set π ( t ) h ( x ) = arg max a ∈A P h [ f ( t ) h + 1 ]( x, a ) , but this requires knowledge of the transition distribution. Instead, given parameters ε, δ ∈ ( 0 , 1 ) , SimGolf invokes Algorithm 7 via π ( t ) h ( x ) ∈ arg max a ∈A ̂ P h,ε,δ [ f ( t ) h + 1 ]( x, a ) . The operator ̂ P h,ε,δ [ f ] (Algorithm 7), when given input ( x, a ) ∈ X × A and f h + 1 ∶ X → R , uses the local simulator to generate N sim ≥ 1 next states x ( 1 ) h + 1 , . . . , x ( N sim ) h + 1 i.i.d. ∼ T h (⋅ ∣ x, a ) to estimate the bellman back-up P h [ f h + 1 ] via 1 N sim ∑ N sim i = 1 ( r ( i ) h + f h + 1 ( x ( i ) h + 1 )) , where r ( 1 ) h , . . . , r ( N sim ) h i.i.d. ∼ R h ( x, a ) . The number of samples N sim in Algorithm 7 is set as a function of ( ε, δ ) such that with probability at least 1 -δ , ∣ ̂ P h,ε,δ [ f h + 1 ]( x, a ) - P h [ f h + 1 ]( x, a )∣ ≤ ε .

Invoking the algorithm. The base invocation of RVFS takes the form

Whenever this call returns, the greedy policy induced by ̂ V 1 ∶ H is guaranteed to be near-optimal. Naively, the policy induced by ̂ V 1 ∶ H is non-executable, and must be computed by invoking the local simulator through Line 14. To provide an end-to-end guarantee to learn an executable policy, the outer-level algorithm, RVFS.bc (Algorithm 6, invokes RVFS 0 , then extracts an executable policy from ̂ V 1 ∶ H using imitation learning.

Subsequent recursive calls take the form

For such a call, the arguments above are:

̂ V h + 1 ∶ H : Value function estimates for subsequent layers.

V ̂ h + 1 ∶ H : Value function confidence sets V ̂ h + 1 ∶ H ⊂ V h + 1 ∶ H , which are used in the test on Line 14 to quantify uncertainty on new state-action pairs and decide whether to expand the core-sets.

C h ∶ H : Core-sets for current and subsequent layers.

B h ∶ H : Buffers of tuples ( x h - 1 , a h - 1 , ̂ V h , V ̂ h , t h ) , which record relevant features of the algorithm's state whenever the test on Line 14 fails and a recursive call is performed.

t h ∶ H : Counters that track the number of times Algorithm 7 is called in the test on Line 14, which facilitate tuning of confidence parameters.

Importantly, the confidence sets V ̂ h + 1 ∶ H do not need to be explicitly maintained, and can be invoked implicitly whenever a regression oracle for the value function class is available (cf. discussion in Section 4). Likewise, the buffers B h ∶ H are only used in our analysis, and do not need to be explicitly maintained.

F.1 RVFS Pseudocode

Algorithm 5 RVFS h : Recursive Value Function Search

1: parameters: Value function class V , suboptimality ε ∈ ( 0 , 1 ) , confidence δ ∈ ( 0 , 1 ) .

2: input:

Level h ∈ { 0 , . . . , H } .

Value function estimates ̂ V h + 1 ∶ H , confidence sets V ̂ h + 1 ∶ H , state-action collections C h ∶ H , buffers B h ∶ H , and counters t h ∶ H .

/* Initialize parameters. */

3: Set M ←⌈ 8 ε -1 C push H ⌉ .

/* Test the fit for the estimated value functions ̂ V h + 1 ∶ H at future layers. */

8: for ( x h -1 , a h -1 ) ∈ C h do 9: for layer ℓ = H,...,h + 1 do 10: for n = 1 , . . . , N test do 11: Draw x h ∼ T h -1 (⋅ ∣ x h -1 , a h -1 ) , then draw x ℓ -1 by rolling out with ̂ π h ∶ H , where 10

∈ A

do

12:

13:

a

/* Number of times ̂ P ℓ -1 ,ε,δ ′ (Algorithm 7) is called in the test on Line

14.

*/

19: if h = 0 then return: ( ̂ V 1 ∶ H , ⋅ , ⋅ , ⋅ , ⋅) .

/* Re-fit ̂ V h and build a new confidence set. */

21: Set D h ( x h -1 , a h -1 ) ← ∅ .

22: for i = 1 , . . . , N reg do

24: Let ̂ V h ( x h ) be a Monte-Carlo estimate for E ̂ π h ∶ H [∑ H ℓ = h r ℓ ∣ x h ] computed by collecting N est (∣C h ∣) trajectories starting from x h and rolling out with ̂ π h ∶ H .

26: Let ̂ V h ∶= arg min f ∈ ̂ V ∑ (

x

h

-

1

,a

h

-

1

)∈C

h

∑

27: Compute value function confidence set

28: return ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) .

(

x

,v

)∈D

(

x

,a

)

(

f

(

x

h

) -

v

h

)

2

.

h

h

h

h

-

1

-

h

1

for

ℓ

-

1

Algorithm 6 RVFS.bc : Learn an executable policy with RVFS via behavior cloning.

1: input: Value function class V , policy class Π , suboptimality ε ∈ ( 0 , 1 ) , confidence δ ∈ ( 0 , 1 ) . /* Set parameters for RVFS. */

3: Set ̂ V 1 ∶ H ← arbitrary , V ̂ 1 ∶ H ←V , C 0 ∶ H ←∅ , B 0 ∶ H ←∅ , and t i ← 0 , for all i ∈ [ 0 .. H ] .

2: Set ε RVFS ← εH -1 / 48 .

/* Set parameters for BehaviorCloning. */ 4: Set M ←⌈ 8 ε -1 RVFS C push H ⌉ and N test ← 2 8 M 2 Hε -1 RVFS log ( 80 M 6 H 8 ε -2 RVFS δ -1 ) .

6: ( ̂ V 1 ∶ H , ⋅ , ⋅ , ⋅ , ⋅) ← RVFS 0 ( ̂ V 1 ∶ H , V ̂ 1 ∶ H , C 0 ∶ H , B 0 ∶ H , t 0 ∶ H ; V , N reg , N test , ε RVFS , δ / 10 )

5: N reg ← 2 8 M 2 ε -1 RVFS log ( 80 ∣V∣ 2 HM 2 δ -1 ) and δ ′ = δ 40 M 7 N 2 H 8 ∣V∣ . /* Get value functions from RVFS */

. /* Extract executable policy via BehaviorCloning algorithm for imitation learning. */

8: Compute ̂ π 1 ∶ H ← BehaviorCloning ( Π , ε, ̂ π RVFS 1 ∶ H , δ / 2 )

7: Define ̂ π RVFS h (⋅) ∈ arg max a ∈A ̂ P h,ε RVFS ,δ ′ [ ̂ V h + 1 ](⋅ , a ) .

9: return: ̂ π 1 ∶ H .

Algorithm 7 ̂ P h,ε,δ [ f ] : Estimate conditional expectation E [ r h + f ( x h + 1 ) ∣ x h = ⋅ , a h = ⋅] .

1: parameters: Layer h , suboptimality ε ∈ ( 0 , 1 ) , confidence δ ∈ ( 0 , 1 ) , target function f .

2: input: ( x, a ) ∈ X × A .

3: Set N sim ∶= 2log ( 1 / δ )/ ε 2 .

4: Set D ← ∅

5: for i = 1 , . . . , N sim do

6: Sample r h ∼ R h ( x, a ) and x h + 1 ∼ T h (⋅ ∣ x, a ) .

7: Update D ← D ∪ {( r h , x h + 1 )} .

8: return: N -1 sim ⋅ ∑ ( r,x )∈D ( r + f ( x )) .

// Uses local simulator access.

F.2 RVFS exo Pseudocode

28: return ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) .

G Organization

This remainder of Part III of the appendix contains the proofs for the main results concerning the RVFS algorithm (Theorem 4.1 and Theorem B.1).

First, in Appendix H we give a brief overview of the analysis and introduce a restricted set of benchmark policies which will be used throughout the proofs for Theorem 4.1 and Theorem B.1. The benchmark policy class is central to the regret decomposition for RVFS , and facilitates an analysis that does not require optimism.

In Appendix I, we prove Theorem 4.1 under Setup II ( V π -realizability). This constitutes the main technical development for Theorem 4.1. The central technical results proven here are Theorem I.1, Theorem I.2 and which generalize Theorem 4.1.

In Appendix J, we prove Theorem 4.1 under Setup I ( V ⋆ -realizability), as a straightforward consequence of the tools developed in Appendix I (Theorem I.1 and Theorem I.2).

Finally, in Appendix K, we prove Theorem B.1 (analysis of RVFS exo for the ExBMDP problem). This analysis has a similar structure to the proof of Theorem 4.1 under Setup II in Appendix I, and builds on the same analysis techniques, but requires specialized arguments due to extra technical challenges in the ExBMDP setting.

Appendix M gives a self-contained presentation of the BehaviorCloning algorithm for imitation learning, which is used within RVFS.bc and RVFS exo . bc .

H Overview of Analysis and Preliminaries

In this section, we present some notation, technical tools, and preliminary results we require for the analysis of RVFS in the settings we described in Section 4. We start by defining a set of restricted benchmark policies used in the regret decomposition for RVFS .

H.1 Overview of Analysis

In this section we give a brief overview of the analysis techniques behind Theorem 4.1 and Theorem B.1. We focus on Theorem 4.1 to begin.

Recall that RVFS is recursive in the sense that whenever the test in Line 14 fails for a layer h ∈ [ H ] , an recursive call RVFS h is initiated. Throughout the recursion, via the steps in Item 1 and Item 2, RVFS maintains the following invariant: whenever a call to RVFS h (an instance of RVFS initiated at layer h ) terminates, the confidence sets V ̂ h + 1 ∶ H that it outputs satisfy, with high probability:

In addition, RVFS h can only return if the value function tests in Line 14 (which involve the confidence sets V ̂ h + 1 ∶ H ) all succeed. From the invariant in (Inv1), it can be shown that the tests only succeed if the estimated value functions ̂ V h + 1 ∶ H satisfy:

where ε test > 0 is a parameter set by the algorithm. We use pushforward coverability to show that RVFS can only expand the core-sets C 1 ∶ H a polynomial number of times before the algorithm terminates and (Inv2) is satisfied.

The invariant in (Inv2) is useful because it ensures that the greedy policies

induced by the learned value functions ̂ V 1 ∶ H are near-optimal. To make this precise, recall that given parameters ε, δ ∈ ( 0 , 1 ) , the action ̂ π h ( x ) of RVFS 's policy at layer h and state x ∈ X , is given by

where ̂ V h + 1 is the estimated value function at layer h + 1 . The operator ̂ P h,ε,δ [ ̂ V h + 1 ] (Algorithm 7), when given input ( x, a ) ∈ X × A , ensures that probability at least 1 -δ , ∣ ̂ P h,ε,δ [ ̂ V h + 1 ]( x, a ) -

P h [ ̂ V h + 1 ]( x, a )∣ ≤ ε . Combining this with the invariant in (Inv2) and the fact that V ⋆ h ≡ P h [ V ⋆ h + 1 ] , one can see that with high probability (over the randomness in x h ∼ P ̂ π and ̂ P ), we have:

Analysis under Setup I. For Theorem 4.1 ( Setup I ), Eq. (18) together with the definition of ̂ π h in Eq. (17) and the gap assumption (Assumption 4.4) implies that if ε ≤ ∆ / 8 , we have that with high probability (over the randomness in x h ∼ P ̂ π and ̂ P ),

This suffices to show that ̂ π is near-optimal, since by the performance lemma [33], the suboptimality of ̂ π can be bounded as

This suffices to prove the performance bound in Theorem 4.1 under Setup I .

Analysis under Setup II. For Theorem 4.1 ( Setup II ), an immediate challenge in applying a similar analysis to Setup I is the lack of suboptimality gap ∆ , which makes it impossible to directly bound the probability that ̂ π ≠ π ⋆ in Eq. (19). To address this, we introduce a restricted benchmark policy class Π ε ⊂ Π S in Appendix H.2 below. The class Π ε ⊂ Π S is constructed such that that there exists a policy π ∈ Π ε that (i) is O ( ε ) -suboptimal, and (ii) emulates certain properties of a gap. Together, these properties facilitate analysis similar to Setup I . Overall, this argument is similar to the 'virtual policy iteration' analysis in Yin et al. [65].

Analysis of RVFS exo . The analysis of RVFS exo for ExBMDPs (Theorem B.1) uses the same idea as the analysis for Setup II , except that we can only realize V π for endogenous policies that act on ϕ ⋆ ( x h ) . To address this, we use the randomized rounding scheme in RVFS exo , and the crux of the proof is to show that with high probability, the rounded policies in RVFS exo 'snap' onto endogenous policies, facilitating an argument similar to Setup II .

Generalizing the analysis. We mention in passing that RVFS can be slightly modified to recover other existing sample complexity guarantees for RL with linear function approximation and local simulator access that do not require pushforward coverability, including linearQ ⋆ realizability with gap [40] and Q π -realizability [65]; we leave a more general treatment for future work.

H.2 Benchmark Policy Class and Randomized Policies

As described above, central to our analysis is a set of O ( ε ) -suboptimal policies against which we benchmark the policies returned RVFS , which emulate certain consequences of the ∆ -gap assumption (Assumption 4.4). Before introducing this concept formally, we first define the notion of a randomized policy .

Induced stochastic policies. Given an arbitrary collection of independent random variables ̃ Q = ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A , we say that a policy π is induced by ̃ Q if π satisfies

where we use the bold notation π h ( x ) as shorthand for the random variable a h ∼ π h ( x ) ∈ ∆ (A) ; in other words, for each x ∈ X , π h ( x ) ∈ ∆ (A) is the distribution induced by sampling Q h ( x, ⋅) and playing the action π h ( x ) ∈ arg max a ′ ∈A ̃ Q h ( x, a ′ ) . If there are ties in (20), we break them by picking the action with the smallest index; we assume without loss of generality that actions in A are index from 1 , . . . , ∣A∣ .

Benchmark policy class. We now define the benchmark policy class as follows.

Definition H.1 (Benchmark policy class) . For ε ∈ ( 0 , 1 ) , let Π ε ⊆ Π S be the set of stochastic policies such that π ∈ Π ε if and only if there exists a collection of independent random variables ̃ Q = ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A in [ 0 , H ] such that:

π is induced by ̃ Q (i.e. Eq. (20) is satisfied); and

For all ( h, x, a ) ∈ [ H ] × X × A , we have ∣( ̃ Q h -Q π h )( x, a )∣ ≤ ε , almost surely under the draw of ̃ Q .

Intuitively, the set Π ε contains the set of all (stochastic) policies corresponding to (randomized) state-action value functions that are point-wise O ( ε ) close to Q ⋆ . We formalize this claim in the next lemma.

Lemma H.1 (Suboptimality of benchmark policies) . Let ε ∈ ( 0 , 1 ) be given. Let ˜ π ∈ Π ε be a stochastic policy induced by a collection of (independent) random state-action value functions ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A ⊂ [ 0 , H ] such that for all h ∈ [ H ] and all ( x, a ) ∈ X × A :

Then, for all h ∈ [ H ] ,

Proof of Lemma H.1. Using backward induction over ℓ = H,..., 1 , we start by showing that all ℓ :

almost surely. We then instantiate this with ℓ = 1 and use the fact that ∥ ̃ Q h -Q ˜ π h ∥ ∞ ≤ ε to get the desired result.

Base case [ ℓ = H ] . By definition of the state-action value function, we have, for all π ∈ Π S , Q ⋆ H ≡ Q π H . Thus, since sup ( x,a )∈X×A ∣( ̃ Q H -Q ˜ π H )( x, a )∣ ≤ ε almost surely (by definition of ̃ Q 1 ∶ H ), we get that

almost surely. This implies (21) for ℓ = H .

General case [ ℓ < h ] . Fix h ∈ [ H -1 ] and suppose that (22) holds for all ℓ ∈ [ h + 1 , . . . , H ] almost surely. We show that it holds for ℓ = h almost surely. Fix ( x, a ) ∈ X × A . We have

almost surely, where the last step follows by the induction hypothesis. We now bound T h [ ̃ Q h + 1 ]( x, a ) - ̃ Q h ( x, a ) . We have, almost surely, that

where the penultimate inequality follows by the definition of ˜ π h + 1 , and the last inequality follows by the fact that ∥ ̃ Q h + 1 -Q ˜ π h + 1 ∥ ∞ ≤ ε almost surely, by assumption. Plugging (24) into (23) completes the induction, and so we have that

In particular, taking the max over a on both sides and using the definition of ˜ π , we get that

almost surely. Combining this with the fact that ̃ Q 1 ( x, ˜ π 1 ( x )) ≤ Q ˜ π 1 ( x, ˜ π 1 ( x )) + ε , almost surely (since ∥ ̃ Q 1 -Q ˜ π 1 ∥ ∞ ≤ ε almost surely by assumption) implies that

Taking expectation over ˜ π 1 ( x ) and bounding 2 Hε + ε by 3 Hε leads to the desired result.

H.3 Additional Preliminaries

The following lemma gives a guarantee for the Bellman backup approximation algorithm ̂ P (Algorithm 7) that is tailored to the analysis of RVFS .

Lemma H.2. Let ε, δ, δ ′ ∈ ( 0 , 1 ) , B > 0 , and h ∈ [ H ] , be given and let V be a finite function class. For any sequence ( x i ) i ≥ 1 ⊂ X of state action pairs, the outputs ( ̂ P h,ε,δ ′ [ f ]( x i , a )) i ≥ 1 ,a ∈A of Algorithm 7 satisfy, with probability at least 1 -δ ,

Proof of Lemma H.2. By Hoeffding's inequality [28] and the union bound over a ∈ A and f ∈ V , we have that for any i ≥ 1 , with probability at least 1 -δ /( 2 i 2 ) ,

The desired result follows by the union bound over i ≥ 1 and the fact that ∑ i ≥ 1 1 / i 2 = π 2 / 6 ≤ 2 .

I Guarantee under V π -Realizability (Proof of Theorem 4.1, Setup II)

In this section, we prove Theorem 4.1 under Setup II . First, in Appendix I.1 we state a number of supporting technical lemmas, then use them to prove a more general version of Theorem 4.1, Theorem I.2, which holds under a weaker realizability assumption (informally, V π -realizability only for near-optimal policies π ); Theorem 4.1 follows as an immediate consequence. The remainder of the section (Appendix I.2 through to Appendix I.6) contains the proofs for the intermediate results.

I.1 Analysis: Proof of Theorem 4.1 (Setup II)

We analyze RVFS in the setting of Theorem 4.1 ( Setup II ), where we have a function class V satisfying V π -realizability (Assumption 4.5). We will actually show that the conclusion of Theorem 4.1 holds under a weaker function approximation setup we refer to as relaxed V π -realizability : instead of requiring V π -realizability for all π ∈ Π S , we only require it for policies π in the set of near-optimal policies corresponding to the benchmark policy class Π ε real for some ε real > 0 ( Π ε is defined in Appendix H).

Assumption I.1 (Relaxed V π -realizability) . For ε real > 0 and all π ∈ Π ε real and h ∈ [ H ] , we have V π h ( x ) ∈ V ⊆ { f ∶ X → [ 0 , H ]} .

We will analyze RVFS under Assumption I.1 and Assumption 4.1. However, it turns out that all of the main results for RVFS can be derived under this assumption: As we will see in Appendix J in the sequel, when the ∆ -gap assumption (Assumption 4.4) is satisfied, then Π ε real = { π ⋆ } for all ε real < ∆ , allowing us to prove Theorem 4.1 under Setup I as a special case of relaxed V π -realizability. Our analysis for the ExBMDP setting in Appendix K requires more work, but uses that for ExBMDPs, Assumption I.1 is satisfied for a subset of Π ε real corresponding to endogenous policies.

We begin with our analysis under Setup II by bounding the number of times the test in Line 14 fails. Since the sizes of the core sets C 1 ∶ H in RVFS are directly related to the number of test failures, the next result, which bounds ∣C h ∣ for h ∈ [ H ] , allows us to show that RVFS terminates in polynomial iterations with high probability. The proof is in Appendix I.2.

Lemma I.1 (Bounding the number of test failures) . Let δ, ε ∈ ( 0 , 1 ) be given, and suppose that Assumption 4.1 (pushforward coverability) holds with parameter C push > 0 . Further, let f ∈ V , be given, where V is an arbitrary function class. Then there is an event E of probability at least 1 -δ under which any call RVFS 0 ( f, V 1 ∶ H , ∅ , ∅ , 0; V , ε, δ ) (Algorithm 5) terminates, and throughout the execution of RVFS 0 , we have

In particular, Lemma I.1 ensures that with high probability, every call to RVFS h (made recursively via the call to RVFS 0 ) terminates in polynomial iterations. When RVFS h terminates, all the tests in Line 14 must have passed for all ℓ > h . Using this and a standard concentration argument, we get the following guarantee for the estimated value functions and confidence sets returned by each call to RVFS h . The proof is in Appendix I.3.

Lemma I.2 (Consequence of passing the tests) . Let h ∈ [ 0 .. H ] and ε, δ ∈ ( 0 , 1 ) be given and consider a call to RVFS 0 in the setting of Lemma I.1. Further, let E be the event of Lemma I.1. There exists an event E ′ h of probability at least 1 -δ / H such that under E ∩ E ′ h , if a call to RVFS h within the execution of RVFS 0 terminates and returns ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) , then for any ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

where (̂ π τ ) τ ≥ h is the stochastic policy induced by ̂ V h ∶ H and M and N test are defined as in Algorithm 5. Furthermore, under the event E , the total number of times the operator ̂ P is called in the test of Line 14 of Algorithm 5 is at most O ( C push N test H 4 ε -1 ) .

We now give a guarantee for the estimated value functions ̂ V 1 ∶ H computed within RVFS in Line 26 (the proof is in Appendix I.4).

Lemma I.3 (Value function regression guarantee) . Let h ∈ [ 0 .. H ] and δ, ε ′ ∈ ( 0 , 1 ) be given, and consider a call to RVFS 0 in the setting of Lemma I.1. Further, let Π ′ ⊆ Π S be a finite policy class such that the class V realizes the value functions V π for π ∈ Π ′ (i.e. V satisfies Assumption I.1 with Π ε real replaced by Π ′ ). Then, there is an event E ′′ h of probability at least 1 -δ / H under which for all k ≥ 1 , if

RVFS h gets called for the k th time during the execution of RVFS 0 ; and

this k th call terminates and returns ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) ,

then if (̂ π τ ) τ ≥ h is the policy induced by ̂ V h ∶ H and N reg is set as in Algorithm 5, we have that for all π ∈ Π ′ ,

where the datasets {D h ( x, a ) ∶ ( x, a ) ∈ C h } are as in the definition of V ̂ h in (15).

Next, we use this result to show that the confidence sets V ̂ 1 ∶ H returned by RVFS 0 are 'valid' in the sense that they contain a value function ( V ˜ π h ) corresponding to a near-optimal stochastic policy ˜ π in the benchmark class Π 4 ε . In the sequel, we use this fact to substitute V ˜ π ℓ for f ℓ in Eq. (26) and bound the suboptimality of the learned policy ̂ π .

Lemma I.4 (Confidence sets) . Let ε, δ ∈ ( 0 , 1 ) be given and suppose that Assumption 4.1 (pushforward coverability) holds with parameter C push > 0 . Let f ∈ V be arbitrary, and suppose that V satisfies Assumption I.1 with ε real = 4 ε . Then, there is an event E ′′′ of probability at least 1 -3 δ under which a call to RVFS 0 ( f, V , ∅ , ∅ , 0; V , ε, δ ) (Algorithm 5) terminates and returns tuple ( ̂ V 1 ∶ H , V ̂ 1 ∶ H , C 1 ∶ H , B 1 ∶ H , t 1 ∶ H ) such that

where ˜ π 1 ∶ H ∈ Π S is the stochastic policy defined recursively via

where ̂ Q τ ( x, a ) ∶= ̂ P τ,ε,δ ′ [ ̂ V τ + 1 ]( x, a ) is a realization of the stochastic output of the ̂ P operator in Algorithm 7 given input ( x, a ) , and δ ′ is as in Algorithm 5. Furtherore, we have ˜ π ∈ Π 4 ε .

The proof of the lemma is in Appendix I.5.

Equipped with the preceding lemmas, we now state the main technical result of this section, Theorem I.1, a generalization of Theorem 4.1 which holds under relaxed V π -realizability (Assumption I.1). The proof is in Appendix I.6.

Theorem I.1 (Guarantee for RVFS under relaxed V π -realizability) . Let δ, ε ∈ ( 0 , 1 ) be given, and suppose that Assumption 4.1 (pushforward coverability) holds with parameter C push > 0 . Let f ∈ V be arbitrary, and assume that V that satisfies Assumption I.1 with ε real = 4 ε . Then, with probability at least 1 -5 δ , RVFS 0 ( f, V , ∅ , ∅ , 0; V , ε, δ ) (Algorithm 5) terminates and returns value functions ̂ V 1 ∶ H that satisfy

where ̂ π h ( x ) ∈ arg max a ∈A ̂ P h,ε,δ ′ [ ̂ V h + 1 ]( x, a ) for all h ∈ [ H ] , with ˜ π ∈ Π 4 ε defined as in Lemma I.4 and δ ′ defined as in Algorithm 5. Furthermore, the number of episodes is bounded by

Next, we state a guarantee for the outer-level algorithm, RVFS.bc , under relaxed V π -realizability. Recall that RVFS.bc invokes RVFS 0 , then extracts an executable policy by applying the BehaviorCloning algorithm (see Appendix M), with the 'expert' policy set to be the output of RVFS .

Theorem I.2 (Main guarantee of RVFS.bc ) . Let δ, ε ∈ ( 0 , 1 ) be given, and define ε RVFS = εH -1 / 48 . Suppose that

Assumption 4.1 (pushforward coverability) holds with parameter C push > 0 ;

the function class V satisfies Assumption I.1 with ε real = 1 (i.e. all π -realizability); and

the policy class Π satisfies Assumption 4.3.

Then, with probability at least 1 -δ , ̂ π 1 ∶ H = RVFS.bc ( Π , V , ε, δ ) (Algorithm 6) satisfies

Furthermore, the total number sample complexity in the RLLS framework is bounded by

The proof is in Appendix I.7. Note that Theorem I.2 is a restatement of Theorem 4.1 in Setup II (restated for convenience). As a result, Theorem 4.1 is an immediate corollary.

Proof of Theorem 4.1. The result follows from Theorem I.2, since Assumption 4.5 is stronger than Assumption I.1.

I.2 Proof of Lemma I.1 (Number of Test Failures)

Proof of Lemma I.1. Fix h ∈ [ H ] . We note that the size of C h corresponds to the number of times the test in Line 14 fails for ℓ = h throughout the execution of RVFS 0 ( f, V , ∅ , ∅ ; V , ε, δ ) .

Let M ∶= ⌈ 8 ε -1 C push H ⌉ denote the desired upper bound on ∣C h ∣ . Suppose that the test in Line 14 fails at least twice for ℓ = h (if the test fails at most twice, then ∣C h ∣ ≤ 2 and so (25) holds for ℓ = h trivially), and let

denote the elements of the set B h in the order at which they are added to the latter in Line 15 of Algorithm 5. Note that ∣B h ∣ = ∣C h ∣ . Note also that t ( i ) h represents the number of times the subroutine ̂ P h -1 ,ε,δ ′ has been called in the test of Line 14 throughout the execution of RVFS 0 and up to the time the test failed for ( x ( i ) h -1 , a ( i ) h -1 ) . We will use this fact in a concentration argument in the sequel.

By definition of ( ̂ V ( i ) h ) and Lemma C.4 (Freedman's inequality) instantiated with

Q={ ̂ V ( i ) h -f h ∶ f ∈ ̂ V ( i ) h } ;

y h = x h ;

B = H ; and

and the union bound over i ∈ [ M ∧ ∣C h ∣] , we get that there is an event E h of probability at least 1 -δ /( 2 H ) under which

Now, define f ( i ) h ∈ arg max f ∈ ̂ V ( i ) h ∣ E [ ̂ V ( i ) h ( x h ) -f h ( x h ) ∣ x h -1 = x ( i ) h -1 , a h -1 = a ( i ) h -1 ]∣ . From (29), we have that under E h :

We now use this to bound the number of times the test in Line 14 fails for ℓ = h . Suppose for the sake of contradiction that the test fails at least N times for some N ≥ M (i.e. ∣C h ∣ = N ≥ M ). Conditioned on E h , we have by Lemma C.8 and Eq. (30),

Now, substituting the expression of ˜ ε 2 reg in (29) and using the definition of ε 2 reg in Line 6 of Algorithm 5, we get

where the last inequality uses that M = ⌈ 8 ε -1 C push H ⌉ and

see Line 5 of Algorithm 5.

On the other hand, by Lemma H.2, there is an event E ′ h of probability at least 1 -δ /( 2 MH ) under which for all f ∈ V , all i ∈ [ M ] , and δ ′ as in Algorithm 5:

where β ( t ( i ) h ) is as in Algorithm 5. Thus, under E ′ h , the test in Line 14 fails for ℓ = h at least M times only if

Unless N < M , this is a contradiction to Eq. (31). We conclude that under the event E h ∩E ′ h , the test in Line 14 fails at most N < M = ⌈ 8 ε -1 C push H ⌉ times for ℓ = h , and so under E 1 ∩E ′ 1 ∩⋅⋅ ⋅ ∩ E H ∩E H , we have

By the union bound, we have P [E 1 ∩E ′ 1 ∩⋅⋅⋅ ∩ E H ∩E ′ H ] ≥ 1 -δ , which completes the proof.

I.3 Proof of Lemma I.2 (Consequence of Passing the Tests)

Proof of Lemma I.2. Let h ∈ [ H ] be given. Fix ℓ ∈ [ h + 1 .. H ] and let x ( 1 ) ℓ -1 , x ( 2 ) ℓ -1 , . . . denote the sequence of states used in the tests of Line 14 throughout the execution of RVFS 0 ; we assume that the sequence is ordered in the sense that if i < j , then x ( i ) ℓ -1 is used in the test of Line 14 before x ( j ) ℓ -1 . Let T ℓ ∈ N ∪{+∞} be the random variable representing the total number of times the operator ̂ P ℓ -1 ,ε,δ ′ is invoked in Line 14 throughout the execution of RVFS 0 ( T ℓ is also the random length of the sequence x ( 1 ) ℓ -1 , x ( 2 ) ℓ -1 , . . . ; if RVFS 0 terminates, then T ℓ is finite. The first step of the proof will be to show that under the event E of Lemma I.1, T ℓ is no larger than M 3 N test H 3 at any point during the execution of RVFS 0 . This will help us establish key concentration results, leading to the desired inequality (26).

Bounding T ℓ under E . First, note that under the event E of Lemma I.1, we have that for any τ ∈ [ H ] ,

and so RVFS τ gets called at most M times throughout the execution of RVFS 0 . For the rest of this paragraph, we condition on E and fix τ ∈ [ 0 .. H ] . Within any given call to RVFS τ (throughout the execution of RVFS 0 ), the operator ̂ P ℓ -1 ,ε,δ ′ is invoked at most

Due to the for-loops in Line 8, Line 9, & Line 10

Number of times RVFS returns to Line 8 (see below)

times. This is because the for-loop in Line 8 of RVFS τ resumes whenever a test in Line 14 fails for one of the layers τ + 1 , . . . , H (see Line 18) once the recursive calls return, and the total number of test failures across all these layers is bounded by HM (by (33)). Now, since RVFS τ gets called at most M times throughout the execution of RVFS 0 (as argued in the prequel), the total number of times the operator ̂ P ℓ -1 ,ε,δ ′ is invoked in Line 14 within RVFS τ is at most

Finally, the total number of times the operator ̂ P ℓ -1 ,ε,δ ′ is called in Line 14 throughout the execution of RVFS 0 is at most H times larger (accounting for the contributions from RVFS τ for all τ ∈ [ H ] ); that is, it is at most M 3 N test H 3 . We conclude that the random variable T ℓ satisfies

under E .

Specifying E ′ h . In this paragraph, we no longer condition on E . We will specify the event E ′ h in the lemma statement. Let δ ′ be defined as in Algorithm 5. By Lemma H.2, we have that there is an event E ′ h,ℓ of probability at least 1 -δ /( 2 H 2 ) under which:

On the other hand, for k ∈ [ T ℓ -N test + 1 ] , we have by Lemma C.4 (Freedman's inequality) instantiated with

n = N test and y i = I { sup f ∈ ̂ V ℓ max a ∈A ∣(P ℓ -1 [ ̂ V ℓ ] - P ℓ -1 )[ f ℓ ]( x ( k + i ) ℓ -1 , a )∣ > 3 ε } , for all i ∈ [ N test ] ;

Q={ id } ;

B = 1 ; and

λ = 1 ;

that there is an event E ′′ h,ℓ,k of probability at least 1 -δ /( 4 k 2 H 2 ) under which

Now, let E ′′ h,ℓ ∶= ⋂ k ∈[ T ℓ -N test + 1 ] E ′′ h,ℓ,k . By the union bound and the fact that ∑ k ≥ 1 1 / k 2 = π 2 / 6 ≤ 2 , we have that P [E ′′ h,ℓ ] ≥ 1 -δ /( 2 H 2 ) . Furthermore, under E ′′ h,ℓ , we have

We define E ′ h ∶= E ′ h, 1 ∩E ′′ h, 1 ∩⋅⋅ ⋅ ∩ E ′ h,H ∩E ′′ h,H . Note that by the union bound, we have P [E ′ h ] ≥ 1 -δ H as desired.

Termination of RVFS h under E ∩ E ′ h . We now show that under E ∩ E ′ h , if RVFS h terminates, its output satisfies (26). For the rest of the proof, we condition on E ∩E ′ h . Suppose that RVFS h terminates and returns ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) . In this case, the value function ̂ V ℓ must have passed the tests in Line 14 for all ( x h -1 , a h -1 ) ∈ C h , n ∈ N test , and a ℓ -1 ∈ A . Fix ( x h -1 , a h -1 ) ∈ C h and let k ∈ [ T ℓ -N test ⋅ A + 1 ] be such that ( x k + j ℓ -1 ) j ∈[ 0 ..N test -1 ] represents a subsequence of states that pass the tests in Line 14 at layer ℓ for ( x h -1 , a h -1 ) within the call to RVFS h . The fact that the sequence ( x ( i ) ℓ -1 ) i ≥ 1 is ordered (see definition in the first paragraph of this proof) and that ( x k + j ℓ -1 ) j ∈[ 0 ..N test -1 ] pass the tests imply that

The states ( x ( k + i ) ℓ -1 ) i ∈[ 0 ..N test -1 ] at layer ℓ -1 are i.i.d., and are obtained by rolling out with ̂ π h ∶ H starting from ( x h -1 , a h -1 ) ; and

The test in Line 14 succeeds for all ( x ( k + j ) ℓ -1 ) j ∈[ 0 ..N test -1 ] ; that is

This implies that

where (37) follows by (34) and the choice of δ ′ in Algorithm 5.

Now, by Item 1, we have that x ( k + i ) ℓ -1 has probability law P ̂ π h ∶ H [⋅ ∣ x h -1 = x h -1 , a h -1 = a h -1 ] for all i ∈ [ 0 .. N test -1 ] , and so by (36), we have:

Concluding. We have established that under E ∩ E ′ h , we have for all ℓ ∈ [ h + 1 .. H ] and all ( x h -1 , a h -1 ) ∈ C h :

Furthermore, we have P [E h ] ≥ 1 -δ / H

I.4 Proof of Lemma I.3 (Value Function Regression Guarantee)

Proof of Lemma I.3. Fix π ∈ Π ′ ⊆ Π S and k ≥ 1 , and consider the k th call to RVFS h as per the lemma statement, and let S k be the state of RVFS 0 during the k th call to RVFS h and immediately before Line 20, i.e. immediately before gathering data for the regression step in RVFS h .

Relating the regression targets to V ̂ π h . Observe that ̂ V h is the least-squares solution of the objective in Line 26, where the targets are empirical estimates of V ̂ π h . In particular, if we let {D h ( x, a ) ∶ ( x, a ) ∈ C h } be the datasets in the definition of V ̂ h in (15), then for any ( x h -1 , a h -1 ) ∈ C h and ( x h , v h ) ∈ D h ( x h -1 , a h -1 ) , the target v h satisfies

where ̂ V h ( x h ) is an empirical estimate of V ̂ π h ( x h ) obtained by sampling N est (∣C h ∣) = N est ( k ) episodes (for N est (⋅) defined as in Algorithm 5) by rolling out with ̂ π after starting from x h and playing action a at layer h ; note that ∣C h ∣ = k because we are considering the k th call to RVFS h . Thus, by Hoeffding's inequality and the union bound over ( x h -1 , a h -1 ) ∈ C h and ( x h , -) ∈ D h ( x h -1 , a h -1 ) , there is an event E ′′ h,k (S k ) of probability at least 1 -δ /( 8 k 2 H ) under which

where N reg is as in Line 5, and the last inequality follows by ∣C h ∣ ≤ k since we are considering the k th call to RVFS h . Thus, under E ′′ h,k , we have

where the second-to-last inequality is by (39) and the last inequality follows by the choice of N est in Algorithm 5.

Bounding the discrepancy V ̂ π h -V π h . On the other hand, by the performance difference lemma, the value function V ̂ π h satisfies:

Now, let ( x ( 1 ) h -1 , a ( 1 ) h -1 ) , ( x ( 2 ) h -1 , a ( 2 ) h -1 ) , . . . denote the elements of C h in the order in which they are added to the latter in Line 15. By Lemma C.2 (Freedman's inequality) instantiated with

n = N reg ⋅ k .

H i = σ ( x ( 1 ) h , . . . x ( i -1 ) h ) , for all i ∈ [ n ] ;

R = H ; and

λ = 1 / H ;

we get that there is an event E ̃ ′′ h,k,π (S k ) of probability at least 1 -δ /( 8 k 2 H ∣ Π ′ ∣) under which:

where the last inequality follows by (41) and the law of total expectation.

Regression guarantee. Since π ∈ Π ′ ⊆ Π S and Assumption I.1 holds, Lemma C.5 (regression guarantee) instantiated with

f ⋆ ( x ) = V π h ( x ) ;

B = H ;

ξ = H ;

implies that there is an event ˘ E ′′ h,k,π (S k ) of probability at least 1 -δ /( 4 k 2 H ∣ Π ′ ∣) under which we have:

where the last step follows by the triangle inequality. Thus, by plugging (42) and (40) into (43), we get that under E ′′ h,k (S k ) ∩ ̃ E ′′ h,k,π (S k ) ∩ ˘ E ′′ h,k,π (S k ) :

Applying the union bound to conclude. Let S k be the random state of RVFS 0 during the k th call to RVFS h and immediately before Line 20, i.e. immediately before gathering data for the regression step in RVFS h . Further, let S + k be the random state of RVFS 0 during the k th call to RVFS h and immediately before Line 26, i.e. immediately before the regression step in RVFS h . If RVFS 0 terminates before the k th call to RVFS h , we use the convention that S k = S + k = t , where t denotes a terminal state, and define E ′′ h,k ( t ) = ̃ E ′′ h,k,π ( t ) = ˘ E ′′ h,k,π ( t ) = { t } . Further, we define

Note that by the argument in the sequel and the union bound, we have that

where S k denotes the state of RVFS 0 during the k th call to RVFS h and immediately before Line 20. By letting S ′ 1 , S ′ 2 , . . . denote an identical, independent copy of the sequence S 1 , S 2 , . . . , we have by the chain rule:

where the last inequality follows from the fact that for any sequence x 1 , x 2 , ⋅ ⋅ ⋅ ∈ ( 0 , 1 ) , we have ∏ k ≥ 1 ( 1 -x k ) ≥ 1 - ∑ k ≥ 1 x k . Combining (46) with (44) implies that E ′′ h gives the desired result.

I.5 Proof of Lemma I.4 (Guarantee for Confidence Sets)

To prove Lemma I.4, we need one additional result pertaining to the order in which the instances ( RVFS h ) h ∈[ H ] are called.

Lemma I.5. Let h ∈ [ 0 .. H ] be given, and consider the setting of Lemma I.4. Further, consider a call to RVFS 0 ( f, V , ∅ , ∅ ; V , ε, δ ) that terminates, and let h ∈ [ H ] be any layer such that RVFS h is called during the execution of RVFS 0 . Then, after the last call to RVFS h terminates, no instance of RVFS in ( RVFS τ ) τ > h is called before RVFS 0 terminates.

Proof of Lemma I.5. Suppose there is an instance of RVFS in ( RVFS τ ) τ > h that is called after the last call to RVFS h terminates. Let τ > h be the lowest layer where RVFS τ is called after the last call to RVFS h terminates. Let RVFS last τ denote the corresponding instance of RVFS τ . Further, let ℓ < τ be such that RVFS ℓ is the parent instance of RVFS last τ (i.e. the instance that called RVFS last τ ). Note that we cannot have ℓ = h as this would imply that an instance of RVFS h terminates after RVFS last τ , and we have assumed that RVFS last τ terminates after the last call RVFS h . It is also not possible to have ℓ > h as this would imply that τ is not the lowest layer where RVFS τ is called after the last call to RVFS h terminates. Thus, we must have that ℓ < h . Now, the for-loop in Line 16 ensures that that there is an instance of RVFS h that is called after RVFS last τ terminates and before RVFS ℓ does. This contradicts the assumption that RVFS last τ is called after the last call to RVFS h .

Proof of Lemma I.4. We start by showing that ˜ π ∈ Π 4 ε by constructing the corresponding collection of random state-action value functions ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A ⊂ [ 0 , H ] in the definition of Π 4 ε . In particular, for ( h, x, a ) ∈ [ H ] × X × A , we define

where ̂ Q τ ( x, a ) ∶= ̂ P τ,ε,δ ′ [ ̂ V τ + 1 ]( x, a ) . Note that ̃ Q h ( x, a ) only depends on the randomness of ̂ P h,ε,δ ′ [ ̂ V h + 1 ]( x, a ) , and so ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A are independent random variables. Furthermore, since P h [ V ˜ π h + 1 ] ≡ Q ˜ π h , we have that

Finally, since ˜ π h (⋅) ∈ arg max a ∈A ̃ Q h (⋅ , a ) , we have that ˜ π ∈ Π 4 ε .

Weshow V ˜ π h ∈ ̂ V h . We prove that V ˜ π h ∈ ̂ V h , for all h ∈ [ H ] , under the event E ′′′ ∶= E ∩E ′ 1 ∩E ′′ 1 ∩⋅⋅ ⋅∩ E ′ H ∩E ′′ H , where E , (E ′ h ) , and (E ′′ h ) are the events defined in Lemma I.1, Lemma I.2, and Lemma I.3, respectively. Throughout, we condition on E ′′′ . First, note that by Lemma I.1, RVFS 0 terminates. Let ( ̂ V 1 ∶ H , V ̂ 1 ∶ H , C 1 ∶ H , B 1 ∶ H , t 1 ∶ H ) be the tuple it returns.

We will show via backwards induction over ℓ = H + 1 , . . . , 1 , that

where ˜ π 1 ∶ H is the stochastic policy defined recursively via

Base case [ ℓ = H + 1 ]. This holds trivially because V π H + 1 ≡ 0 for any π ∈ Π S by convention.

( ) ∶= ̂ P [ ̂ ]( )

General case [ ℓ ≤ H ]. Fix h ∈ [ H ] and suppose that (47) holds for all ℓ ∈ [ h + 1 .. H + 1 ] . We show as a consequence that (47) holds for ℓ = h . First, note that if RVFS h is never called during the execution of RVFS 0 , then V ̂ h = V , and so (47) trivially holds for ℓ = h under Assumption I.1 with ε real = 4 ε .

Now, suppose that RVFS h is called at least once, and let ( ̂ V + h ∶ H , V ̂ + h ∶ H , C + h ∶ H , B + h ∶ H , t + h ∶ H ) be the output of the last call to RVFS h during the execution of RVFS 0 . We claim that

To see this, first note that the for-loop in Line 16 ensures that no instance of ( RVFS τ ) τ > h can be called after the last call to RVFS h (by Lemma I.5). Thus, the estimated value functions, confidence sets, and core sets for layers h + 1 , . . . , H remain unchanged after the last call to RVFS h ; that is, (48) holds.

Thus, by Lemma I.2, and since we are conditioning on E ′ h + 1 ∶ H , we have that for all ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

where M = ⌈ 8 ε -1 C push H ⌉ . Now, by the induction hypothesis, we have V ˜ π ℓ ∈ ̂ V ℓ , and so substituting V ˜ π ℓ for f ℓ in (49), we get that for all ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

Therefore, by Lemma L.1 (instantiated with µ [⋅] = P ̂ π [⋅ ∣ x h -1 = x h -1 , a h -1 = a h -1 ] , τ = ℓ -1 , and V τ + 1 = V ˜ π ℓ ), we have that ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

where δ ′ is as in Algorithm 5.

Applying the regression guarantee to conclude the induction. Note that ˜ π ∈ Π ′ , where Π ′ ⊂ Π S is the set of stochastic policies such that π ∈ Π ′ if and only if there exists V 1 ∶ H ∈ V such that π is defined recursively as

for τ = H,..., 1 , where Q τ ( x, a ) ∶= ̂ P τ,ε,δ ′ [ V τ + 1 ]( x, a ) . The policy class Π ′ is finite and ∣ Π ′ ∣ ≤ ∣V∣ . Furthermore, we have Π ′ ⊆ Π 4 ε as shown at the beginning of this proof. Therefore, if we let {D h ( x, a ) ∶ ( x, a ) ∈ C h } be the datasets in the definition of V ̂ h in (15), we have by Lemma I.3 (under Assumption I.1 with ε real = 4 ε ) and the conditioning on E ′′ h + 1 ∶ H and E :

where the last inequality follows by the fact that ∣C h ∣ ≤ M under E . Combining (51) with (50), implies that

where the last inequality follows by the fact that δ ∈ ( 0 , 1 ) and the definition of ε 2 reg in Algorithm 5. By the definition of V ̂ h in (15), (52) implies that V ˜ π h ∈ ̂ V h , which completes the induction.

I.6 Proof of Theorem I.1 (Main Guarantee of RVFS )

Proof of Theorem I.1. We condition on the event ̃ E ∶= E ∩ E ′′′ ∩ E ′ 1 ∩ ⋅ ⋅ ⋅ ∩ E ′ H , where E , E ′′′ , and (E ′ h ) are the events in Lemma I.1, Lemma I.4, and Lemma I.2, respectively. Note that by the union bound, we have P [ ̃ E] ≥ 1 -5 δ . By Lemma I.2, we have that

where M = ⌈ 8 ε -1 C push H ⌉ and N test = 2 8 M 2 Hε -1 log ( 8 M 6 H 8 ε -2 δ -1 ) . On the other hand, by Lemma I.4, we have

Thus, substituting V ˜ π h for f h in (53) we get that for all h ∈ [ H + 1 ] .

This together with Lemma L.1, instantiated with µ [⋅] = P ̂ π [⋅] ; τ = h -1 ; V τ + 1 = V ˜ π h ; and δ = δ ′ (with δ ′ as in Algorithm 5), translates to:

where the last step follows from the fact that N test = 2 8 M 2 Hε -1 log ( 8 M 6 H 8 ε -2 δ -1 ) (with M as in Line 3).

Bounding the sample complexity. We now bound the number of episodes used by Algorithm 5 under the event E ̃ . First, we fix h ∈ [ H ] , and focus on the number of episodes used within a to call RVFS h , excluding any episodes used by any subsequent calls to RVFS τ for τ > h . We start by counting the number of episodes used to test the fit of the estimated value functions ̂ V h + 1 ∶ H . Starting from Line 8, there are for-loops over ( x h -1 , a h -1 ) ∈ C h , ℓ = H,...,h + 1 , and n ∈ [ N test ] to collect partial episodes using the learned policy ̂ π in Algorithm 5, where N test = 2 8 M 2 Hε -1 log ( 8 M 6 H 8 ε -2 δ -1 ) and M = ⌈ 8 ε -1 C push H ⌉ . Note that executing ̂ π requires the local simulator and uses N sim = 2log ( 4 M 7 N 2 test H 2 ∣V∣/ δ )/ ε 2 local simulator queries to output an action at each layer (since Algorithm 5 calls Algorithm 7 with confidence level δ ′ = δ /( 8 M 7 N 2 test H 8 ∣V∣) ). Also, note that whenever a test fails in Line 14 and the recursive RVFS calls return, the for-loop in Line 8 resumes. We also know (by Lemma I.1) that the number of times the test fails in Line 14 is at most M . Thus, the number of times the for-loop in Line 8 resumes is bounded by HM ; here, H accounts for possible test failures across all layers τ ∈ [ h + 1 .. H ] . Thus, the total sample complexity required to generate episodes between lines Line 8 and Line 11 is bounded by

Note that the test in Line 14 also uses episodes because it calls the operator ̂ P for every a ∈ A . Thus, the number of episodes used for the test in Line 14 is bounded by

We now count the number of episodes used to re-fit the value function in Line 16 and onwards. Note that starting from Line 16, there are for-loops over ( x h -1 , a h -1 ) ∈ C h and i ∈ [ N reg ] to generate A ⋅ N est (∣C h ∣) ≤ A ⋅ N est ( M ) partial episodes using ̂ π , where N est ( k ) = 2 N 2 reg log ( 8 AN reg Hk 3 / δ )

is defined as in Algorithm 5. Since ̂ π uses the local simulator and requires N est samples (see Algorithm 7) to output an action at each layer, the number of episodes used to refit the value function is bounded by

Therefore, by (54), (55), and (56), the number of episodes used within a single call to RVFS h (not accounting for episodes used by recursive calls to RVFS τ , for τ > h ) is bounded by

Finally, by Lemma I.1, RVFS h may be called at most M times throughout the execution of RVFS 0 . Using this together with (57) and accounting for the number of episodes from all layers h ∈ [ H ] , we get that the total number of episodes is bounded by

Substituting the expressions of M , N test , N est , N sim , and N reg from Algorithm 5 and Algorithm 7, we obtain the desired number of episodes, which concludes the proof.

I.7 Proof of Theorem I.2 (Guarantee of RVFS.bc )

Proof of Theorem I.2. Let ̂ V 1 ∶ H be the value function estimates produced by RVFS 0 within Algorithm 6, and let ̂ π RVFS h (⋅) ∈ arg max a ∈A ̂ P h,ε RVFS ,δ ′ [ ̂ V h + 1 ](⋅ , a ) , for all h ∈ [ H ] with ̂ V H + 1 ≡ 0 with ε RVFS and δ ′ as in Algorithm 6. Further, let and let ˜ π 1 ∶ H ∈ Π S be the stochastic policy defined recursively via

for τ = H,..., 1 , where ̂ Q τ ( x, a ) ∶= ̂ P τ,ε RVFS ,δ ′ [ ̂ V τ + 1 ]( x, a ) . By Theorem I.1, there is an event E ̃ of probability at least 1 -δ / 2 under which:

and where the last inequality follows by the choice of ε RVFS in Algorithm 6.

For the rest of the proof, we condition on E ̃ . By (59), (63), and Proposition M.1 instantiated with ε mis = 0 (due to all π -realizability), we have that there is an event E ̃ ′ of probability at least 1 -δ / 2 under which the policy ̂ π 1 ∶ H produced by BehaviorCloning ensures that

Now, by Lemma C.6 (the performance difference lemma), we have for ˜ π as in (58):

and so by (63), we have

Finally, since ˜ π ∈ Π 4 ε RVFS (see (59)), we have by Lemma H.1,

where the last inequality follows by the choice ε RVFS in Algorithm 6. Combining this with (61) and (62), we conclude that under ̃ E ∩ ̃ E ′ :

By the union bound, we have P [ ̃ E ∩ ̃ E ′ ] ≥ 1 -δ , and so the desired suboptimality guarantee in (28) holds with probability at least 1 -δ .

Bounding the sample complexity. The sample complexity is dominated by the call to RVFS 0 within RVFS.bc (Algorithm 6). Since RVFS.bc calls RVFS 0 with ε = ε RVFS = εH -1 / 48 , we conclude from Theorem I.1 that the total sample complexity is bounded by

J Guarantee under V ⋆ -Realizability (Proof of Theorem 4.1, Setup I)

In this section, we prove Theorem 4.1 under Setup I ( V ⋆ / π ⋆ -realizability (Assumptions 4.2 and 4.3) and ∆ -gap (Assumption 4.4)). We prove this result as a consequence of the more general results (Theorem I.2) in Appendix I by appealing to the relaxed V π -realizability condition in Assumption I.1.

J.1 Analysis: Proof of Theorem 4.1 (Setup I)

We begin by showing that Assumption 4.2 and Assumption 4.4 together imply that Assumption I.1 holds for any ε real ≤ ∆ / 2 ; we prove this by showing that the benchmark policy class Π ε ′ (Appendix H.2) reduces to { π ⋆ } when ε ′ ≤ ∆ / 2 .

Lemma J.1. Assume that V satisfies Assumption 4.2 ( V ⋆ -realizability), and that Assumption 4.4 (gap) holds with ∆ > 0 . Then, for all ε ′ ≤ ∆ / 2 , we have Π ε ′ = { π ⋆ } and V satisfies Assumption I.1 with ε real = ε ′ .

Informally Lemma J.1, whose proof is in Appendix J.2, states that under Assumption 4.2, Assumption 4.4 with ∆ > 0 , and Assumption 4.1 (pushforward coverability) with C push > 0 , we are essentially in the setting of Theorem I.1 (guarantee of RVFS under relaxed V π -realizability), as long as we choose ε real ≤ ∆ / 2 . With this, we now state and prove a central guarantee for RVFS under V ⋆ -realizability with a gap.

Lemma J.2 (Intermediate guarantee for RVFS under Setup I ) . Let δ ∈ ( 0 , 1 ) be given, and suppose that:

Assumption 4.1 (pushforward coverability) holds with parameter C push > 0 ;

Assumption 4.4 (gap) holds with parameter ∆ > 0 ;

The function class V satisfies Assumption 4.2 ( V ⋆ -realizability).

Then, for any f ∈ V and ε ∈ ( 0 , ∆ / 8 ) , with probability at least 1 -δ , RVFS 0 ( f, V , ∅ , ∅ ; V , ε, δ ) (Algorithm 5) terminates and returns value functions ̂ V 1 ∶ H that satisfy

where ̂ π h ( x ) ∈ arg max a ∈A ̂ P τ,ε,δ ′ [ ̂ V h + 1 ]( x, a ) , for all h ∈ [ H ] , with δ ′ is defined as in Algorithm 5. ⋆

Proof of Lemma J.2. From Lemma J.1, we have that Π 4 ε = { π } , and so Theorem I.1 implies that with probability at least 1 -δ ,

where the equality follows by the fact that π ⋆ is deterministic.

From here, Theorem 4.1 follows swiftly as a consequence.

Proof of Theorem 4.1 (Setup I). Let ̂ V 1 ∶ H be the value function estimates produced by RVFS 0 within Algorithm 6, and let ̂ π RVFS h (⋅) ∈ arg max a ∈A ̂ P h,ε RVFS ,δ ′ [ ̂ V h + 1 ](⋅ , a ) , for all h ∈ [ H ] with ̂ V H + 1 ≡ 0 with ε RVFS and δ ′ as in Algorithm 6. By Lemma J.2, there is an event E ̃ of probability at least 1 -δ / 2 under which:

where the last inequality follows by the choice of ε RVFS in Algorithm 6.

For the rest of the proof, we condition on E ̃ . By (63) and Assumption 4.3 ( π ⋆ -realizability), the policy ̂ π RVFS 1 ∶ H returned by RVFS 0 satisfies the condition in Proposition M.1 with ε mis = ε /( 4 C push H 3 ) . Thus, by Proposition M.1, there is an event E ̃ ′ of probability at least 1 -δ / 2 under which the policies ̂ π 1 ∶ H produced by RVFS.bc satisfy

We now condition on ̃ E ∩ ̃ E ′ . By Lemma C.6 (performance difference lemma), we have

where the last inequality follows by (63).

Finally, by the union bound, we have P [ ̃ E ∩ ̃ E ′ ] ≥ 1 -δ , and so the desired suboptimality guarantee in (28) holds with probability at least 1 -δ .

Bounding the sample complexity. The sample complexity is dominated by the call to RVFS 0 within RVFS.bc (Algorithm 6). Since RVFS.bc calls RVFS 0 with ε = ε RVFS = εH -1 / 48 , we conclude from Theorem I.1 that the total number of episodes is bounded by

Bounding the number of oracle calls. We now bound the number of calls to the oracle described in Remark 4.1 that RVFS makes. To do this, note that bounding the number of calls to the oracle is equivalent to bounding the number of executions of Line 14 in Algorithm 5, or, equivalently, the number of times the operator ̂ P in Line 14 is invoked during the entire execution of RVFS , including all subsequent recursive calls to RVFS . The proof of Lemma I.2 establishes a direct bound on the number of times the operator ̂ P is called, giving an upper limit of O ( C push N test H 4 ε -1 ) , where N test is a polynomial in the problem parameters (see Algorithm 5) and does not depend on X .

J.2 Proof of Lemma J.1 (Relaxed V π -Realizability under Gap)

Proof of Lemma J.1. Fix ε ′ ∈ ( 0 , 1 ) and π ∈ Π ε ′ . Let ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A be independent random variables such that

Such a collection of random state-action value functions ( ̃ Q h ( x, a )) ( h,x,a )∈[ H ]×X×A is guaranteed to exist for pi by the definition of Π ε ′ . We will show via backward induction over ℓ = H + 1 , . . . , 1 that

almost surely, with the convention that π H + 1 ≡ π ⋆ H + 1 ≡ π unif . This convention makes the base case, ℓ = H + 1 , hold trivially.

Now, we consider the general case. Fix h ∈ [ H ] and suppose that (64) holds for all ℓ ∈ [ h + 1 .. H + 1 ] . We will show that it holds for ℓ = h .

Thanks to the induction hypothesis, we have for all x ∈ X h + 1 and a ∈ A :

Fix x ∈ X . We will show that π h ( x ) = π ⋆ h ( x ) almost surely. Note that thanks to (65), the fact that ∥ ̃ Q h -T h [ Q π h + 1 ]∥ ∞ ≤ ε ′ almost surely, implies that

and so

almost surely. Using this, we have, almost surely

On the other hand, if π h ( x ) ≠ π ⋆ h ( x ) , then

which would contradict (66). Thus, π h ( x ) = π ⋆ h ( x ) , which concludes the induction and shows that π ≡ π ⋆ . We conclude that Π ε ′ = { π ⋆ } .

K Guarantee for Weakly Correlated ExBMDPs (Proof of Theorem B.1)

In this section, we prove Theorem B.1, the main guarantee for RVFS exo . First, in Appendix K.1 we state a number of supporting technical lemmas and use them to prove Theorem B.1. The remainder of the section (Appendix K.2 through Appendix K.6) contains the proofs for the intermediate results.

K.1 Analysis: Proof of Theorem B.1

Recall that the the V π -realizability assumption required by RVFS for Theorem 4.1 is not satisfied in ExBMDPs, as the value functions for policies that act on the exogenous noise variables cannot be realized as a function of the true decoder ϕ ⋆ . In RVFS exo , we address this issue by applying the randomized rounding technique in Line 11 to the learned value functions. The crux of the analysis will be to show that for an appropriate choice of the rounding parameters ζ 1 ∶ H , the policies produced by RVFS exo are endogenous in the sense that we can write π ( x ) = π ( ϕ ⋆ ( x )) for all x ∈ X . This will allow us to leverage the decoder realizability (Assumption 3.3), which implies that the function class V = V 1 ∶ H given by

satisfies V π -realizability for all endogenous policies π .

In what follows, we first motivate the randomized rounding approach in RVFS exo in detail and prove that it succeeds, then use this result to proceed with an analysis similar to that of Theorem 4.1 ( Setup II ), re-using many of the technical tools developed for Theorem 4.1.

K.1.1 Randomized Rounding for Endogeneity

Naively, to ensure that the policies we execute are endogenous, it would seem that we require knowledge of the true decoder ϕ ⋆ . Alas, knowing ϕ ⋆ trivializes the ExBMDP problem by reducing it to the tabular setting. To avoid requiring knowledge of ϕ ⋆ , we apply a randomized rounding to the policies learned by RVFS exo to ensure their endogeneity.

Let ε > 0 be fixed going forward. Recall that compared to RVFS , RVFS exo (Algorithm 8) takes an additional input ζ 1 ∶ H ⊂ ( 0 , 1 / 2 ) and executes the following coarsened policies:

The rounding parameters ζ 1 ∶ H , which can be thought of as an offset, are chosen randomly; this will be elucidated in the sequel.

Following a similar analysis to Appendix I ( Setup II ), we can associate a near-optimal benchmark policy ˜ π ∈ Π 2 ε with ̂ π in order to emulate certain properties of the ∆ -gap assumption. In particular, generalizing the construction in Eq. (27), we define a near-optimal benchmark policy ˜ π recursively via:

for τ = H,..., 1 , where ̂ Q τ (⋅ , a ) ∶= ̂ P τ,ε,δ [ ̂ V τ + 1 ](⋅ , a ) .

Naively, to use the benchmark policy ˜ π within the analysis based on relaxed V π -realizability (Assumption I.1) in Appendix I , we would require the function class V to realize ( V ˜ π h ) . However, as argued earlier, this is not feasible unless ˜ π is an endogenous policy. Fortunately, it turns out that if ζ 1 ∶ H (the additional input to RVFS exo ) are drawn randomly from uniform distribution over [ 0 , 1 / 2 ] , then with constant probability, ˜ π is indeed endogenous. What's more, under such an event, and for all possible choices of ( ̂ V h ) in (68) uniformly, ˜ π 'snaps' onto the stochastic endogenous policy ¯ π (⋅ ; ζ 1 ∶ H , ε ) defined recursively as follows:

for h = H,..., 1 . Informally, this happens because, as long as ζ 1 ∶ H ⊂ ( 0 , 1 / 2 ) avoid certain pathological locations in ( 0 , 1 / 2 ) , the coarsened state-action value functions ε ⋅ ⌈P τ [ V ¯ π τ + 1 ]( x, a )/ ε + ζ h ⌉ defining ¯ π exhibit a 'gap' of order Θ ( ε 2 ) separating optimal actions from the rest. This 'snapping' behavior is analogous to what happens in Setup I with V ⋆ -realizability and ∆ -gap, where Π ε reduces to { π ⋆ } for all ε < ∆ / 2 (see Lemma J.1). We formalize these claims in the next two lemmas. We start by showing that ¯ π is endogenous and that ¯ π ∈ Π 2 ε . The proof is in Appendix K.2.

Lemma K.1 (Endogenous Benchmark policies) . For any δ ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 / 2 ) , and ζ 1 ∶ H ⊂ ( 0 , 1 / 2 ) , the stochastic policy ¯ π (⋅ ; ζ 1 ∶ H , ε ) defined in Eq. (69) is endogenous, and we have ¯ π (⋅ ; ζ 1 ∶ H , ε ) ∈ Π 2 ε .

Next, we show that ˜ π 'snaps' onto ¯ π for the certain choices of ζ 1 ∶ H . The proof is in Appendix K.3.

Lemma K.2 (Snapping probability) . Let δ ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 / 2 ) be given, and P ζ denote the probability law of ζ 1 , . . . , ζ H ∼ unif ([ 0 , 1 / 2 ]) . Then, there is an event E rand of probability at least 1 -24 SAHε under ζ 1 ∶ H ∼ P ζ such that for all ̃ V ∈ (X × [ H ] → [ 0 , H ]) simultaneously,

where ˜ π h (⋅ ; ̃ V , ζ 1 ∶ H , ε, δ ) is defined as in (68) with ̂ V = ̃ V , and ¯ π is defined as in (69).

The lemma together, with Lemma K.1, implies that with constant probability under P ζ , the benchmark policies ( ˜ π h ) used in the analysis of RVFS exo are endogenous and satisfy ˜ π ∈ Π 2 ε .

K.1.2 Pushforward Coverability

In order to proceed with the analysis strategy in Appendix I, we need to verify that pushforward coverability is satisfied for ExBMDPs under the weak correlation assumption. We do so in the next lemma; see Appendix K.4 for a proof.

Lemma K.3 (Pushforward coverability) . A weakly correlated ExBMDP with constant C exo (see Assumption B.1) satisfies C push -pushforward coverability (Assumption 4.1) with C push = C exo ⋅ SA , where S ∈ N is the number of endogenous states.

Equipped with the preceding lemmas, we proceed with an analysis similar to the approach for Theorem 4.1 ( Setup II ) in Appendix I. In what follows, we state a number of technical lemmas that apply the relevant results from Appendix I to the ExBMDP setting we consider here.

K.1.3 Bounding the Number of Test Failures

Since the size of the core sets C 1 ∶ H in RVFS exo is directly proportional to the number of test failures, the next result, which bounds ∣C h ∣ for all h ∈ [ H ] , allows us to show that RVFS exo (Algorithm 8) terminates in a polynomial number of iterations.

Lemma K.4 (Bounding the number of test failures) . Let δ, ε ∈ ( 0 , 1 ) and ζ 1 ∶ H ∈ [ 0 , 1 / 2 ] be given, and suppose that Assumption B.1 (weak correlation) holds with C exo > 0 . Let f ∈ V , be given, where V is an arbitrary function class. Then, there is an event E of probability at least 1 -δ under which the call to RVFS exo 0 ( f, V H , ∅ , ∅ , 0; V , ε, ζ 1 ∶ H , δ ) (Algorithm 8) terminates, and throughout the execution of RVFS exo 0 , we have

Proof of Lemma K.4. The results follows from Lemma K.3 and Lemma I.1.

K.1.4 Value Function Regression Guarantee

We next give a guarantee for the estimated value functions ̂ V 1 ∶ H computed within RVFS exo in Line 26 of Algorithm 8.

Lemma K.5 (Value function regression guarantee) . Let h ∈ [ 0 .. H ] , δ, ε ∈ ( 0 , 1 ) , and ζ 1 ∶ H ∈ [ 0 , 1 / 2 ] be given, and consider a call to RVFS exo 0 in the setting of Lemma K.4. Further, let V be defined as in Eq. (67), and assume that Φ satisfies Assumption 3.3. Then, for any endogenous policy π in Π S , there is an event E ′′ h of probability at least 1 -δ / H under which for any k ≥ 1 , if

RVFS exo h gets called for the k th time during the execution of RVFS exo 0 ; and

this k th call terminates and returns ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) ,

then if (̂ π τ ) τ ≥ h is the policy induced by ̂ V h ∶ H and N reg is set as in Algorithm 8, we have

where the datasets {D h ( x, a ) ∶ ( x, a ) ∈ C h } are as in the definition of V ̂ h in (16).

Proof of Lemma K.5. Since Φ satisfies Assumption 3.3, the function class V = V 1 ∶ H satisfies V π -realizability for all endogenous policies π (see Lemma D.1). Thus, the proof of Lemma K.5 follows from that of Lemma I.3 (see Appendix I.4).

K.1.5 Confidence Sets

We now state a version of the confidence set validity lemma (Lemma I.4) that supports the ExBMDP setting.

Lemma K.6 (Confidence sets) . Let ε ∈ ( 0 , 1 / 2 ) and ζ 1 ∶ H ⊂ [ 0 , 1 / 2 ] be given, and suppose that

Assumption B.1 holds with C exo > 0 ;

The decoder class Φ satisfies Assumption 3.3;

ζ 1 ∶ H ∈ E rand , where E rand is the event in Lemma K.2.

Let f ∈ V be arbitrary. There is an event E ′′′ of probability at least 1 -3 δ under which a call to RVFS exo 0 ( f, V , ∅ , ∅ , 0; V , ε, ζ 1 ∶ H , δ ) terminates and returns tuple ( ̂ V 1 ∶ H , V ̂ 1 ∶ H , C 1 ∶ H , B 1 ∶ H , t 1 ∶ H ) such that

where ¯ π 1 ∶ H is the policy defined recursively via

While the proof of this lemma is very similar to that of Lemma I.4, we need a dedicated treatment to handle the rounding in RVFS exo . The fully proof of Lemma K.8 is in Appendix I.5.

K.1.6 Main Guarantee for RVFS exo

We now state the central technical guarantee for RVFS exo , Lemma K.7, which shows that the base invocation of the algorithm returns a set of value functions ̂ V 1 ∶ H that induce a near-optimal policy ̂ π , as long as the randomized rounding parameters ζ 1 ∶ H satisfy ζ 1 ∶ H ∈ E rand , where E rand is the success event in Lemma K.2. The proof of the theorem is in Appendix K.6.

Lemma K.7 (Main guarantee for RVFS exo ) . Let δ, ε ∈ ( 0 , 1 ) and ζ 1 ∶ H ⊂ [ 0 , 1 2 ] be given, and suppose that

Assumption B.1 holds with C exo > 0 ;

The decoder class Φ satisfies Assumption 3.3;

ζ 1 ∶ H ∈ E rand , where E rand is the event in Lemma K.2.

Then, for any f ∈ V , with probability at least 1 -5 δ , a call to RVFS exo 0 ( f, V H , ∅ , ∅ , 0; V , ε, ζ 1 ∶ H , δ ) (Algorithm 8) terminates and returns value functions ̂ V 1 ∶ H such that

where ̂ π h ( x ) ∈ arg max a ∈A ⌈ ̂ P h,ε,δ ′ [ ̂ V h + 1 ]( x, a )/ ε + ζ h ⌉ , for all h ∈ [ H ] , ¯ π is defined as in Eq. (69), and δ ′ is defined as in Algorithm 8. Furthermore, the number of episodes used by RVFS exo 0 is bounded by

K.1.7 Concluding: Main Guarantee for RVFS exo . bc

To conclude, we prove Theorem B.1, which shows that RVFS exo . bc succeeds with high probability. Recall that RVFS exo . bc (i) invokes RVFS exo multiple times for random samples ζ 1 ∶ H to ensure that the success event for Lemma K.7 occurs for at least one invocation, and (ii) extracts an executable policy using behavior cloning. Regarding the former point, note that the probability of the success event of Lemma K.2 can be boosted by sampling i.i.d. ζ ( 1 ) 1 ∶ H , . . . , ζ ( n ) 1 ∶ H ∼ P ζ inputs to RVFS exo for n ≥ 1 ; as long as n is polynomially large, with high probability at least one of the inputs ζ ( 1 ) 1 ∶ H , . . . , ζ ( n ) 1 ∶ H will satisfy the conclusion of Lemma K.2. Thus, it suffices to pick the policy with the highest value function among the different calls to RVFS exo . Using this, we prove Theorem B.1.

Proof of Theorem B.1. Recall that Algorithm 9 picks the final policy ̂ π ( i opt ) 1 ∶ H based on empirical value function estimates. In particular, for every i ∈ [ N boost ] (with N boost as in Algorithm 9), the estimate ̂ J (̂ π ( i ) 1 ∶ H ) for J (̂ π ( i ) 1 ∶ H ) is computed using N eval episodes. Thus, by Hoeffding's inequality and the union bound, we have that there is an event ˘ E of probability at least 1 -δ / 4 under which

Therefore, by definition of i opt in Algorithm 9, we have that under ˘ E :

where the last inequality follows by the choice of N eval in Algorithm 9. On the other hand, by Lemma K.2, there is an event E success of P ζ -probability at least

under which there exists j ∈ [ N boost ] such that ζ ( j ) 1 ∶ H ∈ E rand , where E rand is defined as in Lemma K.2. In what follows, we condition on the event E success and let j ∈ [ N boost ] be such that ζ ( j ) 1 ∶ H ∈ E rand . Further, we use ̂ π RVFS 1 ∶ H to denote the policy returned by the instance of RVFS exo that is used by RVFS exo . bc to learn ̂ π ( j ) 1 ∶ H .

By Proposition M.1 (instantiated with ε mis = 0 ), there is an event E ̃ ′ of probability at least 1 -δ / 4 under which the policy ̂ π ( j ) produced by BehaviorCloning satisfies

By Lemma K.7 and the fact that ζ ( j ) 1 ∶ H ∈ E rand , there is an event E ̃ of probability at least 1 -δ / 2 under which:

where ¯ π (⋅) ∶= ¯ π (⋅ ; ζ ( j ) 1 ∶ H , ε RVFS ) which is defined in (69).

Moving forward, we condition on ˘ E ∩ ̃ E ∩ ̃ E ′ . By Lemma C.6 (the performance difference lemma), we have

where the last inequality follows by (73). Now, by Lemma K.1, we have ¯ π ∈ Π 2 ε RVFS , and so by Lemma H.1,

where the last inequality follows by the choice of ε RVFS in Algorithm 9. Combining (75) with (71), (72), and (74), we get that

Finally, by the union bound, we have P [ ˘ E ∩ ̃ E ∩ ̃ E ′ ] ≥ 1 -δ , and so the desired suboptimality guarantee holds with probability at least 1 -δ .

Bounding the sample complexity. The sample complexity is dominated by the calls to RVFS exo 0 within RVFS exo . bc (Algorithm 9). Since RVFS exo . bc calls RVFS exo 0 with suboptimality parameter ε RVFS = εH -1 / 48 , we get by Lemma K.7 that the total sample complexity is bounded by

K.2 Proof of Lemma K.1 (Endogenous Benchmark Policies)

Proof of Lemma K.1. Fix δ ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 / 2 ) , and ζ ′ 1 ∶ H ⊂ [ 0 , 1 / 2 ] . We show via backward induction over ℓ = H + 1 , . . . , 1 that ¯ π τ (⋅ ; ζ ′ 1 ∶ H , ε ) is endogenous for all τ ∈ [ ℓ .. H + 1 ] , with the convention that ¯ π H + 1 = π unif . The base case holds trivially by convention.

Fix h ∈ [ H ] and suppose that the induction hypothesis holds for all ℓ ∈ [ h + 1 .. H + 1 ] . We show that it holds for ℓ = h . First, by the induction hypothesis, ¯ π ℓ (⋅ ; ζ ′ 1 ∶ H , ε ) is endogenous for all ℓ ∈ [ h + 1 .. H ] . Thus, there exists a function f h + 1 ∶ S → [ 0 , H -h ] such that

Therefore, we have for all ( x, a ) ∈ X × A :

where the last equality follows by the ExBMDP transition structure. Eq. (76) together with the fact that the rewards are endogenous (by assumption) implies that there exists g h ∶ S ×A → [ 0 , H -h + 1 ] such that

which in turn implies that x ↦⌈P h [ V ¯ π h + 1 ]( x, a )/ ε + ζ ′ h ⌉ is only a function of x through ϕ ⋆ ( x ) for all a ∈ A . Thus, ¯ π h is an endogenous policy and the induction is completed.

For the second claim, observe that for the functions ̃ Q 1 , /uni22EF , ̃ Q H ∈ [ 0 , H ] X×A defined as

we have

which implies that ¯ π (⋅ ; ζ ′ 1 ∶ H , ε ) ∈ Π 2 ε .

K.3 Proof of Lemma K.2 (Snapping Probability)

Proof of Lemma K.2. Fix ε ∈ ( 0 , 1 ) and δ ∈ ( 0 , 1 / 2 ) . For τ ≤ ℓ ∈ [ H ] , let P ζ τ ∶ ℓ denote the probability law of ζ τ , . . . , ζ ℓ . We also use the shorthand P ζ τ for P ζ τ ∶ τ , for all τ ∈ [ H ] . We show via backward induction over ℓ = H + 1 , . . . , 1 that there exists an event E ℓ of P ζ ℓ ∶ H -probability at least 1 -24 SA ( H -ℓ + 1 ) ε under which for all ̃ V ∈ (X × [ H ] → [ 0 , H ]) :

with the convention that ¯ π H + 1 ≡ ˜ π H + 1 ≡ π unif . We then set E rand = E 1 .

The base case follows trivially by convention.

We now proceed with the inductive step. Fix h ∈ [ H ] and suppose that the induction hypothesis holds for all ℓ ∈ [ h + 1 .. H ] . We show that it holds for ℓ = h . Throughout, we condition on E h + 1 . By definition of E h + 1 , we have for all ̃ V ∈ (X × [ H ] → [ 0 , H ]) :

This implies that for all ̃ V ∈ (X × [ H ] → [ 0 , H ]) :

by the definition of ˜ π h in (68), where ̂ Q h (⋅ , a ) ∶= ̂ P h,ε,δ [ ̃ V h + 1 ](⋅ , a ) . From this, we see that to prove ˜ π h (⋅ ; ̃ V , ζ 1 ∶ H , ε, δ ) = ¯ π h (⋅ ; ζ 1 ∶ H , ε ) for all ̃ V , it suffices to show that for all x ∈ X and ̃ V ,

Observe that a sufficient condition for this to hold is that

where δ represents all the possible values that the difference ̂ Q h ( x, a ) - P h [ V ¯ π h + 1 ]( x, a ) is allowed to take. By Lemma K.1, we know that ¯ π is endogenous, and so there exists a function g h ∶ S × A → [ 0 , H -h + 1 ] such that

Toward proving Eq. (77), observe that for any ( s, a ) ∈ S ∈ A , if ζ h is such that

then, for all δ ∈ [-4 ε 2 , 4 ε 2 ] and all x ∈ X such that ϕ ⋆ ( x ) = s , we have

Therefore, if we let E h ( s, a ) denote the event in (78), then under ⋂ ( s,a )∈S×A E h ( s, a ) , the desired condition in (77) holds. At this point, setting E h = (⋂ ( s,a )∈S×A E h ( s, a )) ∩ E h + 1 would complete the induction as long as P ζ h ∶ H [E h ] ≥ 1 -24 SA ( H -h + 1 ) ε . We now show that this is indeed the case by bounding the probability of the event ⋂ ( s,a )∈S×A E h ( s, a ) . By the union bound, we have

where E h ( s, a ) c denotes the complement of E h ( s, a ) . We now bound the probability

Fix ( s, a ) ∈ S × A . We have that ζ h ∈ E( s, a ) c if and only if

Now, since ζ h ∈ [ 0 , 1 / 2 ] , Lemma L.3 (instantiated with ( x, ζ, ν ) = ( g h ( s, a )/ ε, ζ h , 4 ε ) ) implies that (80) holds only if

Further, note that since ζ h is uniformly distributed over [ 0 , 1 / 2 ] , the P ζ h -probability of the event in (81) is at most the sum of the lengths of the intervals

multiplied by 2, which is equal to 24 ε . Therefore, we have

Combining this with (79), we obtain

Thus, by setting E h = (⋂ ( s,a )∈S×A E h ( s, a )) ∩ E h + 1 , we get that

which completes the induction.

K.4 Proof of Lemma K.3 (Coverability in Weakly Correlated ExBMDP)

Proof of Lemma K.3. Fix h ∈ [ 2 .. H ] and define the measure µ as

for all h ∈ [ H ] and x ∈ X . We show that µ satisfies Assumption 4.1 with C push = C exo ⋅ SA . First, note that µ is indeed a probability measure over X . Fix ( x, a, x ′ ) ∈ X × A × X . We have

and so by the ExBMDP structure:

and by Assumption B.1

This completes the proof.

K.5 Proof of Lemma K.6 (Confidence Sets)

To prove Lemma K.6, we need the following consequence of tests in Line 14 passing for all ℓ ∈ [ h + 1 .. H ] .

Lemma K.8 (Consequence of passed tests) . Let h ∈ [ 0 .. H ] , ε > 0 , and ζ 1 ∶ H ∈ [ 0 , 1 / 2 ] be given and consider a call to RVFS exo 0 (Algorithm 8) in the setting of Lemma K.4. Further, let E be the event of Lemma K.4. There exists an event E ′ h of probability at least 1 -δ / H such that under E ∩ E ′ h , if a call to RVFS exo h during the execution of RVFS exo 0 terminates and returns ( ̂ V h ∶ H , V ̂ h ∶ H , C h ∶ H , B h ∶ H , t h ∶ H ) , then for any ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

where (̂ π τ ) τ ≥ h ⊂ Π S , M , and N test are as in RVFS exo h (Algorithm 8).

Proof of Lemma K.8. This is just a restatement of Lemma I.2, and the proof is exactly the same as the latter.

We will also use Lemma I.5; even though this result is stated in section for the V π -realizable setting, it is also applicable to the ExBMDP variant of RVFS as it merely says something about the order in which the ( RVFS exo h ) instances are called. With this, we now prove Lemma K.6.

Proof of Lemma K.6. The proof is very similar to that of Lemma I.2, with differences to account for the 'coarsening' of the learned and benchmark policies.

We prove the desired result for E ′′′ ∶= E ∩ E ′ 1 ∩E ′′ 1 ∩⋅⋅⋅ ∩ E ′ H ∩E ′′ H , where E , (E ′ h ) , and (E ′′ h ) are the events in Lemma K.4, Lemma K.8, and Lemma K.5, respectively. Throughout, we condition on E ′′′ . First, note that by Lemma K.4, RVFS exo 0 terminates. Let ( ̂ V 1 ∶ H , V ̂ 1 ∶ H , C 1 ∶ H , B 1 ∶ H , t 1 ∶ H ) be its returned tuple.

We show via backward induction over ℓ = H + 1 , . . . , 1 , that

where ˜ π 1 ∶ H is the stochastic policy defined recursively via

∀

x

∈ X

,

˜

π

(

x

;

ζ

∶

, ε, δ

) ∈

a

∈A

⌈ ̂

Q

⌈P

τ

τ

(

[

x, a

V

˜

π

τ

+

)/

1

ε

](

+

ζ

x, a

τ

⌉

)/

,

ε

+

ζ

τ

⌉

,

if

∥ ̂

Q

τ

(

x,

⋅) - P

,

otherwise for τ = H,..., 1 , where ̂ Q τ (⋅ , a ) ∶= ̂ P τ,ε,δ [ ̂ V τ + 1 ](⋅ , a ) . Note that since ζ 1 ∶ H ∈ E rand (for E rand is defined in Lemma K.2), we have ˜ π ≡ ¯ π , where ¯ π is as in (70). Thus, instantiating the induction hypothesis with ℓ = h and using the definition of the confidence sets ( ̂ V ℓ ) in (16) together with V ˜ π h ≡ V ¯ π h (since ˜ π ≡ ¯ π ) implies the desired result.

Base case [ ℓ = H + 1 ]. Holds trivially since V π H + 1 ≡ 0 for any π ∈ Π S by convention.

General case [ ℓ ≤ H ]. Fix h ∈ [ H ] and suppose that (82) holds for all ℓ ∈ [ h + 1 .. H + 1 ] . We show that this remains true for ℓ = h . First, note that if RVFS exo h is never called during the execution of RVFS exo 0 , then V ̂ h = V h , and so (82) holds for ℓ = h , since ˜ π = ¯ π is endogenous under ζ 1 ∶ H ∈ E rand , where E rand is the event in Lemma K.2.

Now, suppose that RVFS exo h is called at least once, and let ( ̂ V + h ∶ H , V ̂ + h ∶ H , C + h ∶ H , B + h ∶ H , t + h ∶ H ) be the output of the last call to RVFS exo h throughout the execution of RVFS exo 0 . Next, we show that

The for-loop in Line 16 ensures that no instance of ( RVFS exo τ ) τ > h can be called after the last call to RVFS exo h (see Lemma I.5). Thus, the estimated value functions, confidence sets, and core sets for layers h + 1 , . . . , H remain unchanged after the last call to RVFS exo h ; that is, (83) holds. Thus, by Lemma K.8, and since we are conditioning on E ′ h + 1 ∶ H , we have that for all ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

Now, by the induction hypothesis, we have V ˜ π ℓ ∈ ̂ V ℓ , and so substituting V ˜ π ℓ for f ℓ in (84), we get that for all ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

. Therefore, by Lemma L.2 (instantiated with µ [⋅] = P ̂ π [⋅ ∣ x h -1 = x h -1 , a h -1 = a h -1 ] , τ = ℓ -1 , ε ′ = ε 2 , and V τ + 1 = V ˜ π ℓ ), we have that for all ( x h -1 , a h -1 ) ∈ C h and ℓ ∈ [ h + 1 .. H + 1 ] :

{

τ

1

H

τ

[

V

˜

π

τ

+

1

](

x,

⋅)∥

∞

4

2

ε

,

Now, since ¯ π (⋅ ; ζ 1 ∶ H , ε ) is endogenous and ˜ π ≡ ¯ π (thanks to ζ 1 ∶ H ∈ E rand ), Lemma K.5 (applied with π = ¯ π ) and the conditioning on E ′′ h + 1 ∶ H imply that:

where the datasets {D h ( x, a ) ∶ ( x, a ) ∈ C h } are as in the definition of V ̂ h in (16). Combining (86) with (85), we conclude that

where we have used that ∣C h ∣ ≤ M . By the definition of V ̂ h in (16), (87) implies that V ˜ π h ∈ ̂ V h , which completes the induction.

K.6 Proof of Lemma K.7 (Main Guarantee of RVFS exo )

Proof of Lemma K.7. We condition on the event ̃ E ∶= E ∩ E ′′′ ∩ E ′ 1 ∩ ⋅ ⋅ ⋅ ∩ E ′ H , where E , E ′′′ , and (E ′ h ) are the events in Lemma K.4, Lemma K.6, and Lemma K.8, respectively. Note that by the union bound, we have P [ ̃ E] ≥ 1 -5 δ . By Lemma K.6, we have that

where M = ⌈ 8 ε -2 C exo SAH ⌉ and N test = 2 8 M 2 Hε -2 log ( 8 M 6 H 8 ε -2 δ -1 ) . On the other hand, by Lemma K.6, we have

h

∀

∈ [

H

]

,

V

˜

π

h

∈ ̂

V

h

.

Thus, substituting V ˜ π h for f h in (88) we get that for all h ∈ [ H + 1 ] .

This together with Lemma L.2, instantiated with µ [⋅] = P ̂ π [⋅] ; τ = h -1 ; V τ + 1 = V ˜ π h ; and δ = δ ′ (with δ ′ as in Algorithm 5), translates to:

where the last step follows from the fact that N test = 2 8 M 2 Hε -2 log ( 8 M 6 H 8 ε -2 δ -1 ) (with M as in Line 3). Now, since ζ 1 ∶ H ∈ E rand (by assumption), Lemma K.2 implies that ˜ π ≡ ¯ π , where the latter is the deterministic policy defined in (69). Thus, by (89), we have

where the first equality follows by the fact that P [ ¯ π h ( x ) ≠ ̂ π h ( x )] = D tv ( ˜ π h ( x ) , ̂ π h ( x )) , for all x ∈ X , since ¯ π h is deterministic.

Bounding the sample complexity. We now bound the number of episodes used by Algorithm 8 under E ̃ . First, we fix h ∈ [ H ] , and focus on the number of episodes used within a to call RVFS exo h ; excluding any episodes used by any recursive calls to RVFS exo τ for τ > h . We start by counting the number of episodes used to test the fit of the estimated value functions ̂ V h + 1 ∶ H . Starting from Line 8, there are for-loops over ( x h -1 , a h -1 ) ∈ C h , ℓ = H,...,h + 1 , and n ∈ [ N test ] to collected partial episodes using the learned policy ̂ π in Algorithm 8, where N test = 2 8 M 2 Hε -2 log ( 8 M 6 H 8 ε -2 δ -1 ) and M = ⌈ 8 ε -2 C exo SAH ⌉ . Note that ̂ π uses the local simulator and requires N sim = 2log ( 4 M 7 N 2 test H 2 ∣V∣/ δ )/ ε 2 samples to output an action at each layer (since Algorithm 8 calls Algorithm 7 with confidence level δ ′ = δ /( 8 M 7 N 2 test H 8 ∣V∣) ). Also, note that whenever a test fails in Line 14, the for-loop in Line 8 resumes. We also know (by Lemma K.4) that the number of times the test fails in Line 14 is at most M . Thus, the number of times the for-loop in Line 8 resumes is bounded by HM ; here, H accounts for test failures for all layers τ ∈ [ h + 1 .. H ] . Thus, the number of episodes required to between lines Line 8 and Line 11 is bounded by

Note that the test in Line 14 also uses local simulator access because it calls the operator ̂ P for every a ∈ A . Thus, the number of episodes used for the test in Line 14 is bounded by

We now count the number of episodes used to re-fit the value function; Line 16 onwards. Note that starting from Line 16, there are for-loops over ( x h -1 , a h -1 ) ∈ C h and i ∈ [ N reg ] to generate A ⋅ N est (∣C h ∣) ≤ A ⋅ N est ( M ) partial episodes using ̂ π , where N est ( k ) = 2 N 2 reg log ( 8 AN reg Hk 3 / δ ) is as in Algorithm 8. And, since ̂ π uses local simulator access and requires N est samples (see Algorithm 7) to output an action at each layer, the number of episodes used to refit the value function is bounded by

Therefore, by (90), (91), and (92), the number of episodes used within a single call to RVFS exo h (not accounting for episodes used by recursive calls to RVFS exo τ , for τ > h ) is bounded by

Finally, by Lemma K.4, RVFS exo h may be called at most M times throughout the execution of RVFS exo 0 . Using this together with (93) and accounting for the number of episodes from all layers h ∈ [ H ] , we get that the total number of episodes is bounded by

Substituting the expressions of M , N test , N est , N sim , and N reg from Algorithm 8 and Algorithm 7, we obtain the desired number of episodes, which concludes the proof.

L Additional Technical Lemmas

Lemma L.1. Let τ ∈ [ H ] and ε, δ, ν ∈ ( 0 , 1 ) be given. Consider two value functions V τ + 1 , ̂ V τ + 1 ∈ [ 0 , H ] and a measure µ ∈ ∆ (X) such that

Further, for x ∈ X , let ̂ π τ ( x ) ∈ arg max a ∈A ̂ Q τ ( x, a ) ∶= ̂ P τ,ε,δ [ ̂ V τ + 1 ]( x, a ) and inductively define a randomized policy ˜ π via

Then, we have

Proof of Lemma L.1. In this proof, we let P µ denote the probability law of x τ and P P denote the probability law of ̂ P τ,ε,δ . Denote by E the P µ -measurable event that max a ∈A ∣(P τ [ ̂ V τ + 1 ] - P τ [ V τ + 1 ])( x τ , a )∣ ≤ 3 ε . Fix x ∈ E , and let E x be the P P -measurable event that max a ∈A ∣ ̂ P τ,ε,δ [ ̂ V τ + 1 ]( x, a ) - P τ [ V τ + 1 ]( x, a )∣ ≤ 4 ε . From the definition of ˜ π τ , we have that

and since P P [̂ π τ ( x ) = a ∣ E x ] = P P [ ˜ π τ ( x ) = a ∣ E x ] ∀ a ∈ A , we have that

where (95) follows from x ∈ E and the last inequality follows from Lemma H.2. Therefore, we have

where the first inequality follows by the fact that the total variation distance is bounded by 1, and the last inequality follows by (94) and (96).

Lemma L.2. Let τ ∈ [ H ] and ε ′ , δ, ν ∈ ( 0 , 1 ) , and ζ 1 ∶ H ∈ [ 0 , 1 / 2 ] be given. Further, consider two value functions V τ + 1 , ̂ V τ + 1 ∈ [ 0 , H ] and measure µ ∈ ∆ (X) such that

Further, for x ∈ X τ , let ̂ π τ ( x ) ∈ arg max a ∈A ⌈ ̂ Q τ ( x, a )/ ε ′ + ζ τ ⌉ , where ̂ Q τ ( x, a ) ∶= ̂ P τ,ε ′ ,δ [ ̂ V τ + 1 ]( x, a ) , and inductively define

Then, we have

Proof of Lemma L.2. In this proof, we let P µ denote the probability law of x τ and P P denote the probability law of ̂ P τ,ε,δ . Denote by E be the P µ -measurable event that max a ∈A ∣(P τ [ ̂ V τ + 1 ] - P τ [ V τ + 1 ])( x τ , a )∣ ≤ 3 ε ′ . Fix x ∈ E , and let E x be the P P -measurable event

that max a ∈A τ,ε ′ ,δ V τ + 1 x, a τ V τ + 1 x, a 4 ε ′ . From the definition of ˜ π τ , we have that

D tv (̂ π τ ( x ) , ˜ π τ ( x

∣ ̂ P [ ̂ ]( ) - P [ ]( )∣ ≤ ))

and since P P [̂ π τ ( x ) = a ∣ E x ] = P P [ ˜ π τ ( x ) = a ∣ E x ] ∀ a ∈ A ,

≤ P P [E c x ] ,

where (98) follows from x ∈ E and the last inequality follows from Lemma H.2. Therefore, we have

where the first inequality follows by the fact that the total variation is bounded by 1, and the last inequality follows by (97) and (99).

Lemma L.3. Let x ∈ R and ν ∈ ( 0 , 1 / 2 ) be given. Further, let ζ ∈ ( 0 , 1 / 2 ) . Then,

Proof of Lemma L.3. To prove the claim, it suffices to show the following items:

We start by showing the first item. We proceed by showing the contrapositive; that is, we will show that if x + ζ ≤ ⌈ x ⌉ -ν or x + ζ > ⌈ x ⌉ , then x + ζ + ν ≤ ⌈ x + ζ ⌉ . Suppose that x + ζ ≤ ⌈ x ⌉ -ν . This, together with the fact that ζ ≥ 0 , implies that

Now, suppose that x + ζ > ⌈ x ⌉ . Then, we have

where the penultimate inequality follows by ζ, ν ∈ ( 0 , 1 / 2 ) .

We now prove the second claim. Again, we proceed by showing the contrapositive; that is, we will show that if {⌈ x ⌉ + ν < x + ζ or ⌈ x ⌉ ≥ x + ζ } and ζ > ν , then x + ζ -ν > ⌈ x + ζ ⌉ -1 .

Suppose that ⌈ x ⌉ + ν < x + ζ and ζ > ν . The first inequality together with ν ≥ 0 implies that ⌈ x + ζ ⌉ > ⌈ x ⌉ . On the other hand, since ζ ≤ 1 / 2 , we have ⌈ x + ζ ⌉ ≤ ⌈ x ⌉ + 1 , and so

only if

where the last inequality follows by the current assumption that ⌈ x ⌉ + ν < x + ζ .

where the last inequality follows by ζ > ν . Rearranging (100) completes the proof.

M BehaviorCloning Algorithm and Analysis

In this section, we give a self-contained presentation and analysis for the standard behavior cloning algorithm for imitation learning (e.g., Ross and Bagnell [46]), displayed in Algorithm 10. Given access to trajectories from an expert policy ̂ π 1 ∶ H (which may be non-executable in the sense of Definition 2.1) the algorithm learns an executable policy π bc with similar performance. We use this scheme within RVFS.bc and RVFS exo . bc .

Algorithm 10 BehaviorCloning : Imitation Learning Algorithm.

1: input: Policy class Π ⊆ Π S , expert policy ̂ π 1 ∶ H , suboptimality ε ∈ ( 0 , 1 ) , and confidence δ ∈ ( 0 , 1 ) .

2: Set N bc = 16 H 2 log (∣ Π ∣/ δ )/ ε .

3: Set D ← ∅ .

4: for i = 1 , . . . , N bc do

5: Generate trajectory τ = (( x 1 , a 1 ) , . . . , ( x H , a H )) ∼ P ̂ π .

6: Update D ← D ∪ { τ } .

7: Compute π bc ∈ arg min π ∈ Π ∑ (( x 1 ,a 1 ) ,..., ( x H ,a H ))∈D ∑ h ∈[ H ] I { π h ( x h ) ≠ a h } .

8: Return π bc .

Proposition M.1. Let ε, δ ∈ ( 0 , 1 ) be given and let Π ⊆ Π S and ̂ π 1 ∶ H be an expert policy such that

Then, the policy π bc 1 ∶ H = BehaviorCloning ( Π , ε, ̂ π 1 ∶ H , δ ) returned by Algorithm 10 satisfies, with probability at least 1 -δ ,

Proof of Proposition M.1. First, by the performance difference lemma, we have

We now bound the probability terms on the right-hand side. Fix h ∈ [ H ] and let D be the dataset in Algorithm 10, which consists of N bc i.i.d. trajectories (( x 1 , a 1 ) , . . . , ( x H , a H )) generated by rolling with ̂ π 1 ∶ H . By Lemma C.4 (with i.i.d. data, B = H , and Q= Π ), we have that, with probability at least 1 -δ ,

and

Taking the infimum over π on both sides of (103) and using the definition of π bc h in Algorithm 10 gives:

where the last inequality follows from (101). Using this together with (104), instantiated with π ≡ π bc , we get that with probability at least 1 -δ :

Plugging this into (102), we get that with probability at least 1 -δ :

where the last inequality follows by the fact that N bc = 16 H 2 log ( 2 ∣ Π ∣/ δ )/ ε . This completes the proof.

Peer reviews:
comments: SUMMARY:
This paper study the online RL having access to a local simulator with general function approximation. Their results unclock new statistical guarantees. First, $Q^*$ realizability together with coverability assumption are enough for sample-efficient online RL under this setting. Second, their results further implies that the Exogenous Block MDP problem is tractable under this setting. Finally, they complement their theoretical finding with a computationally efficient algorithm.

STRENGTHS:
The paper is in general clearly written. The idea of leveraging local simulator to facilitate online RL with general functionn approximation is novel and leads to several important observations that previous works are out of reach.

WEAKNESSES:
Maybe due to space limit, some terms lack of definition or discussion, see questions for details. 
The algorithm in section 4 is highly technical and somewhat difficult to follow. Based on the current context, it appears that the role played by the local simulator is underexplained. One might question if most parts of the algorithm are adaptations of previous techniques to the general function approximation setting. It would be great if the novelty and insights can be highlighted.

QUESTIONS:
1. What is the double sampling problem mentioned on line 170?

2. What is the distribution shift mentioned on line 313?

3. What is the $v_h$ on line 16 of Alg 2?

Minor issues:

1. Definition of the confidence set in Alg 1, the summation $\sum_{h\leq t}$, is this a typo?
2. Double such that on line 221, and the comma and period in the end.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that local simulator access removes Bellman completeness for MDPs with bounded coverability.  This generalizes the results of existing works that are limited to low-rank or linear structures. \n\nOn the statistical front, the authors analyze the sample complexity of SimGOLF using coverability, but the algorithm itself is computationally inefficient. \nTo resolve this they propose RVFS, which is computationally efficient with respect to (1) a regression oracle, and (2) a convex optimization oracle over the value function class $\\mathcal{V}$. In order to obtain sample complexity guarantees here, however, stronger assumptions such as pushforward coverability or $Q^\\star$ gap are required. \n\n(For some context behind the proceeding comments, I am familiar with the analysis of GOLF and its variants [4,31,61], but not with the local simulator literature [38,63,57] beyond a brief glance. )\n\nSTRENGTHS:\nThe motivation for the paper is clear, and generalizing RL in local simulators to structures beyond linear is a valuable contribution. \n\nSimGOLF/ Thm 3.1 are clean results, and I appreciate that the authors tackled the problem of computational efficiency in RVFS.\n\nWEAKNESSES:\nMy impression is that the analysis of SimGOLF / Theorem 3.1 has limited technical novelty given [61]. That's not to say it's not a valuable contribution. \n\n**Oracle efficiency** \n\nMy biggest concern is that RVFS does not seem as \"oracle-efficient\" as claimed, and the discussion on this is somewhat lacking. For example, the authors could have stated more exactly what their definition of computational efficiency is (or is it just that it \"reduces to convex optimization\"?). They might have also provided an analysis on the # of calls to computational oracles before convergence. \n\nIn particular, it seems RVFS needs to solve the convex optimization in Line 8 for every $(x,a)$ in the core set, for $N_{test}$ times (which is like $\\varepsilon^{-1}$...?), and then for every possible action. Further, RVFS makes recursive calls to itself. Can this avoid $\\mathrm{poly}(\\mathcal{X}, \\mathcal{A}, \\varepsilon^{-1})$? \n\nMore minor, but it seems $\\mathcal{V}$ might need to be a convex set for efficient calculation of Line 8 (L355-358), which does not immediately gel with the \"holds for general function approximation\" or \"neural networks\" claims? In the grand scheme of things I'm sort of not that bothered by this. \n\n**Assumptions for RVFS**\n\nI am not opposed to make stronger assumptions for RVFS necessarily, but I would have liked to know more about why. For example, is it possible to give some intuition for why coverability alone is insufficient to analyze RVFS? And how do the gap or pushforward coverability assumptions help with this? \n\n**Comparison to existing work** \n\nThe authors mention that local simulator algorithms utilizing core sets exist for linear value function approximation exist, and I imagine that RVFS might also be applied to these settings. However, I find the consideration of previous literature to be quite limited, and I would have appreciated some discussion on how the sample/computational complexity of the algorithms compare (if they are comparable).\n\nQUESTIONS:\nThe questions below also include the specific ones from \"weaknesses\" that I would like to be answered. I would be happy to raise my score if my concerns are adequately addressed. \n\n1. Is it possible to analyze the complexity of oracle calls required to achieve the learning guarantee in Theorem 4.1? Will it be dependent on $\\mathcal{X}$? \n2. I imagine RVFS can be applied to the linear $V^\\star$ or $Q^\\star$ settings from [55,57,38]. Are the guarantees for RVFS comparable? I'm not necessarily looking for \"better\" performance because RVFS is more general, but just some sense of how big the gap might be (if there is one). Or just some discussion on the topic as I myself am not familiar.     \n3. Could you comment on the barriers to obtaining guarantees for RVFS under coverability, and/or why gap / pushforward helps? \n4. The definition of the confidence set in Algorithm 1 seems a little weird (maybe typo?), particularly the $\\sum_{h \\le t}$. Currently it seems like you throw samples from previous iterations away, is that intentional?\n\nLIMITATIONS:\nI believe the limitations of RVFS could benefit from greater discussion per my previous comments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the SimGolf algorithm, which leverages local simulator access to reset to previously visited states and sample multiple trajectories. This approach enhances sample efficiency and accuracy in value function approximation, particularly in high-dimensional MDPs. The SimGolf algorithm uses local simulator access to achieve new statistical guarantees, allowing for efficient learning in environments with low coverability. Additionally, the paper presents RVFS (Recursive Value Function Search), a computationally efficient algorithm that achieves sample complexity guarantees under a strengthened statistical assumption known as pushforward coverability.\n\nSTRENGTHS:\n- Introduces a novel approach that uses reset capability in reinforcement learning to significantly improve sample efficiency.\n- Provides strong theoretical analysis and new statistical guarantees for reinforcement learning with local simulator access.\n- Proposes two innovative algorithms (SimGolf and RVFS) with clear theoretical benefits.\n- The paper is generally well-written and explains the new algorithms and their theoretical foundations clearly.\n- Addresses a significant problem in reinforcement learning by enhancing sample efficiency and providing robust theoretical guarantees.\n\nWEAKNESSES:\n- The paper breaks with the expected shape of a NeurIPS paper (numbered list in abstract, missing discussion or future work). While deviations are acceptable if justified, the current format lacks some important aspects.\n- The abstract includes references and attempts to serve as a conclusion, which is unconventional and detracts from its clarity. The abstract should be a short, plain summary of the paper It is also not the place to reference other works, as the abstract should be self contained.\n- The paper lacks a dedicated conclusion or discussion section, which would be crucial for contextualizing results and suggesting future work.\n- There is no experimental data provided (which is fine for a theory paper), but there should then at least be a discussion on expected practical benefits and an outline of plans for empirical validation as part of a future works section. In the area of RL we observe a significant gap between the theoretical understanding of performance guarantees and the actual observed performance of certain setups in practice. I would therefore regard the benefit of a paper, that restricts itself to only theoretical work itself as limited; actual experiments are needed to validate the practical applicability of the new findings. Please outline such in a future works section.\n\nQUESTIONS:\n- How does SimGolf perform in practical environments compared to regular RL and MCTS?\n- What specific types of environments are expected to benefit most from SimGolf?\n- Are there any preliminary empirical results or plans for validation in real-world applications?\n- The training process produces a final policy and set of Q functions. It should therefore be possible to use the presented algorithm for 'pre-training' in a simulator with reset capabilities, but then continue training with a SAC-like approach on real world data?\n\n- The paper lacks an explicit differentiation from existing methods. Is the following summary correct for contextualizing the work? (This is mostly just for my own understanding of the presented work)\n   - Regular RL requires only the ability to take actions and observe resulting new states, enabling linear rollouts. It performs single, uninterrupted rollouts without the ability to reset.\n   - MCTS (Monte Carlo Tree Search) requires a complete and known transition function, allowing iteration over all possible child nodes from each state. It builds and expands a search tree incrementally by iterating over all possible actions and simulating outcomes. This method utilizes comprehensive exploration through rollouts and backpropagation within the search tree, resulting in high accuracy of value estimation. However, it is computationally intensive and requires full knowledge of the transition dynamics.\n   - SimGolf strikes a balance between the simplicity of regular RL and the higher performance of MCTS. It requires the ability to reset to previously visited states and resample the next state under the same or different actions. This method does not necessitate complete iteration over all possible actions and transitions. Instead, it enhances sample efficiency by allowing targeted resampling after initial rollouts (under the same or different actions). The agent can reset to critical states identified during the initial exploration and simulate multiple future trajectories from those states, improving the accuracy of value function estimation. SimGolf uses the reset capability to gather multiple samples from key state-action pairs, updating a confidence set of value functions based on empirical Bellman error estimates, combining the benefits of comprehensive exploration with more modest demands on the environment.\n\nI was assigned to this paper after the regular review period so I did not have time to go through the presented math inn detail. Rely on the other reviews to check those.\n\nLIMITATIONS:\n- The lack of experimental validation means the practical benefits and real-world performance of the algorithm remain uncertain.\n- A discussion on future work and plans for empirical validation would be beneficial, detailing how the authors intend to demonstrate practical improvements.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}

comments: SUMMARY:
The paper presents some theoretical results for new reinforcement learning algorithms with a sophisticated approach to a simulator environment.

STRENGTHS:
Paper presents an extensive theoretical study.

WEAKNESSES:
Practical applications of the algorithm remain questionable.

The modifications themselves might seem trivial.

QUESTIONS:
A clarification of the application of the algorithms and their implementation might improve the paper.

LIMITATIONS:
I don't think there's any negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-30