Title: Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline RL

online RL

exploration

non-reactive

fine-tuning
Abstract: In some applications of reinforcement learning, 
a dataset of pre-collected experience is already available
but it is also possible to acquire some additional online data to help improve the quality of the policy.
However, it may be preferable to gather additional data with a single, non-reactive exploration policy
and avoid the engineering costs associated with switching policies. 

In this paper we propose an algorithm with provable guarantees 
that can leverage an offline dataset to design a single non-reactive policy for exploration. 
We theoretically analyze the algorithm and measure the quality of the final policy 
as a function of the local coverage of the original dataset and the amount of additional data collected.

Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data

Abstract

In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies.

In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.

1 Introduction

Reinforcement learning (RL) is a general framework for data-driven, sequential decision making [Puterman, 1994, Sutton and Barto, 2018]. In RL, a common goal is to identify a near-optimal policy, and there exist two main paradigms: online and offline RL.

Online RL is effective when the practical cost of a bad decision is low, such as in simulated environments (e.g., [Mnih et al., 2015, Silver et al., 2016]). In online RL, a well designed learning algorithm starts from tabula rasa and implements a sequence of policies with a value that should approach that of an optimal policy. When the cost of making a mistake is high, such as in healthcare [Gottesman et al., 2018] and in self-driving [Kiran et al., 2021], an offline approach is preferred. In offline RL, the agent uses a dataset of pre-collected experience to extract a policy that is as good as possible. In this latter case, the quality of the policy that can be extracted from the dataset is limited by the quality of the dataset.

Many applications, however, fall between these two opposite settings: for example, a company that sells products online has most likely recorded the feedback that it has received from its customers, but can also collect a small amount of additional strategic data in order to improve its recommendation engine. While in principle an online exploration algorithm can be used to collect fresh data, in practice there are a number of practical engineering considerations that require the policy to be

deployed to be non-reactive . We say that a policy is non-reactive, (or passive, memoryless) if it chooses actions only according to the current state of the system. Most online algorithms are, by design, reactive to the data being acquired.

An example of a situation where non-reactive policies may be preferred are those where a human in the loop is required to validate each exploratory policy before they are deployed, to ensure they are of high quality [Dann et al., 2019] and safe [Yang et al., 2021], as well as free of discriminatory content [Koenecke et al., 2020]. Other situations that may warrant non-reactive exploration are those where the interaction with the user occurs through a distributed system with delayed feedback. In recommendation systems, data collection may only take minutes, but policy deployment and updates can span weeks [Afsar et al., 2022]. Similar considerations apply across various RL application domains, including healthcare [Yu et al., 2021], computer networks [Xu et al., 2018], and new material design [Raccuglia et al., 2016]. In all such cases, the engineering effort required to implement a system that handles real-time policy switches may be prohibitive: deploying a single, non-reactive policy is much preferred.

Non-reactive exploration from offline data Most exploration algorithms that we are aware of incorporate policy switches when they interact with the environment [Dann and Brunskill, 2015, Dann et al., 2017, Azar et al., 2017, Jin et al., 2018, Dann et al., 2019, Zanette and Brunskill, 2019, Zhang et al., 2020b]. Implementing a sequence of non-reactive policies is necessary in order to achieve near-optimal regret: the number of policy switches must be at least ˜ O ( H |S| |A| log log K ) where S , A , H, K are the state space, action space, horizon and the total number of episodes, respectively [Qiao et al., 2022]. With no switches, i.e., when a fully non-reactive data collection strategy is implemented, it is information theoretically impossible [Xiao et al., 2022] to identify a good policy using a number of samples polynomial in the size of the state and action space.

However, these fundamental limits apply to the case where the agent learns from tabula rasa. In the more common case where offline data is available, we demonstrate that it is possible to leverage the dataset to design an effective non-reactive exploratory policy. More precisely, an available offline dataset contains information (e.g., transitions) about a certain area of the state-action space, a concept known as partial coverage . A dataset with partial coverage naturally identifies a 'sub-region' of the original MDP-more precisely, a sub-graph-that is relatively well explored. We demonstrate that it is possible to use the dataset to design a non-reactive policy that further explores such sub-region. The additional data collected can be used to learn a near-optimal policy in such sub-region.

In other words, exploration with no policy switches can collect additional information and compete with the best policy that is restricted to an area where the original dataset has sufficient information. The value of such policy can be much higher than the one that can be computed using only the offline dataset, and does not directly depend on a concentrability coefficient [Munos and Szepesvári, 2008, Chen and Jiang, 2019].

Perhaps surprisingly, addressing the problem of reactive exploration in reinforcement learning requires an approach that combines both optimism and pessimism in the face of uncertainty to explore efficiently. While optimism drives exploration, pessimism ensures that the agent explores conservatively, in a way that restricts its exploration effort to a region that it knows how to navigate, and so our paper makes a technical contribution which can be of independent interest.

Contributions To the best of our knowledge, this is the first paper with theoretical rigor that considers the problem of designing an experiment in reinforcement learning for online, passive exploration, using a dataset of pre-collected experience. More precisely, our contributions are as follows:

We introduce an algorithm that takes as input a dataset, uses it to design and deploy a non-reactive exploratory policy, and then outputs a locally near-optimal policy.

We introduce the concept of sparsified MDP, which is actively used by our algorithm to design the exploratory policy, as well as to theoretically analyze the quality of the final policy that it finds.

We rigorously establish a nearly minimax-optimal upper bound for the sample complexity needed to learn a local ε -optimal policy using our algorithm. 1

2 Related Work

In this section we discuss some related literature. Our work is related to low-switching algorithms, but unlike those, we focus on the limit case where no-switiches are allowed. For more related work about low-switching algorithms, offline RL, task-agnostic RL, and reward-free RL we refer to Appendix F.

Low-switching RL In reinforcement learning, [Bai et al., 2019] first proposed Q-learning with UCB2 exploration, proving an O ( H 3 |S| |A| log K ) switching cost. This was later improved by a factor of H by the UCBadvantage algorithm in [Zhang et al., 2020b]. Recently, [Qiao et al., 2022] generalized the policy elimination algorithm from [Cesa-Bianchi et al., 2013] and introduced APEVE, which attains an optimal O ( H |S| |A| log log K ) switching cost. The reward-free version of their algorithm (which is not regret minimizing) has an O ( H |S| |A| ) switching cost.

Similar ideas were soon applied in RL with linear function approximation [Gao et al., 2021, Wang et al., 2021, Qiao and Wang, 2022] and general function approximation [Qiao et al., 2023]. Additionally, numerous research efforts have focused on low-adaptivity in other learning domains, such as batched dueling bandits [Agarwal et al., 2022], batched convex optimization [Duchi et al., 2018], linear contextual bandits [Ruan et al., 2021], and deployment-efficient RL [Huang et al., 2022].

Our work was inspired by the problem of non-reactive policy design in linear contextual bandits. Given access to an offline dataset, [Zanette et al., 2021a] proposed an algorithm to output a single exploratory policy, which generates a dataset from which a near-optimal policy can be extracted. However, there are a number of additional challenges which arise in reinforcement learning, including the fact that the state space is only partially explored in the offline dataset. In fact, in reinforcement learning, [Xiao et al., 2022] established an exponential lower bound for any non-adaptive policy learning algorithm starting from tabula rasa.

3 Setup

Throughout this paper, we let [ n ] = { 1 , 2 , ..., n } . We adopt the big-O notation, where ˜ O ( · ) suppresses poly-log factors of the input parameters. We indicate the cardinality of a set X with |X| .

Markov decision process We consider time-homogeneous episodic Markov decision processes (MDPs). They are defined by a finite state space S , a finite action space A , a trasition kernel P , a reward function r and the episodic length H . The transition probability P ( s ′ | s, a ) , which does not depend on the current time-step h ∈ [ H ] , denotes the probability of transitioning to state s ′ ∈ S when taking action a ∈ A in the current state s ∈ S . Typically we denote with s 1 the initial state. For simplicity, we consider deterministic reward functions r : S × A → [0 , 1] . A deterministic non-reactive (or memoryless, or passive) policy π = { π h } h ∈ [ H ] maps a given state to an action.

The value function is defined as the expected cumulated reward. It depends on the state s under consideration, the transition P and reward r that define the MDP as well as on the policy π being implemented. It is defined as V h ( s ; P , r, π ) = E P ,π [ ∑ H i = h r ( s i , a i ) | s h = s ] , where E P ,π denotes the expectation generated by P and policy π . A closely related quantity is the state-action value function, or Q -function, defined as Q h ( s, a ; P , r, π ) = E P ,π [ ∑ H i = h r ( s i , a i ) | s h = s, a h = a ] . When it is clear from the context, we sometimes omit ( P , r ) and simply write them as V π h ( s ) and Q π h ( s, a ) . We denote an MDP defined by S , A and the transition matrix P as M = ( S , A , P ) .

3.1 Interaction protocol

Algorithm 1 Design of experiments in reinforcement learning

Input: Offline dataset D

Output: Return π final

In this paper we assume access to an offline dataset D = { ( s, a, s ′ ) } where every state-action ( s, a ) is sampled in an i.i.d. fashion from some distribution µ and s ′ ∼ P ( · | s, a ) , which is common in the offline RL literature [Xie et al., 2021a, Zhan et al., 2022, Rashidinejad et al., 2021, Uehara and Sun, 2021]. We denote N ( s, a ) and N ( s, a, s ′ ) as the number of ( s, a ) and ( s, a, s ′ ) samples in the offline dataset D , respectively. The interaction protocol considered in this paper consists of three distinct phases, which are displayed in algorithm 1. They are:

the offline phase , where the learner uses an offline dataset D of pre-collected experience to design the non-reactive exploratory policy π ex ;

the online phase where π ex is deployed to generate the online dataset D ′ ;

the planning phase where the learner receives a reward function and uses all the data collected to extract a good policy π final with respect to that reward function.

The objective is to minimize the number of online episodic interactions needed to find a policy π final whose value is as high as possible. Moreover, we focus on the reward-free RL setting [Jin et al., 2020a, Kaufmann et al., 2021, Li et al., 2023b], which is more general than reward-aware RL. In the offline and online phase, the data are generated without specific reward signals, and the entire reward information is then given in the planning phase. One of the primary advantages of using reward-free offline data is that it allows for the collection of data without the need for explicit reward signals. This can be particularly beneficial in environments where obtaining reward signals is costly, risky, ethically challenging, or where the reward functions are human-designed.

4 Algorithm: balancing optimism and pessimism for experimental design

In this section we outline our algorithm Reward-Free Non-reactive Policy Design (RF-NPD), which follows the high-level protocol described in algorithm 1. The technical novelty lies almost entirely in the design of the exploratory policy π ex . In order to prepare the reader for the discussion of the algorithm, we first give some intuition in section 4.1 followed by the definition of sparsified MDP in section 4.2, a central concept of this paper, and then describe the implementation of line 1 in the protocol in algorithm 1 in section 4.3. We conclude by presenting the implementation of lines 2 and 3 in the protocol in algorithm 1.

4.1 Intuition

In order to present the main intuition for this paper, in this section we assume that enough transitions are available in the dataset for every edge ( s, a ) → s ′ , namely that the critical condition

holds for all tuples ( s, a, s ′ ) ∈ S × A × S (the precise value for Φ will be given later in eq. (5.1)). Such condition is hardly satisfied everywhere in the state-action-state space, but assuming it in this section simplifies the presentation of one of the key ideas of this paper.

The key observation is that when eq. (4.1) holds for all ( s, a, s ′ ) , we can use the empirical transition kernel to design an exploration policy π ex to eventually extract a near-optimal policy π final for any desired level of sub-optimality ε , despite eq. (4.1) being independent of ε . More precisely, let ̂ P be the empirical transition kernel defined in the usual way ̂ P ( s ′ | s, a ) = N ( s, a, s ′ ) /N ( s, a ) for any tuple ( s, a, s ′ ) . The intuition-which will be verified rigorously in the analysis of the algorithm-is the following:

If eq. (4.1) holds for every ( s, a, s ′ ) then ̂ P can be used to design a non-reactive exploration policy π ex which can be deployed on M to find an ε -optimal policy π final using ≍ 1 ε 2 samples.

We remark that even if the condition 4.1 holds for all tuples ( s, a, s ′ ) , the empirical kernel ̂ P is not accurate enough to extract an ε -optimal policy from the dataset D without collecting further data. Indeed, the threshold Φ = ˜ Θ( H 2 ) on the number of samples is independent of the desired sub-optimality ε > 0 , while it is well known that at least ∼ 1 ε 2 offline samples are needed to find an ε -optimal policy. Therefore, directly implementing an offline RL algorithm to use the available

offline dataset D does not yield an ε -optimal policy. However, the threshold Φ = ˜ Θ( H 2 ) is sufficient to design a non-reactive exploratory policy π ex that can discover an ε -optimal policy π final after collecting ∼ 1 ε 2 online data.

4.2 Sparsified MDP

The intuition in the prior section must be modified to work with heterogeneous datasets and dynamics where N ( s, a, s ′ ) ≥ Φ may fail to hold everywhere. For example, if P ( s ′ | s, a ) is very small for a certain tuple ( s, a, s ′ ) , it is unlikely that the dataset contains N ( s, a, s ′ ) ≥ Φ samples for that particular tuple. In a more extreme setting, if the dataset is empty, the critical condition in eq. (4.1) is violated for all tuples ( s, a, s ′ ) , and in fact the lower bound of Xiao et al. [2022] states that finding ε -optimal policies by exploring with a non-reactive policy is not feasible with ∼ 1 ε 2 sample complexity. This suggests that in general it is not possible to output an ε -optimal policy using the protocol in algorithm 1.

However, a real-world dataset generally covers at least a portion of the state-action space, and so we expect the condition N ( s, a, s ′ ) ≥ Φ to hold somewhere; the sub-region of the MDP where it holds represents the connectivity graph of the sparsified MDP . This is the region that the agent knows how to navigate using the offline dataset D , and so it is the one that the agent can explore further using π ex . More precisely, the sparsified MDP is defined to have identical dynamics as the original MDP on the edges ( s, a ) -→ s ′ that satisfy the critical condition 4.1. When instead the edge ( s, a ) -→ s ′ fails to satisfy the critical condition 4.1, it is replaced with a transition ( s, a ) -→ s † to an absorbing state s † .

Definition 4.1 (Sparsified MDP) . Let s † be an absorbing state, i.e., such that P ( s † | s † , a ) = 1 and r ( s † , a ) = 0 for all a ∈ A . The state space in the sparsified MDP M † is defined as that of the original MDP with the addition of s † . The dynamics P † of the sparsified MDP are defined as

̸

For any deterministic reward function r : S × A → [0 , 1] , the reward function on the sparsified MDP is defined as r † ( s, a ) = r ( s, a ) ; for simplicity we only consider deterministic reward functions.

The empirical sparsified MDP ̂ M † = ( S ∪ { s † } , A , ̂ P † ) is defined in the same way but by using the empirical transition kernel in eq. (4.2). The empirical sparsified MDP is used by our algorithm to design the exploratory policy, while the (population) sparsified MDP is used for its theoretical analysis. They are two fundamental concepts in this paper. By formulating the sparsified MDP, we restrict the transitions and rewards within the area where we know how to navigate, embodying the principle of pessimism. Various forms of pessimistic regularization have been introduced to address the challenges of partially covered offline data. Examples include a pessimistic MDP [Kidambi et al., 2020] and limiting policies to those covered by offline data [Liu et al., 2020].

4.3 Offline design of experiments

In this section we describe the main sub-component of the algorithm, namely the sub-routine that uses the offline dataset D to compute the exploratory policy π ex . The exploratory policy π ex is a mixture of the policies π 1 , π 2 , . . . produced by a variant of the reward-free exploration algorithm of [Kaufmann et al., 2021, Ménard et al., 2021]. Unlike prior literature, the reward-free algorithm is not interfaced with the real MDP M , but rather simulated on the empirical sparsified MDP ̂ M † . This avoids interacting with M with a reactive policy, but it introduces some bias that must be controlled. The overall procedure is detailed in algorithm 2. To be clear, no real-world samples are collected by algorithm 2; instead we use the word 'virtual samples' to refer to those generated from ̂ M † .

At a high level, algorithm 2 implements value iteration using the empirical transition kernel ̂ P † , with the exploration bonus defined in eq. (4.3) that replaces the reward function. The exploration bonus can be seen as implementing the principle of optimism in the face of uncertainty; however, the possibility of transitioning to an absorbing state with zero reward (due to the use of the absorbing state in the definition of ̂ P † ) implements the principle of pessimism.

This delicate interplay between optimism and pessimism is critical to the success of the overall procedure: while optimism encourages exploration, pessimism ensures that the exploration efforts are directed to the region of the state space that the agent actually knows how to navigate, and prevents the agent from getting 'trapped' in unknown regions. In fact, these latter regions could have combinatorial structures [Xiao et al., 2022] which cannot be explored with non-reactive policies.

More precisely, at the beginning of the k -th virtual episode in algorithm 2, n k ( s, a ) and n k ( s, a, s ′ ) denote the counters for the number of virtual samples simulated from ̂ M † at each ( s, a ) and ( s, a, s ′ ) tuple. We define the bonus function

which is used to construct the empirical uncertainty function U k h , a quantity that serves as a proxy for the uncertainty of the value of any policy π on the spasified MDP. Specifically, for the k -th virtual episode, we set U k H +1 ( s, a ) = 0 and s ∈ S , a ∈ A . For h ∈ [ H ] , we further define:

Note that, the above bonus function takes a similar form of the bonus function in [Ménard et al., 2021]. This order of O (1 /x ) is set to achieve the optimal sample complexity, and other works have also investigated into other forms of bonus function [Kaufmann et al., 2021]. Finally, in line 10 through line 12 the current policy π k -which is the greedy policy with respect to U k -is simulated on the empirical reference MDP ̂ M † , and the virtual counters are updated. It is crucial to note that the simulation takes place entirely offline, by generating virtual transitions from ̂ M † . Upon termination of algorithm 2, the uniform mixture of policies π 1 , π 2 , . . . form the non-reactive exploration policy π ex , ensuring that the latter has wide 'coverage' over M † .

4.4 Online and planning phase

Algorithm 2 implements line 1 of the procedure in algorithm 1 by finding the exploratory policy π ex . After that, in line 2 of the interaction protocol the online dataset D ′ is generated by deploying π ex on the real MDP M to generate K de trajectories. Conceptually, the online dataset D ′ and the offline dataset D identify an updated empirical transition kernel ˜ P and its sparsified version 2 ˜ P † . Finally, in

line 3 a reward function r is received, and the value iteration algorithm (See Appendix E) is invoked with r as reward function and ˜ P † as dynamics, and the near-optimal policy π final is produced. The use of the (updated) empirical sparsified dynamics ˜ P † can be seen as incorporating the principle of pessimism under uncertainty due to the presence of the absorbing state.

Our complete algorithm is reported in algorithm 3, and it can be seen as implementing the interaction protocol described in algorithm 1.

Algorithm 3 Reward-Free Non-reactive Policy Design (RF-NPD)

Input: Offline dataset D , target suboptimality ε > 0 , failure tolerance δ ∈ (0 , 1] .

1: Construct the empirical sparsified MDP ̂ M † .

2: Offline phase: run RF-UCB ( ̂ M † , K ucb , ε, δ ) to obtain the exploratory policy π ex .

3: Online phase: deploy π ex on the MDP M for K de episodes to get the online dataset D ′ .

4: Planning phase: receive the reward function r , construct ˜ M † from the online dataset D ′ , compute π final (which is the optimal policy on ˜ M † ) using value iteration (Appendix E).

Output: π final .

5 Main Result

In this section, we present a performance bound on our algorithm, namely a bound on the suboptimality of the value of the final policy π final when measured on the sparsified MDP M † . The sparsified MDP arises because it is generally not possible to directly compete with the optimal policy using a non-reactive data collection strategy and a polynomial number of samples due to the lower bound of Xiao et al. [2022]; more details are given in Appendix C.

In order to state the main result, we let K = K ucb = K de , where K ucb and K de are the number of episodes for the offline simulation and online interaction, respectively. Let C be some universal constant, and choose the threshold in the definition of sparsified MDP as

Theorem 5.1. For any ε > 0 and 0 < δ < 1 , if we let the number of online episodes be

then with probability at least 1 -δ, for any reward function r, the final policy π final returned by Algorithm 3 satisfies the bound

The theorem gives a performance guarantee on the value of the policy π final , which depends both on the initial coverage of the offline dataset D as well as on the number of samples collected in the online phase. The dependence on the coverage of the offline dataset is implicit through the definition of the (population) sparsified M † , which is determined by the counts N ( · , · ) .

In order to gain some intuition, we examine some special cases as a function of the coverage of the offline dataset.

Empty dataset Suppose that the offline dataset D is empty. Then the sparsified MDP identifies a multi-armed bandit at the initial state s 1 , where any action a taken from such state gives back the reward r ( s 1 , a ) and leads to the absorbing state s † . In this case, our algorithm essentially designs an allocation strategy π ex that is uniform across all actions at the starting state s 1 . Given enough online samples, π final converges to the action with the highest instantaneous reward on the multi-armed bandit induced by the start state. With no coverage from the offline dataset, the lower bound of Xiao et al. [2022] for non-reactive policies precludes finding an ε -optimal policy on the original MDP M unless exponentially many samples are collected.

Known connectivity graph On the other extreme, assume that the offline dataset contains enough information everywhere in the state-action space such that the critical condition 4.1 is satisfied for

all ( s, a, s ′ ) tuples. Then the sparsified MDP and the real MDP coincide, i.e., M = M † , and so the final policy π final directly competes with the optimal policy π ∗ for any given reward function in eq. (5.2). More precisely, the policy π final is ε -suboptimal on M if ˜ O ( H 2 |S| 2 |A| /ε 2 ) trajectories are collected in the online phase, a result that matches the lower bound for reward-free exploration of Jin et al. [2020b] up to log factors. However, we achieve such result with a data collection strategy that is completely passive, one that is computed with the help of an initial offline dataset whose size |D| ≈ Φ ×|S| 2 |A| = ˜ O ( H 2 |S| 2 |A| ) need not depend on final accuracy ε .

Partial coverage In more typical cases, the offline dataset has only partial coverage over the stateaction space and the critical condition 4.1 may be violated in certain state-action-successor states. In this case, the connectivity graph of the sparsified MDP M † is a sub-graph of the original MDP M augmented with edges towards the absorbing state. The lack of coverage of the original dataset arises through the sparsified MDP in the guarantees that we present in theorem 5.1. In this section, we 'translate' such guarantees into guarantees on M , in which case the 'lack of coverage' is naturally represented by the concentrability coefficient

see for examples the papers [Munos and Szepesvári, 2008, Chen and Jiang, 2019] for background material on the concentrability factor. More precisely, we compute the sample complexity-in terms of online as well as offline samples-required for π final to be ε -suboptimal with respect to any comparator policy π , and so in particular with respect to the optimal policy π ∗ on the 'real' MDP M . The next corollary is proved in appendix B.3.

Corollary 5.2. Suppose that the offline dataset contains

online samples are collected during the online phase. Then with probability at least 1 -δ , for any reward function r , the policy π final is ϵ -suboptimal with respect to any comparator policy π

The online sample size is equivalent to the one that arises in the statement of theorem 5.1 (expressed as number of online trajectories), and does not depend on the concentrability coefficient. The dependence on the offline dataset in theorem 5.1 is implicit in the definition of sparsified MDP; here we have made it explicit using the notion of concentrability.

Corollary 5.2 can be used to compare the achievable guarantees of our procedure with that of an offline algorithm, such as the minimax-optimal procedure detailed in [Xie et al., 2021b]. The proceedure described in [Xie et al., 2021b] achieves (5.3) with probability at least 1 -δ by using

offline samples 3 . In terms of offline data, our procedure has a similar dependence on various factors, but it depends on the desired accuracy ε through ˜ O (1 /ε ) as opposed to ˜ O (1 /ε 2 ) which is typical for an offline algorithm. This implies that in the smallε regime, if sufficient online samples are collected, one can improve upon a fully offline procedure by collecting a number of additional online samples in a non-reactive way.

Finally, notice that one may improve upon an offline dataset by collecting more data from the distribution µ , i.e., without performing experimental design. Compared to this latter case, notice that our online sample complexity does not depend on the concentrability coefficient . Further discussion can be found in appendix B.

samples and that additional

6 Proof

In this section we prove theorem 5.1, and defer the proofs of the supporting statements to the Appendix A.

Let us define the comparator policy π † ∗ used for the comparison in eq. (5.2) to be the (deterministic) policy with the highest value function on the sparsified MDP:

We can bound the suboptimality using the triangle inequality as

The middle term after the first inequality is negative due to the optimality of π final on ˜ P † and r † . It suffices to prove that for any arbitrary policy π and reward function r the following statement holds with probability at least 1 -δ

Bounding the estimation error using the population uncertainty function In order to prove eq. (6.1), we first define the population uncertainty function X , which is a scalar function over the state-action space. It represents the maximum estimation error on the value of any policy when it is evaluated on ˜ M † instead of M . For any ( s, a ) ∈ S × A , the uncertainty function is defined as X H +1 ( s, a ) := 0 and for h ∈ [ H ] ,

We extend the definition to the absorbing state by letting X h ( s † , a ) = 0 for any h ∈ [ H ] , a ∈ A . The summation ∑ s ′ used above is over s ′ ∈ S ∪ { s † } , but since X h ( s † , a ) = 0 for any h ∈ [ H ] , a ∈ A , it is equivalent to that over s ′ ∈ S . Intuitively, X h ( s, a ) takes a similar form as Bellman optimality equation. The additional (1+1 /H ) factor and additional term 9 Hϕ ( m ( s, a )) quantify the uncertainty of the true Q function on the sparsifed MDP and 9 Hϕ ( m ( s, a )) will converge to zero when the sample size goes to infinity. This definition of uncertainty function and the following lemma follow closely from the uncertainty function defined in [Ménard et al., 2021].

The next lemma highlights the key property of the uncertainty function X , namely that for any reward function and any policy π, we can upper bound the estimation error via the uncertainty function at the initial times-step; it is proved in appendix A.2.1.

Lemma 6.1. With probability 1 -δ , for any reward function r and any deterministic policy π , it holds that

The uncertainty function X contains the inverse number of online samples 1 /m ( s, a ) through ϕ ( m ( s, a )) , and so lemma 6.1 expresses the estimation error in eq. (6.1) as the maximum expected size of the confidence intervals sup π E ˜ P † , ( s,a ) ∼ π √ 1 /m ( s, a ) , a quantity that directly depends on the number m ( · , · ) of samples collected during the online phase.

Leveraging the exploration mechanics Throughout this section, C denotes some universal constant and may vary from line to line. Recall that the agent greedily minimizes the empirical uncertainty function U to compute the exploratory policy π ex . The empirical uncertainty is defined as U k H +1 ( s, a ) = 0 for any k, s ∈ S , a ∈ A and

where n k ( s, a ) is the counter of the times we encounter ( s, a ) until the beginning of the k -the virtual episode in the simulation phase. Note that, U k h ( s, a ) takes a similar form as X h ( s, a ) , except that U k h ( s, a ) depends on the empirical transition probability ̂ P † while X h ( s, a ) depends on the true transition probability on the sparsified MDP. For the exploration scheme to be effective, X and U should be close in value, a concept which is at the core of this work and which we formally state below and prove in appendix A.2.2.

Lemma 6.2 (Bounding uncertainty function with empirical uncertainty functions) . With probability at least 1 -δ , we have for any ( h, s, a ) ∈ [ H ] ×S × A ,

Notice that X h is the population uncertainty after the online samples have been collected, while U k h is the corresponding empirical uncertainty which varies during the planning phase.

Rate of decrease of the estimation error Combining lemmas 6.1 and 6.2 shows that (a function of) the agent's uncertainty estimate U upper bounds the estimation error in eq. (6.1). In order to conclude, we need to show that U decreases on average at the rate 1 /K , a statement that we present below and prove in appendix A.2.3.

Lemma 6.3. With probability at least 1 -δ , we have

Then, for any ε > 0 , if we take

then with probability at least 1 -δ , it holds that

After combining lemmas 6.1 to 6.3, we see that the estimation error can be bounded as

Here, the constant C may vary between lines. Rescaling the universal constant C and the failure probability δ , we complete the upper bound in equation (6.1) and hence the proof for the main result.

References

MMehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys , 55(7):1-38, 2022.

Arpit Agarwal, Rohan Ghuge, and Viswanath Nagarajan. Batched dueling bandits. In International Conference on Machine Learning , pages 89-110. PMLR, 2022.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost. Advances in Neural Information Processing Systems , 32, 2019.

Nicolo Cesa-Bianchi, Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other adaptive adversaries. Advances in Neural Information Processing Systems , 26, 2013.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Jinglin Chen, Aditya Modi, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. On the statistical efficiency of reward-free exploration in non-linear rl. arXiv preprint arXiv:2206.10770 , 2022.

Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning. In Advances in Neural Information Processing Systems (NIPS) , 2015.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 30, 2017.

Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards accountable reinforcement learning. In International Conference on Machine Learning , pages 1507-1516, 2019.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

Yaqi Duan, Chi Jin, and Zhiyuan Li. Risk bounds and rademacher complexity in batch reinforcement learning. In International Conference on Machine Learning , pages 2892-2902. PMLR, 2021.

John Duchi, Feng Ruan, and Chulhee Yun. Minimax bounds on stochastic batched convex optimization. In Conference On Learning Theory , pages 3065-3162. PMLR, 2018.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Dylan J Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. arXiv preprint arXiv:2111.10919 , 2021.

Minbo Gao, Tianle Xie, Simon S Du, and Lin F Yang. A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 , 2021.

Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32, 2019.

Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, Jiayu Yao, Isaac Lage, Christopher Mosch, Li wei H. Lehman, Matthieu Komorowski, Matthieu Komorowski, Aldo Faisal, Leo Anthony Celi, David Sontag, and Finale Doshi-Velez. Evaluating reinforcement learning algorithms in observational health settings, 2018.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, and Tie-Yan Liu. Towards deploymentefficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450 , 2022.

Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems , 33:2747-2758, 2020.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems , pages 4863-4873, 2018.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020a.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning (ICML) , 2020b.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? arXiv preprint arXiv:2012.15085 , 2020c.

Anders Jonsson, Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Edouard Leurent, and Michal Valko. Planning in markov decision processes with gap-dependent sample complexity. Advances in Neural Information Processing Systems , 33:1253-1263, 2020.

Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research , 2022.

Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory , pages 865-891. PMLR, 2021.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Modelbased offline reinforcement learning. arXiv preprint arXiv:2005.05951 , 2020.

BRavi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences , 117(14):7684-7689, 2020.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Gen Li, Yuling Yan, Yuxin Chen, and Jianqing Fan. Optimal reward-agnostic exploration in reinforcement learning. 2023a.

Gen Li, Wenhao Zhan, Jason D Lee, Yuejie Chi, and Yuxin Chen. Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. arXiv preprint arXiv:2305.10282 , 2023b.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202 , 2020.

Jihao Long, Jiequn Han, and E Weinan. An l2 analysis of reinforcement learning in high dimensions with kernel and neural network approximation. arXiv preprint arXiv:2104.07794 , 3, 2021.

Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740 , 2009.

Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning , pages 7599-7608. PMLR, 2021.

Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning , pages 6961-6971. PMLR, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Thanh Nguyen-Tang, Ming Yin, Sunil Gupta, Svetha Venkatesh, and Raman Arora. On instancedependent bounds for offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2211.13208 , 2022.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., New York, NY, USA, 1994. ISBN 0471619779.

Dan Qiao and Yu-Xiang Wang. Near-optimal deployment efficiency in reward-free reinforcement learning with linear function approximation. arXiv preprint arXiv:2210.00701 , 2022.

Dan Qiao, Ming Yin, Ming Min, and Yu-Xiang Wang. Sample-efficient reinforcement learning with loglog (t) switching cost. In International Conference on Machine Learning , pages 18031-18061. PMLR, 2022.

Dan Qiao, Ming Yin, and Yu-Xiang Wang. Logarithmic switching cost in reinforcement learning beyond linear mdps. arXiv preprint arXiv:2302.12456 , 2023.

Shuang Qiu, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. On reward-free rl with kernel and neural function approximations: Single-agent mdp and markov game. In International Conference on Machine Learning , pages 8737-8747. PMLR, 2021.

Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist. Machine-learningassisted materials discovery using failed experiments. Nature , 533(7601):73-76, 2016.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv:2103.12021 , 2021.

Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

Yufei Ruan, Jiaqi Yang, and Yuan Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87, 2021.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484, 2016.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT Press, 2018.

Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance-aware regret bounds for undiscounted reinforcement learning in mdps. In Algorithmic Learning Theory , pages 770-805. PMLR, 2018.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage, 2021.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Rewardfree rl is no harder than reward-aware rl in linear markov decision processes. In International Conference on Machine Learning , pages 22430-22456. PMLR, 2022.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Ruosong Wang, Simon S Du, Lin Yang, and Russ R Salakhutdinov. On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems , 33:17816-17826, 2020a.

Ruosong Wang, Dean P Foster, and Sham M Kakade. What are the statistical limits of offline rl with linear function approximation? arXiv preprint arXiv:2010.11895 , 2020b.

Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems , 34:13524-13536, 2021.

Chenjun Xiao, Ilbin Lee, Bo Dai, Dale Schuurmans, and Csaba Szepesvari. The curse of passive data collection in batch reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 8413-8438. PMLR, 2022.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. arXiv preprint arXiv:2008.04990 , 2020a.

Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence , pages 550-559. PMLR, 2020b.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. arXiv preprint arXiv:2106.06926 , 2021a.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021b.

Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 , 2022.

Zhiyuan Xu, Jian Tang, Jingsong Meng, Weiyi Zhang, Yanzhi Wang, Chi Harold Liu, and Dejun Yang. Experience-driven networking: A deep reinforcement learning based approach. In IEEE INFOCOM 2018-IEEE conference on computer communications , pages 1871-1879. IEEE, 2018.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Accelerating safe reinforcement learning with constraint-mismatched policies, 2021.

Ming Yin and Yu-Xiang Wang. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3948-3958. PMLR, 2020.

Ming Yin, Yu-Xiang Wang, Yaqi Duan, and Mengdi Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near optimal provable uniform convergence in off-policy evaluation for reinforcement learning. arXiv preprint arXiv:2007.03760 , 2020.

Ming Yin, Mengdi Wang, and Yu-Xiang Wang. Offline reinforcement learning with differentiable function approximation is provably efficient. arXiv preprint arXiv:2210.00750 , 2022.

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

Andrea Zanette. When is realizability sufficient for off-policy reinforcement learning? ICML 2023 , 2023.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Andrea Zanette and Martin J. Wainwright. Bellman residual orthogonalization for offline reinforcement learning, 2022. URL https://arxiv.org/abs/2203.12786 .

Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably efficient reward-agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems , 33:11756-11766, 2020.

Andrea Zanette, Kefan Dong, Jonathan N Lee, and Emma Brunskill. Design of experiments for stochastic contextual linear bandits. Advances in Neural Information Processing Systems , 34: 22720-22731, 2021a.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:1362613640, 2021b.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason D Lee. Offline reinforcement learning with realizability and single-policy concentrability. arXiv preprint arXiv:2202.04634 , 2022.

Ruiqi Zhang, Xuezhou Zhang, Chengzhuo Ni, and Mengdi Wang. Off-policy fitted q-evaluation with differentiable function approximators: Z-estimation and inference theory. In International Conference on Machine Learning , pages 26713-26749. PMLR, 2022.

Xuezhou Zhang, Yuzhe Ma, and Adish Singla. Task-agnostic exploration in reinforcement learning. Advances in Neural Information Processing Systems , 33:11734-11743, 2020a.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learningvia reference-advantage decomposition. Advances in Neural Information Processing Systems , 33: 15198-15207, 2020b.

Zihan Zhang, Simon Du, and Xiangyang Ji. Near optimal reward-free reinforcement learning. In International Conference on Machine Learning , pages 12402-12412. PMLR, 2021.

Contents

D Technical lemmas and proofs 40

E Details of the planning phase 42

F More related works 42

A Proof of the main result

A.1 Definitions

In this section, we define some crucial concepts that will be used in the proof of the main result.

A.1.1 Sparsified MDP

First, we restate the definition 4.1 in the main text.

Definition A.1 (Sparsified MDP) . Let s † be an absorbing state, i.e., such that P ( s † | s † , a ) = 1 and r ( s † , a ) = 0 for all a ∈ A . The state space in the sparsified MDP M † is defined as that of the original MDP with the addition of s † . The dynamics P † of the sparsified MDP are defined as

̸

For any deterministic reward function r : S × A → [0 , 1] , the reward function on the sparsified MDP is defined as r † ( s, a ) = r ( s, a ) ; for simplicity we only consider deterministic reward functions.

In the offline phase of our algorithm, we simulate the virtual episodes from the empirical version of sparsified MDP (See Algorithm 2). Now we formally this MDP.

Definition A.2 (Emprical sparsified MDP) . Let s † be the absorbing state defined in the sparsified MDP. The state space in the empirical sparsified MDP ̂ M † is defined as that of the original MDP with the addition of s † . The dynamics ̂ P † of the sparsified MDP are defined as

̸

For any deterministic reward function r : S × A → [0 , 1] , the reward function on the empirical sparsified MDP is defined as r † ( s, a ) = r ( s, a ) ; for simplicity we only consider deterministic reward functions. Here, the counters N ( s, a ) and N ( s, a, s ′ ) are the number of ( s, a ) and ( s, a, s ′ ) in the offline data, respectively.

Finally, in the planning phase, after we interact with the true environment for many online episodes, construct a fine-estimated sparsified MDP, which is used to extract the optmal policy of given reward functions. We formally define it below.

Definition A.3 (Fine-estimated sparsified MDP) . Let s † be the absorbing state defined in the sparsified MDP. The state space in the fine-estimated sparsified MDP ˜ M † is defined as that of the original MDP with the addition of s † . The dynamics ˜ P † of the sparsified MDP are defined as

̸

For any deterministic reward function r : S × A → [0 , 1] , the reward function on the fine-estimated sparsified MDP is defined as r † ( s, a ) = r ( s, a ) ; for simplicity we only consider deterministic reward functions. Here, the counters N ( s, a ) and N ( s, a, s ′ ) are the number of ( s, a ) and ( s, a, s ′ ) in the offline data, respectively, while m ( s, a ) and m ( s, a, s ′ ) are the counters of ( s, a ) and ( s, a, s ′ ) in online episodes.

A.1.2 High probability events

In this section, we define all high-probability events we need in order to make our theorem to hold. Specifically, we define

where

Here, KL( · ; · ) denotes the Kullback-Leibler divergence between two distributions. K ucb is the number of episodes of the sub-routine RF-UCB and π i is the policy executed at the i-th episode of RF-UCB (See algorithm 2). K de is the number of episodes executed in the online phase. n k ( s, a ) is the counter of ( s, a ) before the beginning of the k -th episode in the offline phase and m ( s, a ) is the number of ( s, a ) samples in the online data (See definitions in section A.1). We denote d π,h ( s, a ) , d † π,h ( s, a ) and ̂ d † π,h ( s, a ) as the occupancy measure of ( s, a ) at stage h under policy π, on P (the true transition dynamics), P † (the transition dynamics in the sparsfied MDP) and ̂ P † (the transition dynamics in the empirical sparsified MDP) respectively.

For the event defined above, we have the following guarantee, which is proved in appendix A.3.1.

Lemma A.4. For δ > 0 , we have

happens with probability at least 1 -δ,

A.1.3 Uncertainty function and empirical uncertainty function

In this section, we restate the definition of uncertainty function and empirical uncertainty functions, as well as some intermediate uncertainty functions which will be often used in the proof. First, we restate the definition of bonus function.

Definition A.5 (Bonus Function) . We define

Further, we define

Next, we restate the definition of uncertainty function and empirical uncertainty function. Notice that the uncertainty function does not depend on reward function or specific policy π .

Definition A.6 (Uncertainty function) . We define for any ( h, s, a ) ∈ [ H ] ×S×A and any deterministic policy π,

where we specify

In addition, we define X h ( s † , a ) = 0 for any h ∈ [ H ] , a ∈ A . Here, the notation ∑ s ′ above means summation over s ′ ∈ S ∪ { s † } . But since X h ( s † , a ) = 0 for any h ∈ [ H ] , a ∈ A , this is equivalent to the summation over s ′ ∈ S . ˜ P † is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.

Then, for the reader to better understand the proof, we define some intermediate quantity, which also measure the uncertainty of value estimation, but they may be reward or policy dependent.

Definition A.7 (Intermediate uncertainty function) . We define for any ( h, s, a ) ∈ [ H ] ×S × A and any deterministc policy π,

where

In addition, we define W π h ( s † , a, r ) = 0 for any h ∈ [ H ] , a ∈ A and any reward function r . Here, ˜ P † is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.

The intermediate function W is reward and policy dependent and can be used to upper bound the value estimation error. Further, we need to define some policy-dependent version of the uncertainty function, denoted as X π h ( s, a ) , as well as another quantity Y π h ( s, a, r ) .

Definition A.8. We define X π H +1 ( s, a ) = Y π H +1 ( s, a, r ) = 0 for any π, r, s, a and

where

In addition, we define X π h ( s † , a ) = Y π h ( s † , a, r ) = 0 for any a ∈ A , h ∈ [ H ] and any reward r, any policy π. Here, ˜ P † is the transition probability of fine-estimated sparsified MDP defined in section A.1.1.

Finally, we define the empirical uncertainty functions.

Definition A.9. We define U k H +1 ( s, a ) = 0 for any k ∈ [ K ucb ] and s ∈ S , a ∈ A . Further, we define

where ϕ is the bonus function defined in eq. (A.4) and n k ( s, a ) is the counter of the times we encounter ( s, a ) until the beginning of the k -the episode when running the sub-routine RF-UCB in the offline phase.

A.2 Proof of the key theorems and lemmas

A.2.1 Uncertainty Functions upper bounds the estimation error

Proof. This proof follows closely from the techniques in [Ménard et al., 2021], but here we consider the homogeneous MDP. We let ˜ d † π,h ( s, a ) be the probability of encountering ( s, a ) at stage h when running policy π on ˜ P † , starting from s 1 . Now we assume E happens and fix a reward function r and policy π. From lemma A.14, we know

where W π h ( s, a, r ) is the intermediate uncertainty function defined in definition A.7. Here, we use the policy-dependent version of uncertainty function W π ( s, a, r ) , which will then upper bounded using another two policy-dependent quantities X π h ( s, a ) and Y π h ( s, a, r ) . We define these policydependent uncertainty functions to upper bound the estimation error of specific policy, but since we are considering a reward-free setting, which entails a low estimation error for any policy and reward, these quantities cannot be directly used in the algorithm to update the policy. Next, we claim

where X π h ( s, a ) and Y π h ( s, a, r ) are defined in definition A.8. Actually, this is easy from the definition of X π h ( s, a ) , Y π h ( s, a, r ) and W π h ( s, a, r ) . For h = H + 1 , this is trivial. Assume this is true for h +1 , then the case of h is given by the definition and the fact that min { x, y + z } ≤ min { x, y } + z for any x, y, z ≥ 0 . Therefore, we have

Next, we eliminate the dependency to policy π and obtain an upper bound of estimation error using the policy-independent uncertainty function X h ( s, a ) . This is done by bounding Y π 1 ( s 1 , π 1 ( s 1 ) , r ) by X π h ( s, a ) and then upper bounding X π h ( s, a ) by X h ( s, a ) . From definition A.8, the Cauchy-Schwarz inequality and the fact that r ∈ [0 , 1] , we have for any reward function and any deterministic policy π,

We notice that the right hand side of the last inequality above is the policy value of some specific reward function when running π on ˜ P . Concretely, if the transition probability is ˜ P and the reward function at ( s, a ) is ϕ ( m ( s, a )) , then the state value function at the initial state s 1 is ∑ s,a ∑ H h =1 ˜ d † π,h ( s, a ) ϕ ( m ( s, a )) . This specific reward function is non-negative and uniformly bounded by one, so it holds that ∑ s,a ∑ H i = h ˜ d † π,i ( s, a ) ϕ ( m ( s, a )) ≤ H -h +1 . Moreover, from the definition of ϕ and ϕ (eq. (A.4)), we know ϕ ( m ( s, a )) ≤ Hϕ ( m ( s, a )) . Then, from the definition of X h ( s, a ) in definition A.6, we apply an inductive argument to obtain

for any deterministic policy π. So we have

Therefore, combining the last inequality with (A.7), we have

From the definition of X h ( s, a ) (definition A.6) and X π h ( s, a ) (definition A.8), we can see

for any ( h, s, a ) and any deterministic policy π. Therefore, we conclude that

A.2.2 Proof of lemma 6.2

Proof. It suffices to prove that for any k ∈ [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A , it holds that

since the theorem comes from an average of the inequalities above and the fact that (1 + 1 /H ) H ≤ e . For any fixed k, we prove it by induction on h . When h = H +1 , both sides are zero by definition. Suppose the claim holds for h +1 ; we will prove that it also holds for h. We denote K ucb as the

number of virtual episodes in the offline phase. From lemma A.16, the decreasing property of ϕ ( · ) (lemma A.17) and lemma A.21, we have

The inductive hypothesis gives us for any s ′ ,

which implies

Again, by definition of X h ( s, a ) , we have

Combining (A.8) and (A.9), as well as the definition of U k h (definition A.9), we prove the case of stage h and we conclude the theorem by induction.

A.2.3 Upper bounding the empirical uncertainty function (lemma 6.3)

Proof. First, we are going to prove

From the definition (see algorithm 2), we know an important property of uncertainty function is that π k h is greedy with respect to U k h . So we have

Therefore, we have

We apply Lemma D.7 and Jensen's Inequality to obtain

Inserting the last display to the upper bound, we conclude the proof of the upper bound on U . Then, to prove the sample complexity, we use lemma D.8. We take

in Lemma D.8. From lemma D.8, we know there exists a universal constant C 1 such that when x ≥ C 1 B log( B ) 2 , it holds that B log(1 + x ) (1 + log(1 + x )) ≤ x, which implies whenever

it holds that

For notation simplicity, we define

Comparing the left hand side of (A.11) and the right hand side of (A.10), we know

Right hand side of (A.10)

It is easy to see that log( H ) ≤ log (1 + K ucb / |S| ) , so the right hand side of last inequality is upper bounded by

From (A.11), we know this is upper bounded by ε 2 as long as K ucb ≥ 4 L/ε 2 . From the definition of L and equations (A.10) (A.11), we have when K ucb ≥ 4 L/ε 2 , it holds that

In conclusion, this admits a sample complexity of

A.3 Omitted proofs

A.3.1 Proof for lemma A.4 (high probability event)

The lemma A.4 is proved by combining all the lemmas below via a union bound. Below, we always denote N ( s, a, s ′ ) as the number of ( s, a, s ′ ) in the offline dataset.

Lemma A.10. E P holds with probability at least 1 -δ/ 6 .

Proof. For any fixed ( s, a, s ′ ) such that N ( s, a, s ′ ) ≥ Φ , we denote n i ( s, a ) as the index of the i-th time when we visit ( s, a ) . For notation simplicity, we fix the state-action pair ( s, a ) here and write n i ( s, a ) as n i simply for i = 1 , 2 , ..., N ( s, a ) . Notice that the total visiting time N ( s, a ) is random, so our argument is based on conditional probability. We denote X i = I ( s ′ n i = s ′ ) as the indicator of whether the next state is s ′ when we visited ( s, a ) at the i-th time. From the data generating mechanism and the definition for the reference MDP, we know conditional on the total number of visiting N ( s, a ) and all the indexes n 1 ≤ ... ≤ n N ( s,a ) , it holds that X 1 , X 2 , ..., X N ( s,a ) are independent Bernoulli random variable with successful probability being P † ( s ′ | s, a ) . We denote X as their arithmetic average. Using Empirical Bernstein Inequality (Lemma D.3), and the fact that 1 N ( s,a ) ∑ N ( s,a ) i =1 ( X i -X ) 2 ≤ 1 N ( s,a ) ∑ N ( s,a ) i =1 X 2 i = 1 N ( s,a ) ∑ N ( s,a ) i =1 X i = ̂ P † ( s ′ | s, a ) , we have

Taking integral of the conditional probability, we conclude that the unconditional probability of the event E P is at least 1 -δ/ 6 . Therefore, we conclude.

Lemma A.11. For δ > 0 , it holds that P ( E 4 ) ≥ 1 -δ/ 6 .

Proof. Consider for a fixed triple ( s, a ) ∈ S × A . Let m ( s, a ) denote the number of times the tuple ( s, a ) was encountered in total during the online interaction phase. We define X i as follows. For i ≤ m ( s, a ) , we let X i be the subsequent state s ′ when ( s, a ) was encountered the i -th time in the whole run of the algorithm. Otherwise, we let X i be an independent sample from P † ( s, a ) . By construction, { X i } is a sequence of i.i.d. categorical random variables from S with distribution P † ( · | s, a ) . We denote ˜ P † ,i X = 1 i ∑ i j =1 δ X j as the empirical probability mass and P † X as the probability mass of X i . Then, from Lemma D.4, we have with probability at least 1 -δ/ 6 , it holds that for any i ∈ N ,

which implies

Using a union bound for ( s, a ) ∈ S × A , we conclude.

Lemma A.12 (Lower Bound on Counters) . For δ > 0 , it holds that P ( E 2 ) ≥ 1 -δ/ 6 and P ( E 5 ) ≥ 1 -δ/ 6 .

Proof. We denote n k h as the number of times we encounter ( s, a ) at stage h before the beginning of episode k. Concretely speaking, we define n 1 h ( s, a ) = 0 for any ( h, s, a ) . Then, we define

We fixed an ( s, a, h ) ∈ [ H ] ×S × A and denote F k as the sigma field generated by the first k -1 episodes when running RF-UCB and X k = I [ ( s k h , a k h ) = ( s, a ) ] . Then, we know X k is F k +1 -measurable and E [ X k | F k ] = ̂ d † π k ,h ( s, a ) is F k -measurable. Taking W = ln (6 /δ ) in Lemma D.1 and applying a union bound, we know with probability 1 -δ/ 6 , the following event happens:

To finish the proof, it remains to note that the event above implies the event we want by summing over h ∈ [ H ] for each k ∈ N and each ( s, a ) ∈ S × A . For the E 5 , the proof is almost the same.

Lemma A.13 (Upper Bound on Counters) . For δ > 0 , it holds that P ( E 3 ) ≥ 1 -δ/ 6 .

Proof. We denote n k h as the number of times we encounter ( s, a ) at stage h before the beginning of episode k. Concretely speaking, we define n 1 h ( s, a ) = 0 for any ( h, s, a ) . Then, we define

We fixed an ( s, a, h ) ∈ [ H ] ×S × A and denote F k as the sigma field generated by the first k -1 episodes when running RF-UCB and X k = I [ ( s k h , a k h ) = ( s, a ) ] . Then, we know X k is F k +1 -measurable and E [ X k | F k ] = ̂ d † π k ,h ( s, a ) is F k -measurable. Taking W = ln (6 /δ ) in Lemma D.2 and applying a union bound, we know with probability 1 -δ/ 6 , the following event happens:

To finish the proof, it remains to note that the event above implies the event we want by summing over h ∈ [ H ] for each k ∈ N and each ( s, a ) ∈ S × A .

A.3.2 Proof for lemma A.14 (property of intermediate uncertainty function)

Lemma A.14 (Intermediate Uncertainty Function) . We define for any ( h, s, a ) ∈ [ H ] ×S × A and any deterministc policy π,

where

In addition, we define W π h ( s † , a, r ) = 0 for any h ∈ [ H ] , a ∈ A and any reward function r . Then, under E , for any ( h, s, a ) ∈ [ H ] ×S × A , any deterministic policy π, and any deterministic reward function r (with its augmentation r † ), it holds that

Proof. We prove by induction. For h = H +1 , we know W H +1 ( s, a, r ) = 0 by definition and the left hand side of the inequality we want also vanishes. Suppose the claim holds for h +1 and for any s, a. The Bellman equation gives us

Under E , we know that KL ( ˜ P † ; P † ) ≤ 1 H ϕ ( m ( s, a )) for any ( s, a ) , so from Lemma D.5 we have

where ϕ is the bonus defined as (A.4). Here, we let f in Lemma D.5 be V h +1 ( s ′ ; P † , r † , π ) . So the range will be H -h and the upper bound for KL divergence is given by high-probability event E . We further apply Lemma D.6 to get

Therefore, we have

Using the fact that x + y ≤ √ x + √ y and √ xy ≤ x + y 2 for positive x, y, and the definition of II, we obtain

which implies

If Hϕ ( m ( s, a )) ≤ 1 , then we have by definition

by definition. Otherwise, if Hϕ ( m ( s, a )) ≥ 1 , we have

Therefore, we have in either case

The induction hypothesis gives us that

Inserting this into the upper bound of ∆ h , we prove the case of h by the definition of W π h ( s, a, r ) and the simple fact that | ∆ h | ≤ H -h +1 .

A.3.3 Proof for lemma A.15 and lemma A.16 (properties of uncertainty function)

In this section, we prove some lemmas for upper bounding the uncertainty functions X h ( s, a ) . We first provide a basic upper bound for X h ( s, a ) . The uncertainty function is defined in definition A.6. Lemma A.15. Under the high probability event E (defined in appendix A.1.2) for all ( h, s, a ) ∈ [ H ] ×S × A , it holds that

In addition, for s = s † , from definition, we know the above upper bound naturally holds.

Proof. For any fixed ( h, s, a ) ∈ [ H ] ×S × A , from the definition of the uncertainty function, we know

To prove the lemma, it suffices to bound the difference between ˜ P † ( · | s, a ) ⊤ (max a ′ X h +1 ( · , a ′ )) and P † ( · | s, a ) ⊤ (max a ′ X h +1 ( · , a ′ )) . Under E (which happens with probability at least 1 -δ ), it holds that KL ( ˜ P † h ( · | s, a ) ; P † ( · | s, a ) ) ≤ 1 H ϕ ( m ( s, a )) . Applying Lemma D.5 and a simple bound for variance gives us

The last line comes from √ ab ≤ a + b 2 for any positive a, b, and the fact that 1 2 Hϕ ( m ( s, a )) + 2 3 ϕ ( m ( s, a )) ≤ 2 Hϕ ( m ( s, a )) . Insert this bound into the definition of X h ( s, a ) to obtain

Noticing that X ( s, a ) ≤ H -h +1 ≤ 11 H,

Lemma A.16. There exists a universal constant C ≥ 1 such that under the high probability event E , when 3 K ucb ≥ K de , we have for any ( h, s, a ) ∈ [ H ] ×S × A , it holds that

In addition, for s = s † , from definition, we know the above upper bound naturally holds.

Proof. Here, the universal constant may vary from line to line. Under E , we have

Since X h ( s, a ) ≤ ( H -h ) ≤ CH K ucb K de , we can modify the last display as

Therefore, we conclude.

A.3.4 Proof for lemma A.17, lemma A.18, lemma A.19, lemma A.20 (properties of bonus function)

For our bonus function, we have the following basic property.

Lemma A.17. ϕ ( x ) is non-increasing when x > 0 . For any α ≤ 1 , we have ϕ ( αx ) ≤ 1 α ϕ ( x ) .

Proof. We define f ( x ) := 1 x [ C + D log ( e ( 1 + x D ))] , where C, D ≥ 1 . Then, for x > 0 ,

Taking C = log (6 H |S| |A| δ ) and D = |S| , we conclude thee first claim. For the second claim, it is trivial since the logarithm function is increasing.

Lemma A.18. Under E , we have for any ( s, a ) ∈ S × A ,

Proof. For fixed ( s, a ) ∈ S × A , when H ln (6 H |S| |A| /δ ) ≤ 1 4 ( K de ∑ h ∈ [ H ] w mix h ( s, a ) ) , we know that E 5 implies m ( s, a ) ≥ 1 4 ∑ h ∈ [ H ] K de w mix h ( s, a ) , From Lemma A.17, we know

When H ln (6 H |S| |A| /δ ) > 1 4 ( K de ∑ h ∈ [ H ] w mix h ( s, a ) ) , simple algebra shows that

Therefore, we conclude.

Lemma A.19. Under E , we have for any ( s, a ) ∈ S × A ,

Proof. We consider under the event E 3 , it holds that for any ( s, a ) ∈ S × A ,

If H ln (6 H |S| |A| /δ ) ≤ 1 2 n K ucb ( s, a ) , then we have ∑ K ucb k =1 ∑ H h =1 ̂ d † π k ,h ( s, a ) ≥ 1 4 n K ucb ( s, a ) , which implies

Otherwise, if H ln (6 H |S| |A| /δ ) > 1 2 n K ucb ( s, a ) , then we have

Combining the two cases above, we conclude.

Lemma A.20. Under E , we have for any k ∈ [ K ucb ] and any ( s, a ) ∈ S × A ,

Proof. Under E 2 , it holds that

If H ln (6 H |S||A| /δ ) ≤ 1 4 ∑ i<k ∑ h ∈ [ H ] ̂ d † π i ,h ( s, a ) , then n k ( s, a ) ≥ 1 4 ∑ i<k ∑ h ∈ [ H ] ̂ d † π i ,h ( s, a ) and hence,

This result equals to the right hand side in the lemma, because ∑ i<k ∑ h ∈ [ H ] ̂ d † π i ,h ( s, a ) ≥ 4 H ln (6 H |S||A| /δ ) ≥ 1 (so taking maximum does not change anything). Otherwise, if H ln (6 H |S||A| /δ ) > 1 4 ∑ i<k ∑ h ∈ [ H ] ̂ d † π i ,h ( s, a ) , then

Since 1 ≤ 4 H ln (6 H |S||A| /δ ) , we have

The last inequality comes from simple algebra. Therefore, we conclude.

A.3.5 Proof for lemma A.21 and lemma A.22 (properties of empirical sparsified MDP)

In this section, we state two important properties of the empirical sparsified MDP and prove them. We remark that we do not include P ( s † | s, a ) in these two lemmas, since by definition s † / ∈ S .

Lemma A.21 (Multiplicative Accuracy) . We set

Then, when H ≥ 2 , under E P , we have for any ( s, a, s ′ ) ∈ S × A × S ,

Proof. For N ( s, a, s ′ ) < Φ , both sides are zero. For N ( s, a, s ′ ) ≥ Φ , recall ̂ P † ( s ′ | s, a ) = N ( s,a,s ′ ) N ( s,a ) , then from lemma A.10, with probability at least 1 -δ ,

∣ ∣ ∣ ̂ P † ( s ′ | s, a ) -P † ( s ′ | s, a ) ∣ ∣ ∣ ≤ √ √ √ √ 2 ̂ P † ( s ′ | s, a ) N ( s, a ) log ( 12 |S| 2 |A| δ ) + 14 3 N ( s, a ) log ( 12 |S| 2 |A| δ ) (lemma A.10 and definition of E P ) =   √ √ √ √ 2 N ( s, a, s ′ ) log ( 12 |S| 2 |A| δ ) + 14 3 N ( s, a, s ′ ) log ( 12 |S| 2 |A| δ )   · ̂ P † ( s ′ | s, a ) ( ̂ P † ( s ′ | s, a ) = N ( s,a,s ′ ) N ( s,a ) ) ≤ [√ 1 3 H 2 + 7 9 H 2 ] · ̂ P † ( s ′ | s, a ) ≤ ̂ P † ( s ′ | s, a ) H , where the second line comes from the lower bound on Φ . We conclude.

Lemma A.22 (Bound on Ratios of Visitation Probability) . For any deterministic policy π and any ( h, s, a ) ∈ [ H ] ×S × A , it holds that

Here, recall that we denote d † π,h ( s, a ) and ̂ d † π,h ( s, a ) as the occupancy measure of ( s, a ) at stage h under policy π, on P † (the transition dynamics in the sparsfied MDP) and ̂ P † (the transition dynamics in the empirical sparsified MDP) respectively.

We remark that for s † ̸∈ S the inequality does not necessarily hold.

Proof. We denote T h,s,a as all truncated trajectories ( s 1 , a 1 , s 2 , a 2 , ..., s h , a h ) up to stage h such that ( s h , a h ) = ( s, a ) . Notice that if τ h = ( s 1 , a 1 , s 2 , a 2 , ..., s h , a h ) ∈ T h,s,a , then it holds that s i = s † for 1 ≤ i ≤ h -1 . We denote P ( · ; P ′ , π ) as the probability under a specific transition dynamics P ′ and policy π. For any fixed ( h, s, a ) ∈ [ H ] ×S×A and any fixed τ ∈ T h,s,a , we apply Lemma A.21 to get

̸

and

We conclude by rewriting the visiting probability as

B Additional comparisons

B.1 Comparison with other comparator policy

Our main result compares the sub-optimality of the policy π final against the optimal policy on the sparsified MDP. We can further derive the sub-optimality of our output with respect to any comparator policy on the original MDP M . If we denote π ∗ , π † ∗ and π final as the global optimal policy, the optimal policy on the sparsified MDP and the policy output by our algorithm, respectively, and denote π as the comparator policy, we have

Here, the second term is non-positive from the definition of π † ∗ , the third term is upper bounded by ε due to our main theorem (Theorem 5.1), and the last term is non-positive from the definition of the sparsified MDP. Since the connectivity graph of the sparsified MDP is a sub-graph of the original MDP, for any policy, the policy value on the sparsified MDP must be no higher than that on the original MDP.

At a high level, the ε term in the last line of (B.1) represents the error from the finite online episodes, while the approximation error term V 1 ( s 1 , P , r, π ) -V 1 ( s 1 , P † , r, π ) measures the policy value difference of π on the original MDP and the sparsified one, representing the coverage quality of the offline dataset . If the dataset covers most of what π covers, then this gap should be small. When π is the global optimal policy π ∗ , this means the data should cover the state-actions pairs where optimal policy covers. The approximation error here plays a similar role as the concentrability coefficient in the offline reinforcement learning.

B.2 Comparison with offline reinforcement learning

Our algorithm leverages some information from the offline dataset, so it is natural to ask how we expect that offline dataset to be, compared to traditional offline reinforcement learning does. In offline RL, we typically require the concentrablity condition , namely good coverage for the offline dataset, in order to achieve a polynomial sample complexity. Specifically, if we assume the offline data are sampled by first sampling ( s, a ) i.i.d. from µ and then sampling the subsequent state from the transition dynamics, then the concentrability condition says the following constant C ∗ is well-defined and finite.

The concentrability coefficient can be defined in several alternative ways, either for a set of policies or with respect to a single policy [Chen and Jiang, 2019, Zhan et al., 2022, Xie et al., 2021b, Zanette et al., 2021b]. Here, we follow the definition in [Xie et al., 2021b]. This means, the sampling distribution must covers the region where the global optimal policy covers, which is a very similar intuition obtained from our setting.

[Xie et al., 2021b] also gave optimal sample complexity (in terms of state-action pairs) for an offline RL algorithm is

which is minimax optimal up to logarithm terms and higher order terms. Similar sample complexity were also given in several literature [Yin and Wang, 2020, Yin et al., 2020, Xie and Jiang, 2020b].

Uniform data distribution For simplicity, we first assume µ to be uniform on all state-action pairs and the reward function to be given. Consider we have N state-action pairs in the offline data, which are sampled i.i.d. from the distribution µ . Notice that here, the global optimal policy π ∗ still differs from the optimum on the sparsified MDP π † ∗ , since even if we get enough samples from each ( s, a ) pairs, we might not get enough samples for every ( s, a, s ′ ) and hence, not all ( s, a, s ′ ) will be included in the set of known tuples.

Concretely, if we consider the case when we sample each state-action pair for N/ ( |S| |A| ) times and simply treat the transition frequency as the true probability, then for any N ( s, a, s ′ ) < Φ , it holds that P ( s ′ | s, a ) = N ( s,a,s ′ ) N ( s,a ) = N ( s,a,s ′ ) |S||A| N ≤ Φ |S||A| N . So for any any N ( s, a, s ′ ) ≥ Φ , we know P ( s ′ | s, a ) = P † ( s ′ | s, a ); while for any N ( s, a, s ′ ) < Φ , we have P ( s ′ | s, a ) ≤ Φ |S||A| N and P † ( s ′ | s, a ) = 0 . Therefore, we have

From the value difference lemma (lemma D.11), we can upper bound the approximation error by

Therefore, to get an ε -optimal policy compared to the global optimal one, we need the number of state-action pairs in the initial offline dataset D to be

From the theorem 5.1, the offline data size we need here is actually significantly smaller than what we need for an offline algorithm. As long as sup ( s,a ) d π ∗ ( s, a ) is not too small, for instance, larger than H -1 . 5 , then we shave off the whole 1 /ε 2 term. The order of offline sample complexity here is actually O (1 /ε ) instead of O (1 /ε 2 ) typical in offline RL, and this is significantly smaller in small ε regime. To compensate the smaller offline sample size, actually we need more online sample to obtain an globally ε -optimal policy, and we summarize the general requirement for offline and online sample size in corollary 5.2.

Non-uniform data distribution Assume the data generating distribution µ is not uniform but still supported on all ( s, a ) pairs such that d π ∗ ( s, a ) > 0 , so that the concentrability coefficient in offline RL is still well defined. We simply consider the case when each state-action pair ( s, a ) is sampled by Nµ ( s, a ) times and treat the transition frequency as the true underlying probability. Then, following a very similar argument as in the last paragraph, the number of state-action pairs needed in the initial offline dataset in order to extract an ε -globally optimal policy is

Here, the quantity

plays a similar role of classical concentrability coefficient and also measures the distribution shift between two policies. In the worst case, this coefficient can be |S| |A| C ∗ , resulting in an extra |S| |A| factor compared to the optimal offline sample complexity. However, we still shave off the entire 1 /ε 2 term and also shave off H 1 . 5 in the 1 /ε term.

Partial coverage data Under partial coverage, we expect the output policy π final to be competitive with the value of the best policy supported in the region covered by the offline dataset. In such case, theorem 5.1 provides guarantees with the best comparator policy on the sparsified MDP M † . In order to gain further intuition, it is best to 'translate' such guarantees into guarantees on M .

In the worst case, the data distribution µ at a certain ( s, a ) pair can be zero when d π ( s, a ) > 0 , which implies the concentrability coefficient C ∗ = ∞ . Here, π is an arbitrary comparator policy. In this case, either classical offline RL algorithm or our policy finetuning algorithm cannot guarantee an ε -optimal policy compared to the global optimal policy. However, we can still output a locally ε -optimal policy, compared to the optimal policy on the sparsified MDP.

In order to compare π final to any policy on the original MDP, we have the corollary 5.2, which will be proved in appendix B.3.

The statement in corollary 5.2 is a quite direct consequence of theorem 5.1, and it expresses the sub-optimality gap of π final with respect to any comparator policy π on the original MDP M . It can also be written in terms of the sub-optimality: If we fix a comparator policy π , then with probability at least 1 -δ , for any reward function r , the policy π final returned by algorithm 2 satisfies:

where K de is the number of online episodes and N is the number of state-action pairs in offline data. Here, the sub-optimality depends on an online error as well as on an offline error . The online error is the one that also arises in the statement of theorem 5.1. It is an error that can be reduced by collecting more online samples, i.e., by increasing K , with the typical inverse square-root depedence 1 / √ K .

However, the upper bound suggests that even in the limit of infinite online data, the value of π final will not approach that of π ∗ because of a residual error due to the offline dataset D . Such residual error depends on certain concentrability factor expressed as ∑ s,a d π ( s,a ) µ ( s,a ) ≤ |S||A| sup s,a d π ( s,a ) µ ( s,a ) , whose presence is intuitive: if a comparator policy π is not covered well, our algorithm does not have enough information to navigate to the area that π tends to visit, and so it is unable to refine its estimates there. However the dependence on the number of offline samples N = |D| is through its inverse , i.e., 1 /N as opposed to the more typical 1 / √ N : such gap represents the improvable performance when additional online data are collected non-reactively.

It is useful to compare corollary 5.2 with what is achievable by using a minimax-optimal online algorithm[Xie et al., 2021b]. In this latter case, one can bound the sub-optimality gap for any comparator policy π with high probability as

B.3 Proof of corollary 5.2

Let's denote D = { ( s i , a i , s ′ i ) } i ∈ [ N ] as the offline dataset, where N as the total number of tuples. We keep the notation the same as in the main text. We use N ( s, a ) and N ( s, a, s ′ ) to denote the counter of ( s, a ) and ( s, a, s ′ ) in the offline data D . The state-action pairs are sampled i.i.d. from µ ( s, a ) and the subsequent states are sampled from the transition dynamics. We fix a comparator policy π and assume µ ( s, a ) > 0 for any ( s, a ) such that d π ( s, a ) > 0 , which implies a finite concentrability constant C ∗ . Here, d π ( s, a ) is the occupancy probability of ( s, a ) when executing policy π, averaged over all stages h ∈ [ H ] .

Similar to the Section B.1, we have

where π † ∗ and π final are the optimal policy on the sparsified MDP and the policy output by our algorithm, respectively, and π is the fixed comparator policy. Here, the second term is non-positive from the definition of π † ∗ , the last term is non-positive from the definition of the sparsified MDP . This is because, for any state-action pair ( s, a ) and any fixed policy π, the probability of reaching ( s, a ) under P † will not exceed that under the true transition probability P . If we denote the visiting probability under π and P (or P † resp.) as d π,h ( s, a ) ( d † π,h ( s, a ) resp.), then we have

for any h ∈ [ H ] , s ∈ S , a ∈ A . Note that, for s = s † , this does not hold necessarily. Them, for any policy π, we have

̸

̸

Therefore, we get V 1 ( s 1 , P † , r, π final ) -V 1 ( s 1 , P , r, π final ) ≤ 0 when we take π = π final . From the main result (Theorem 5.1), we know the estimation error is bounded as

where K de is the number of online episodes. Therefore, to make the right hand side of (B.4) less than ε/ 2 , one needs at least ˜ O ( H 2 |S||A| ε 2 ) online episodes. This is exactly what the main result shows.

So it suffices to bound the approximation error term. From the value difference lemma (Lemma D.11), we have

We define

̸

as the average visiting probability. Then, we have

̸

So it suffices to upper bound ∣ ∣ P † ( s ′ | s, a ) -P ( s ′ | s, a ) ∣ ∣ . Notice here we only consider s = s † and s ′ = s † , since the value function starting from s † is always zero.

For ( s, a, s ′ ) , if N ( s, a, s ′ ) ≥ Φ , it holds that P † ( s ′ | s, a ) = P ( s ′ | s, a ) . Otherwise, from the definition, we know P † ( s ′ | s, a ) = 0 , so it suffices to bound P ( s ′ | s, a ) in this case. From lemma B.1, we know that with probability at least 1 -δ/ 2 , for any ( s, a, s ′ ) such that N ( s, a, s ′ ) < Φ , it holds that

Then we deal with two cases. When Nµ ( s, a ) ≥ 6Φ , from lemma B.2 we have

From the definition of Φ , we know that 2 log ( 4 |S| 2 |A| /δ ) ≤ Φ and 2 log (2 |S| |A| /δ ) ≤ Φ , which implies

The last inequality comes from our assumption for Nµ ( s, a ) ≥ 6Φ .

In the other case, when Nµ ( s, a ) < 6Φ , it holds that

for any ( s, a, s ′ ) . Therefore, for any for any ( s, a, s ′ ) such that N ( s, a, s ′ ) < Φ , we have

Combining equations (B.7) and (B.6), we know for any comparator policy π, it holds that

where N is the total number of transitions in the offline data. In order to make the right hand side of last display less than ε/ 2 , one needs at least

offline transitions. Combining the proof for estimation error and approximation error, we conclude.

B.4 Proof for lemma B.1 and lemma B.2

Lemma B.1. With probability at least 1 -δ/ 2 , for any ( s, a, s ′ ) ∈ S × A × S , it holds that

Proof. Wefixed ( s, a, s ′ ) and denote I as the index set where ( s i , a i ) = ( s, a ) for i ∈ I. Werange the indexes in I as i 1 < i 2 < ... < i N ( s,a ) . For j ≤ N ( s, a ) , we denote X j = I ( s ′ i j = s ′ ) , which is the indicator of whether the next state is s ′ when we encounter ( s, a ) the j -th time. When j ≥ N ( s, a ) , we denote X j as independent Bernoulli random variables with successful rate P ( s ′ | s, a ) . Then, we know X j for all j ∈ N are i.i.d. sequence of Bernoulli random variables. From Lemma D.1, we know with probability at least 1 -δ/ 2 , for any positive integer n , it holds that

Wetake n = N ( s, a ) (although N ( s, a ) is random, we can still take it because for any n the inequality above holds) to get

Applying a union bound for all ( s, a, s ′ ) , we conclude.

Lemma B.2. With probability at least 1 -δ/ 2 , for any ( s, a ) ∈ S × A , it holds that

Proof. If µ ( s, a ) = 0 , this is trivial. We fixed an ( s, a ) such that µ ( s, a ) > 0 . For j ≤ N, we denote X j = I ( s j = s, a j = a ) , which is the indicator of whether the j -th state-action pair we encounter in the offline dataset is ( s, a ) . When j ≥ N, we denote X j as independent Bernoulli random variables with successful rate µ ( s, a ) . Then, we know X j for all j ∈ N are i.i.d. sequence of Bernoulli random variables. From Lemma D.1, we know with probability at least 1 -δ/ 2 , for any positive integer n , it holds that

We take n = N to get

Applying a union bound for all ( s, a ) such that µ ( s, a ) > 0 , we conclude.

C Lower bound

In this section we briefly discuss the optimality of the algorithm. Although the following considerations are also mentioned in the main text, here we mention how they naturally lead to a lower bound.

Lower bound for reward-free exploration Consider the MDP class M defined in the proof of the lower bound of Theorem 4.1 in [Jin et al., 2020b]. Assume that the dataset arises from a logging policy π log which induces the condition N ( s, a, s ′ ) ≥ Φ for all ( s, a, s ′ ) ∈ S × A for every instance of the class. In this case, every MDP instance M∈M and its sparsified version M † coincide. Then the concatenation of the logging policy π log and of the policy π final produced by our algorithm (i.e., algorithm 3) can be interpreted as a reactive policy, which must comply with the reward free lower bound established in Theorem 4.1 of [Jin et al., 2020b]. More precisely, the reward-free sample complexity lower bound established in Theorem 4.1 in [Jin et al., 2020b] is

trajectories. This matches the sample complexity of theorem 5.1. Notice that the number of samples originally present in the dataset can be

a term independent of the accuracy ε . Given that when ϵ ≤ 1 we have

our algorithm is unimprovable beyond constant terms and logarithmic terms in a minimax sense.

Lower bound for non-reactive exploration Consider the MDP class M defined in the proof of the lower bound in Theorem 1 of [Xiao et al., 2022]. It establishes an exponential sample complexity for non-reactive exploration when no prior knowledge is available . In other words, in absence of any data about the MDP, non-reactive exploration must suffer an exponential sample complexity. In such case, our theorem 5.1 (correctly) provides vacuous guarantees, because the sparsified MDP M is degenerate (all edges lead to the absorbing state).

Combining the two constructions It is possible to combine the MDP class M 1 from the paper [Xiao et al., 2022] with the MDP class M 2 from the paper [Jin et al., 2020b]. In lieu of a formal proof, here we provide only a sketch of the construction that would induce a lower bound. More precisely, consider a starting state s 1 where only two actionsa = 1 and a = 2 -are available. Taking a = 1 leads to the start state of an instance of the class M 1 , while taking a = 2 leads to the start state of an instance of the class M 2 ; in both cases the transition occurs with probability one and zero reward is collected.

Furthermore, assume that the reward function given over the MDPs in M 1 is shifted such that the value of a policy that takes a = 1 in s 1 and then plays optimally is 1 and that the reward functions on M 2 is shifted such that the value of a policy which takes a = 2 initially and then playes optimally is 2 ε .

In addition, assume that the dataset arises from a logging policy π log which takes a = 2 initially and then visits all ( s, a, s ′ ) uniformly.

Such construction and dataset identify a sparsified MDP which coincide with M 2 with the addition of s 1 (and its transition to M 2 with zero reward). Intuitively, a policy with value arbitrarily close to 1 must take action a = 1 which leads to M 1 , which is the portion of the MDP that is unexplored in the dataset. In this case, unless the agent collects exponentially many trajectories in the online phase, the lower bound from [Xiao et al., 2022] implies that it is not possible to discover a policy with value close to 1 (e.g., larger than 1 / 2 ). On the other hand, our theorem 5.1 guarantees that π final has a value at least ε , because π final is ε -optimal on the sparsified MDP-i.e., ε -optimal when restricted to an instance on M 2 -with high probability using at most ∼ H 2 |S| 2 |A| /ε 2 trajectories. This value is unimprovable given the lower bound of Jin et al. [2020b], which applies to the class M 2 .

For completeness, in the next sub-section we refine the lower bound of [Xiao et al., 2022] to handle mixture policies.

D Technical lemmas and proofs

Lemma D.1 (Lemma F.4 in [Dann et al., 2017]) . Let F i for i = 1 . . . be a filtration and X 1 , . . . X n be a sequence of Bernoulli random variables with P ( X i = 1 | F i -1 ) = P i with P i being F i -1 -measurable and X i being F i measurable. It holds that

Lemma D.2. Let F i for i = 1 . . . be a filtration and X 1 , . . . X n be a sequence of Bernoulli random variables with P ( X i = 1 | F i -1 ) = P i with P i being F i -1 -measurable and X i being F i measurable. It holds that

Proof. Notice that 1 u 2 [exp( u ) -u -1] is non-decreasing on R , where at zero we continuously extend this function. For any t ∈ N , since X t -P t ≤ 1 , we have exp( X t -P t ) -( X t -P t ) -1 ≤ ( X t -P t ) 2 ( e -2) ≤ ( X t -P t ) 2 . Taking expectation conditional on F t -1 and noticing that P t -X t is a Martingale difference sequence w.r.t. the filtration F t , we have

where the last inequality comes from the fact that conditional on F t -1 , X t is a Bernoulli random variable. We define M n := exp [ ∑ n t =1 ( X t -2 P t )] , which is a supermartingale from our derivation above. We define now the stopping time τ = min { t ∈ N : M t > e W } and the sequence τ n = min { t ∈ N : M t > e W ∨ t ≥ n } . Applying the convergence theorem for nonnegative supermartingales (Theorem 4.2.12 in [Durrett, 2019]), we get that lim t →∞ M t is well-defined almost surely. Therefore, M τ is well-defined even when τ = ∞ . By the optional stopping theorem for nonnegative supermartingales (Theorem 4.8.4 in [Durrett, 2019], we have E [ M τ n ] ≤ E [ M 0 ] ≤ 1 for all n and applying Fatou's lemma, we obtain E [ M τ ] = E [lim n →∞ M τ n ] ≤ lim inf n →∞ E [ M τ n ] ≤ 1 . Using Markov's inequality, we can finally bound

Lemma D.3 (Empirical Bernstein Inequality, Theorem 11 in [Maurer and Pontil, 2009]) . Let n ≥ 2 and x 1 , · · · , x n be i.i.d random variables such that | x i | ≤ A with probability 1 . Let ¯ x = 1 n ∑ n i =1 x i , and ̂ V n = 1 n ∑ n i =1 ( x i -¯ x ) 2 , then with probability 1 -δ we have

Lemma D.4 (Concentration for KL Divergence, Proposition 1 in [Jonsson et al., 2020]) . Let X 1 , X 2 , . . . , X n , . . . be i.i.d. samples from a distribution supported over { 1 , . . . , m } , of probabilities given by P ∈ Σ m , where Σ m is the probability simplex of dimension m -1 . We denote by ̂ P n the empirical vector of probabilities. Then, for any δ ∈ [0 , 1] , with probability at least 1 -δ, it holds that

We remark that there is a slight difference between it and the original version. In [Jonsson et al., 2020], they use m -1 instead of m . But since the second term of the right hand side above is increasing with m, our version also holds.

Lemma D.5 (Bernstein Transportation, Lemma 11 in [Talebi and Maillard, 2018]) . For any function f and any two probability measure Q , P which satisfy Q ≪ P , we denote V P [ f ] := Var X ∼ P ( f ( X )) and S ( f ) := sup x f ( x ) -inf x f ( x ) . We assume V P [ f ] and S ( f ) are finite, then we have

LemmaD.6 (Difference of Variance, Lemma 12 in [Ménard et al., 2021]) . Let P , Q be two probability measure on a discrete sample space of cardinality S . Let f, g be two functions defined on S such that 0 ≤ g ( s ) , f ( s ) ≤ b for all s ∈ S , we have that

Further, if KL( P ; Q ) ≤ α, it holds that

Lemma D.7. For any sequence of numbers z 1 , . . . , z n with 0 ≤ z k ≤ 1 , we have

Proof.

Lemma D.8. For B ≥ 16 and x ≥ 3 , there exists a universal constant C 1 ≥ 4 , such that when x ≥ C 1 B log( B ) 2 , it holds that

Proof. We have

We define f ( x ) := x -B [log(6 x )] 2 , then we have f ′ ( x ) = 1 -2 B log(6 x ) x . Since x ≥ 2 C 1 B log( B ) , we have

We can take C 1 ≥ 1 such that C 1 log( B ) -log(12 C 1 B log( B )) ≥ 0 whenever B ≥ 16 . Therefore, we know f ( x ) is increasing when x ≥ C 1 B log( B ) 2 . Then, it suffices to prove

Since [ log ( 6 C 1 B log( B ) 2 )] 2 ≤ 2 log( B ) 2 +2 [ log ( 6 C 1 log( B ) 2 )] 2 , it suffices to prove

When C 1 ≥ 4 , the difference of right hand side and left hand side s always increasing w.r.t. B for fixed C 1 . Therefore, it suffices to prove the case when B = 16 . Noticing that we can always take a sufficiently large uniform constant C 1 such that the inequality above holds when B = 16 , we conclude.

Lemma D.9 (Chain rule of Kullback-Leibler divergence, Exercise 3.2 in [Wainwright, 2019]) . Given two n -variate distributions Q and P , show that the Kullback-Leibler divergence can be decomposed as

where Q j ( · | X j -1 1 ) denotes the conditional distribution of X j given ( X 1 , . . . , X j -1 ) under Q , with a similar definition for P j ( · | X j -1 1 ) .

Lemma D.10 (Bretagnolle-Huber Inequality, Theorem 14.1 in [Lattimore and Szepesvári, 2020]) . Let P and Q be probability measures on the same measurable space (Ω , F ) , and let A ∈ F be an arbitrary event. Then,

where A c = Ω \ A is the complement of A .

Lemma D.11 (Value Difference Lemma, Lemma E.15 in [Dann et al., 2017]) . For any two MDPs M ′ and M ′′ with rewards r ′ and r ′′ and transition probabilities P ′ and P ′′ , the difference in values with respect to the same policy π can be written as

where V ′ H +1 ( s ) = V ′′ H +1 ( s ) = 0 for any state s and the expectation E ′ is taken with respect to P ′ and π and E ′′ with respect to P ′′ and π .

E Details of the planning phase

In this section, we provide some details of the planning phase in algorithm 3. In the planning phase, we are given a reward function r : S ×A → [0 , 1] and we compute an estimate of sparsified transition dynamics ˜ P † , which is formally defined appendix A.1.1. The goal of the planning phase is to compute the optimal policy π final on the MDP specified by the transition dynamics ˜ P † and reward function r † , where r † is the sparsified version of r : r † ( s, a ) = r ( s, a ) and r † ( s † , a ) = 0 for any a ∈ A . To compute the optimal policy, we iteratively apply the Bellman optimality equation. First, we define ˜ Q H ( s, a ) = r † ( s, a ) for any ( s, a ) and solve

Then, for h = H -1 , H -2 , ..., 2 , 1 , we iteratively define

for any ( s, a ) , and then solve

for any s ∈ S . For s † and any h ∈ [ H ] , π final,h ( s † ) can be arbitrary action. Then, from the property of Bellman optimality equation, we know π final is the optimal policy on ˜ P † and r † .

F More related works

Other low-switching algorithms Low-switching learning algorithms were initially studied in the context of bandits, with the UCB2 algorithm [Auer et al., 2002] achieving an O ( A log K ) switching

cost. Gao et al. [2019] demonstrated a sufficient and necessary O ( A log log K ) switching cost for attaining the minimax optimal regret in multi-armed bandits. In both adversarial and stochastic online learning, [Cesa-Bianchi et al., 2013] designed an algorithm that achieves an O (log log K ) switching cost.

Reward-free reinforcement learning In reward-free reinforcement learning (RFRL) the goal is to find a near-optimal policy for any given reward function. [Jin et al., 2020a] proposed an algorithm based on EULER [Zanette and Brunskill, 2019] that can find a ε policy with ˜ O ( H 5 |S| 2 |A| /ε 2 ) trajectories. Subsequently, [Kaufmann et al., 2021] reduces the sample complexity by a factor H by using uncertainty functions to upper bound the value estimation error. The sample complexity was further improved by another H factor by [Ménard et al., 2021].

A lower bound of ˜ O ( H 2 |S| 2 |A| /ε 2 ) was established for homogeneous MDPs by [Jin et al., 2020a], while an additional H factor is conjectured for non-homogeneous cases. [Zhang et al., 2021] proposed the first algorithm with matching sample complexity in the homogeneous case. Similar results are available with linear [Wang et al., 2020a, Wagenmaker et al., 2022, Zanette et al., 2020] and general function approximation [Chen et al., 2022, Qiu et al., 2021].

Offline reinforcement learning In offline reinforcement learning the goal is to learn a near-optimal policy from an existing dataset which is generated from a (possibly very different) logging policy. Offline RL in tabular domains has been investigated extensively [Yin and Wang, 2020, Jin et al., 2020c, Nachum et al., 2019, Rashidinejad et al., 2021, Kallus and Uehara, 2022, Xie and Jiang, 2020a]. Similar results were shown using linear [Yin et al., Xiong et al., 2022, Nguyen-Tang et al., 2022, Zanette et al., 2021b] and general function approximation[Xie et al., 2021a, Long et al., 2021, Zhang et al., 2022, Duan et al., 2021, Jiang and Huang, 2020, Uehara and Sun, 2021, Zanette and Wainwright, 2022, Rashidinejad et al., 2022, Yin et al., 2022]. Offline RL is effective when the dataset 'covers' a near optimal policy, as measured by a certain concentrabiluty factor. In the function approximation setting additional conditions, such as Bellman completeness, may need to be approximately satisfied [Munos and Szepesvári, 2008, Chen and Jiang, 2019, Zanette, 2023, Wang et al., 2020b, Foster et al., 2021, Zhan et al., 2022].

Task-agnostic reinforcement learning Another related line of work is task-agnostic RL, where N tasks are considered during the planning phase, and the reward functions is collected from the environment instead of being provided directly. [Zhang et al., 2020a] presented the first task-agnostic algorithm, UBEZero, with a sample complexity of ˜ O ( H 5 |S| |A| log( N ) /ε 2 ) . Recently, [Li et al., 2023a] proposed an algorithm that leverages offline RL techniques to estimate a well-behaved behavior policy in the reward-agnostic phase, achieving minimax sample complexity. Other works exploring effective exploration schemes in RL include [Hazan et al., 2019, Du et al., 2019, Misra et al., 2020]. [Li et al., 2023b] also considered an offline-to-online reinforcement learning algorithm which explores the environment using two mixed policies in a reward-free mode.

Peer reviews:
comments: SUMMARY:
The authors introduced the concept of sparsified MDP. Based on this concept, they proposed a new algorithm that takes as input a dataset, uses it to design and deploy a non-reactive exploratory policy, and then outputs a locally near-optimal policy. A nearly minimax-optimal upper bound for the sample complexity is also established to learn a local ε-optimal policy using this algorithm.

STRENGTHS:
- This paper proposed a new setting where the agent is access to an offline dataset and can further explore the environment online with a non-reactive policy. In this way, the agent can utilize the offline dataset while avoid the engineering costs associated with switching policies.
- Sparsified MDP provides an perspective on combining both optimism and pessimism principle, which is interesting.
- The authors obtain the sample complexity bound of the new approach, and the bound is tighter than previous method.
- The paper is well-written and easy to follow.

WEAKNESSES:
- The algorithm is based on tabular setting, which limits its application.
- The exploratory policy only explore in the sparsified MDP. In this way, the final policy is strictly limited by the offline dataset even if the budget for online interaction is large.
- The transitions that cannot achieve the threshold is dropped directly, even though they contain some information of the environment.

QUESTIONS:
- Is it possible for the exploratory policy to explore out of the coverage of offline dataset? I know it is impossible if there is no data at all, but   it seems that using the information of the dropped transitions may help.

LIMITATIONS:
Limitations are not explicitly discussed by the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores reinforcement learning applications where a pre-existing dataset of collected experience is available, and suggests the possibility of obtaining additional online data to enhance policy quality. To avoid the costs associated with switching policies, the authors propose utilizing a single non-reactive exploration policy for gathering supplementary data. They present an algorithm with provable guarantees that leverages an offline dataset to design such a policy. The algorithm is analyzed theoretically, and the authors evaluate the final policy's quality based on the local coverage of the original dataset and the amount of additional data collected. Overall, the research contributes to improving reinforcement learning by optimizing data acquisition and policy design.\n\nSTRENGTHS:\nThe advantages of this paper are as follows: \n1. Although I haven't carefully derived each equation, the proofs in this paper are expected to be accurate, with complete steps and rigorous derivations.\n2. This paper represents the pioneering effort in terms of theoretical rigor, addressing the challenge of designing an experiment in reinforcement learning for online passive exploration, leveraging a dataset comprising pre-collected experiences. The setting is novel and has some practical significance.\n3. The proof method presented in this paper is innovative and the sample complexity mentioned in the conclusion is also tight. It will introduce new approaches to the theoretical research of RL. Additionally, the conclusions of this paper also reveal some interesting insights that will enrich the existing theoretical achievements in the offline-to-online field.\n\nWEAKNESSES:\n1. The basic assumption of this article is that the offline-trained policy cannot be switched during the online phase, and it allows for the collection of an unlimited number of samples using this policy. I have two concerns. Firstly, from a theoretical research perspective, this assumption narrows down the problem to a very specific setting, so even with rigorous mathematical proofs, the generalizability of the conclusions may be compromised. Secondly, from a practical application standpoint, if the offline-trained policy itself is poor but cannot be switched during the online phase, and a large number of online samples need to be collected using this policy, there will be even greater security issues. As a result, the advocated security considerations in this paper will no longer exist. I appreciate the mathematical methods used in this paper, but the lack of persuasive motivation will impact the significance of this paper.\n2. The proof process in this paper is too lengthy and difficult to understand. Although the main text provides some introduction to the overall proof logic, there are too many specific terms involved in the proof without providing intuitive explanations for their generation, which increases the difficulty of understanding. I suggest that the author can provide a more concise version, even if the resulting sample complexity is not optimal, but it can be used to understand the overall proof framework.\n3. Another limitation is that it currently only applies to smaller S and A. When S and A are larger, the application will become difficult and the sample complexity will be significant. So I'm curious to know if this approach can be extended to scenarios involving function approximation. If it is possible, what additional considerations or processing steps would be required? If it is not feasible, what challenges exist?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe authors didn't discuss the limitation of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for policy fine-tuning in reinforcement learning using a dataset of pre-collected experience. The algorithm leverages the dataset to design a non-reactive exploratory policy and outputs a locally near-optimal policy. The paper makes theoretical contributions in analyzing the quality of the policy and establishing sample complexity bounds. The work is motivated by the practical need for non-reactive exploration in domains where switching policies is costly and impractical. The paper presents innovative ideas and provides a novel solution to the problem.\n\nSTRENGTHS:\n- The paper introduces an algorithm that addresses the problem of non-reactive policy design in reinforcement learning and provides provable guarantees for the quality of the resulting policy.\n- The concept of sparsiﬁed MDP is introduced and effectively used in the algorithm and theoretical analysis.\n- The paper rigorously establishes a nearly minimax-optimal upper bound for the sample complexity needed to learn a local ε-optimal policy using the proposed algorithm.\n- The paper addresses a practical need for non-reactive exploration in domains where policy switches are costly and provides a solution that can be valuable in such scenarios.\n\nWEAKNESSES:\n- The paper only considers discrete state and action spaces. \n- The paper provides no empirical evaluations or demonstrations of the proposed algorithm. Neither does it shed light on the design of practical algorithms.\n\nQUESTIONS:\n- Can you give more details on the non-reactive property of a policy? It seems that most RL policies in an MDP will be non-reactive, as long as they take only the current state $s_t$ as input. How is the example in line 30-32 related to the non-reactive property?\n- Online exploration may lead to safety violations. How safe will the online phase be in the original MDP?\n- It seems contradictory to be both optimistic and pessimistic at the same time. How to determine the region that the agent knows how to navigate?\n- Can you provide some empirical evaluations of the proposed algorithm, compared with fully offline or fully online algorithms?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposes an algorithm that, given a previously collected dataset of transitions from an MDP, produces a non-reactive policy that can effectively collect additional data that enables a near-optimal policy to be obtained for any possible reward function. The algorithm is model-based and combines elements of optimism (exploration bonuses) and pessimism (early termination at OOD states/actions). Suboptimality is bounded relative to the optimal policy that stays within the “sparsified MDP” which is the subset of the original MDP that is sufficiently covered by the data.

STRENGTHS:
* The paper presents positive, novel (to my knowledge) theoretical results in a well-motivated setting of practical interest.
* The interaction protocol is new AFAIK, but closely related to existing areas such as reward-free RL. It may inspire more work in the future.
* The paper is clear and not hard to follow, despite its technicality.

WEAKNESSES:
* No experiments, despite having “Design of Experiments” in the title :) (But this is a theory paper so I think it is okay)
* IMO the Related Work should cite the MOReL paper [1], which uses a pessimistic MDP construction similar to your sparsified MDP, in which low-density states/actions lead to a special absorbing state. Of course, they are tackling a different setting (offline RL with a particular reward function) and use a somewhat different termination criterion, but the idea is the same.

QUESTIONS:
I’m curious if it is known that better sample complexity can be obtained if you allow the policy to be adaptive? (Although I understand the engineering-related reasons for not doing so.) If so, it could be useful context to briefly comment on in the paper.

LIMITATIONS:
No, limitations are not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The work proposes a method to create a non-reactive exploratory policy from an initial input dataset. Then, leveraging the new data, the algorithm generates a locally near-optimal policy.

The relevance of the algorithm is in the low-switching algorithms, where it is assumed there is a cost to changing a deployed policy.

The proposed algorithm uses the initial data to build a sparsified MDP, an approximation of the original MDP that keeps only the transitions (s, a, s') for which there are at least Φ transitions.

The next part of the algorithm is building the exploratory policy using the sparsified MPD using a value iteration iterative strategy with an exploration bonus instead of actual rewards. The exploration bonus balances optimism and pessimism, boosting exploration towards less explored states while avoiding spending too much time in unknown parts of the MDP.

Finally, additional data is collected from the environment using this exploring policy, and a value iteration algorithm is employed on the combined datasets of experiences to build the final policy.

The authors provide optimality bounds, describing the conditions for the algorithm to discover an epsilon-suboptimal policy.

STRENGTHS:
The strengths of this paper lie in the originality of the setup and in the mathematical work to describe the conditions that need to be met for sub-optimality guarantees.

WEAKNESSES:
The weakness of the paper is the need for empirical. For example, it would be helpful to see how well the algorithm performs given initial data sets of different sizes and coverage.

QUESTIONS:
Could some experiments be added to see the performance of the algorithm in practice, maybe compared to other offline RL methods?

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the setting where it is possible to leverage a dataset of transitions, together with the possibility of deploying a policy to collect additional information. The question then lies into what kind of policy should be deployed and what kind of data should be gathered. The authors argue that deploying an exploratory policy that switches, e.g. a policy that learns and adapts from its experience, leads a great engineering costs. As such, they propose to follow the principle of pessimism together with a non reactive policy that would be constrained to a sub-region of the MDP with enough transitions. They also make use of the principle of optimism to derive the exploratory actions taken within the subMDP. The authors provide a near optimal minimax-optimal upper bound for learning an epsilon-optimal policy.\n\nSTRENGTHS:\nThe contributions and assumptions are made clear in the introduction (however, the concept of reactive policy is only clearly explained in plain a words a bit late). An intuition section is also provided which helps the reader follow the paper.\n\nThe authors provide theoretical guarantees both for the sparsified MDP together with the full MDP. In particular for the results on the full MDP it seems like a reduction in the epsilon coefficient is an important contribution.\n\nThe paper emphasizes a hybrid approach, exploring a setting that combines elements of both offline and online methods, hence offering a more practical and adaptable framework for various real-world applications.\n\nWEAKNESSES:\nAlthough the setting of interest is important (mix of online and offline) the paper doesn’t stress enough the importance of a non-reactive policy. Why is learning from the generated experience such a bad idea in practice? It seems like a slowly changing deployed policy (where changes perhaps happen through a trust region) would be a better choice.\n\nI understand the paper is essentially a theoretical one, however it would be interesting to present some empirical evidence as to the practicability of the proposed algorithm. Indeed, it is not clear if the current dependencies on state, action, epsilon and such quantities in the bounds would provide a meaningful difference.\n\nThe paper misses an opportunity to provide a comprehensive conclusion, one that effectively synthesizes the results, giving the reader a clear understanding of the study's overall implications and potential future directions.\n\nQUESTIONS:\nHow good of a policy (trained from scratch) could be obtained with the number of additional samples required in Corollary 5.2?\n\nAlthough the dependence on the desired accuracy is reduced in Corollary 5.2, how much of a difference can this make given the concentrability coefficient?\n\nLIMITATIONS:\nThe limitations are not clearly stated throughout the work. Some of the questions above try to probe into this.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}