Title: Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reward-agnostic reinforcement learning

policy finetuning

offline reinforcement learning

online reinforcement learning
Abstract: This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds --- pure offline RL and pure online RL --- in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called **single-policy partial concentrability**, which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.

Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning

Abstract

This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds - pure offline RL and pure online RL - in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called single-policy partial concentrability , which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.

1 Introduction

As reinforcement learning (RL) shows promise in achieving super-human empirical success across diverse fields (e.g., games (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019; Mnih et al., 2013), robotics (Brambilla et al., 2013), autonomous driving (Shalev-Shwartz et al., 2016)), theoretical understanding about RL has also been substantially expanded, with the aim of distilling fundamental principles that can inform and guide practice. Among all sorts of theoretical questions being pursued, how to make the best use of data emerges as a question of profound interest for problems with enormous dimensionality.

There are at least two mainstream mechanisms when it comes to data collection: online RL and offline RL. Let us briefly describe their attributes and differences as follows.

Online RL. In this setting, an agent learns how to maximize her cumulative reward through interaction with the unknown environment (by, say, executing a sequence of adaptively chosen actions and utilizing the instantaneous feedback of the environment). Given that all information about the environment is obtained through real-time data collection, the main challenge lies in how to (optimally) manage the trade-off between exploration and exploitation. Towards this, one popular approach advertises the principle of optimism in the face of uncertainty - e.g., employing upper confidence bounds during value estimation to guide exploration - whose effectiveness has been shown for both the tabular case (Auer and Ortner, 2006; Jaksch et al., 2010; Azar et al., 2017; Dann et al., 2017; Jin et al., 2018; Bai et al., 2019; Dong et al., 2019; Zhang et al., 2020b; Ménard et al., 2021b; Li et al., 2021b) and the case with function approximation (Jin et al., 2020b; Zanette et al., 2020; Zhou et al., 2021a; Li et al., 2021a; Du et al., 2021; Jin et al., 2021a; Foster et al., 2021; Chen et al., 2022b).

Offline RL. In contrast, offline RL assumes access to a pre-collected dataset, without given permission to perform any further data collection. The feasibility of reliable offline RL depends heavily

on the quality of the dataset at hand. A central challenge stems from the presence of distribution shift: the distribution of the offline dataset might differ significantly from that induced by the target policy. Another common challenge arises from insufficient data coverage: a nontrivial fraction of the state-action pairs might be inadequately visited in the available dataset, thus precluding one from faithfully evaluating many policies based solely on the offline dataset. To circumvent these obstacles, recent works proposed the principle of pessimism in the face of uncertainty, recommending caution when selecting poorly visited actions (Liu et al., 2020; Kumar et al., 2020; Jin et al., 2021b; Rashidinejad et al., 2021; Uehara and Sun, 2021; Li et al., 2022; Yin et al., 2021; Shi et al., 2022; Cai et al., 2022). Without requiring uniform coverage of all policies, the pessimism approach proves effective as long as the so-called single-policy concentrability is satisfied, which only assumes adequate coverage over the part of the state-action space reachable by the desirable policy.

In reality, however, both mechanisms above come with limitations. For instance, even the singlepolicy concentrability requirement might be too stringent (and hence fragile) for offline RL, as it is not uncommon for the historical dataset to miss a small yet essential part of the state-action space. Pure online RL might also be overly restrictive, given that there might be information from past data that could help initialize online exploration and mitigate the burden of further data collection.

All this motivates the studies of hybrid RL, a scenario where the agent has access to an offline dataset while, in the meantime, (limited) online data collection is permitted as well. Oftentimes, this scenario is practically not only feasible but also appealing: on the one hand, offline data provides useful information for policy pre-training, while further online exploration helps enrich existing data and allows for effective policy fine-tuning. As a matter of fact, multiple empirical works (Rajeswaran et al., 2017; Vecerik et al., 2017; Kalashnikov et al., 2018; Hester et al., 2018; Nair et al., 2018, 2020) indicated that combining online RL with offline datasets outperforms both pure online RL and pure offline RL. Nevertheless, theoretical pursuits about hybrid RL are lagging behind. Two recent works Ross and Bagnell (2012); Xie et al. (2021b) studied a restricted setting, where the agent is aware of a Markovian behavior policy (a policy that generates offline data) and can either execute the behavior policy or any other adaptive choice to draw samples in each episode; in this case, Xie et al. (2021b) proved that under the single-policy concentrability assumption of the offline dataset, having perfect knowledge about the behavior policy does not improve online exploration in the minimax sense. Another strand of works Song et al. (2022); Nakamoto et al. (2023); Wagenmaker and Pacchiano (2022) looked at a more general offline dataset and investigated how to leverage offline data in online exploration. From the sample complexity viewpoint, Wagenmaker and Pacchiano (2022) studied the statistical benefits of hybrid RL in the presence of linear function approximation; the result therein, however, required strong assumptions on data coverage (i.e., all-policy concentrability) and fell short of unveiling provable gains in the tabular case (as we shall elucidate momentarily). In light of such theoretical inadequacy in previous works, this paper is motivated to pursue the following question:

Does hybrid RL allow for improved sample complexity compared to pure online or offline RL in the tabular case?

1.1 Main contributions

We deliver an affirmative answer to the above question. Further relaxing the single-policy concentrability assumption, we introduce a relaxed notation called single-policy partial concentrability (to be made precise in Definition 2), which (i) allows the dataset to miss a fraction of the state-action space visited by the optimal policy and (ii) captures the tradeoff between distribution mismatch and lack of coverage. Armed with this notion, our results reveal provable statistical benefits of hybrid RL compared with both pure online and offline RL. The main contributions are summarized below.

A novel three-stage algorithm. We design a new hybrid RL algorithm consisting of three stages. In the first stage, we obtain crude estimation of the occupancy distribution d π w.r.t. any policy π as well as the data distribution d off of the offline dataset. The second stage performs online exploration; in particular, we execute one exploration policy to imitate the offline dataset and another one to explore the inadequately visited part of the unknown environment, with both policies computed by approximately solving convex optimization sub-problems. Notably, these two stages do not count on the availability of reward information, and thus operate in a reward-agnostic manner. The final stage then invokes the state-of-the-art offline RL algorithm for policy learning, on the basis of all data we have available (including both online and offline data).

Computationally efficient subroutines. Throughout the first two stages of the algorithm, we need to solve a couple of convex sub-problems with exponentially large dimensions. In order to attain computational efficiency, we design efficient Frank-Wolfe-type paradigms to solve the sub-problems approximately, which run in polynomial time. This plays a crucial role in ensuring computational tractability of the proposed three-stage algorithm.

Improved sample complexity. We characterize the sample complexity of our algorithm (see Theorem 1), which provably improves upon both pure online and offline RL. On the one hand, hybrid RL achieves strictly enhanced performance compared to pure offline RL (assuming the same sample size) when the offline data falls short of covering all state-action pairs reachable by the desired policy. On the other hand, the sample size allocated to online exploration in our algorithm might only need to be proportional to the fraction σ of the state-action space uncovered by the offline dataset, thus resulting in sample size saving in general compared to pure online RL (a case with σ = 1 ).

Notation. Let us also introduce several useful notation. For integer m > 0 , we let [ m ] represent the set { 1 , · · · , m } . For any set B , we denote by B c its complement. For any policy π 0 , we let 1 π 0 : Π →{ 0 , 1 } be an indicator function such that 1 π 0 ( π ) = 1 if π = π 0 and 1 π 0 ( π ) = 0 otherwise. For any finite set A , we denote by ∆( A ) the probability simplex over A . Letting X := ( S, A, H, 1 ε , 1 δ ) , we use the notation f ( X ) = O ( g ( X )) or f ( X ) glyph[lessorsimilar] g ( X ) to indicate the existence of a universal constant C 1 > 0 such that f ≤ C 1 g , the notation f ( X ) glyph[greaterorsimilar] g ( X ) to indicate that g ( X ) = O ( f ( X )) , and the notation f ( X ) glyph[equivasymptotic] g ( X ) to mean that f ( X ) glyph[lessorsimilar] g ( X ) and f ( X ) glyph[greaterorsimilar] g ( X ) hold simultaneously. The notation ˜ O ( · ) is defined in the same way as O ( · ) except that it hides logarithmic factors.

2 Preliminaries and problem settings

Episodic finite-horizon MDPs. We study episodic finite-horizon Markov decision processes with S states, A actions, and horizon length H . We use M = ( S , A , H, P = { P h } H h =1 , r = { r h } H h =1 ) to represent such an MDP, where S = [ S ] and A = [ A ] represent the state space and the action space, respectively. For each step h ∈ [ H ] , we let P h : S × A → ∆( S ) represent the transition probability at this step, such that taking action a in state s at step h yields a transition to the next state drawn from the distribution P h ( · | s, a ) ; throughout the paper, we often employ the shorthand notation P h,s,a := P h ( ·| s, a ) . Another ingredient is the reward function specified by r h : S × A → [0 , 1] at step h ; namely, the agent will receive an immediate reward r h ( s, a ) upon executing action a in state s at step h . It is assumed that the reward function is fully revealed upon completion of online data collection. Additionally, we assume throughout that each episode of the MDP starts from an initial state independently generated from some (unknown) initial state distribution ρ ∈ ∆( S ) .

A time-inhomogeneous Markovian policy is often denoted by π = { π h } H h =1 with π h : S → ∆( A ) , where π h ( · | s ) characterizes the (randomized) action selection probability of the agent in state s at step h . If π is a deterministic policy, then we often abuse the noation and let π h ( s ) represent the action selected in state s at step h . We find it convenient to introduce the following notation:

We also need to handle mixed deterministic policies (i.e., each realization of the policy is randomly drawn from a mixture of deterministic policies). A mixed deterministic policy π mixed is denoted by

Moreover, for any policy π , we define its associated value function (resp. Q-function) as follows, representing the expected cumulative rewards conditioned on an initial state (resp. an initial stateaction pair):

Here, the expectation is over the lengthH sample trajectory ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H ) when executing policy π in M , where s h (resp. a h ) denotes the state (resp. action) at step h of this trajectory.

When the initial state is drawn from ρ , we further augment the notation and denote

Importantly, there exists at least one deterministic policy, denoted by π glyph[star] throughout, that is able to maximize V π h ( s ) and Q π h ( s, a ) simultaneously for all ( h, s, a ) ∈ [ H ] ×S × A ; namely,

Moving beyond value functions and Q-functions, we would like to define, for each policy π , the associated state-action occupancy distribution d π = [ d π h ] 1 ≤ h ≤ H such that

in other words, this is the probability of the state-action pair ( s, a ) ∈ S × A being visited by π at step h . We shall also overload d π to represent the state occupancy distribution such that

Given that each episode always starts with a state drawn from ρ, it is easily seen that d π 1 ( s ) = ρ ( s ) for any policy π and any s ∈ S .

Sampling mechanism. We consider a hybrid RL setting that assumes access to a historical dataset as well as the ability to further explore the environment via real-time sampling, as detailed below.

Offline data. Suppose that we have available a historical dataset (also called an offline dataset)

containing K off sample trajectories each of length H . Here, the k -th trajectory in D off is denoted by

where s k, off h and a k, off h indicate respectively the state and action at step h of this trajectory τ k, off . It is assumed that each trajectory τ k, off is drawn independently using policy π off , which takes the form of a mixture of deterministic policies

Note that the learner only has access to the data samples but not π off . Throughout the paper, we use d off = { d off h } 1 ≤ h ≤ H to represent the occupancy distribution of this offline dataset such that

Online exploration. In addition to the offline dataset, the learner is allowed to interact with the unknown environment and collect more data in real time, in the hope of compensating for the insufficiency of the pre-collected data at hand and fine-tuning the policy estimate. More specifically, the learner is able to sample K on trajectories sequentially. In each sample trajectory,

the initial state is generated independently from an (unknown) distribution ρ ∈ ∆( S ) ;

the learner selects a policy to execute the MDP, obtaining a sample trajectory of length H .

The total number of sample trajectories is thus given by

Concentrability assumptions for the offline dataset. To quantify the quality of the historical dataset, prior offline RL literature introduced the following single-policy concentrability coefficient based on certain density ratio of interest; see, e.g., Rashidinejad et al. (2021); Li et al. (2022).

Definition 1 (Single-policy concentrability) . The single-policy concentrability coefficient C glyph[star] of the offline dataset D off is defined as

In words, C glyph[star] employs the glyph[lscript] ∞ -norm of the density ratio d π glyph[star] /d off to capture the shift of distributions between the occupancy distribution induced by the desired policy π glyph[star] and the data distribution at hand. The terminology 'single-policy' underscores that Definition 1 only compares the offline data distribution against the one generated by a single policy π glyph[star] , which stands in stark contrast to other all-policy concentrability coefficients that are defined to account for all policies simultaneously.

One notable fact about Definition 1 is that: for C glyph[star] to be finite, the historical data distribution needs to cover all state-action-step tuples reachable by π glyph[star] . This requirement is, in general, inevitable if only the offline dataset is available; see the minimax lower bounds in Rashidinejad et al. (2021); Li et al. (2022) for more precise justifications. However, a requirement of this kind could be overly stringent for the hybrid setting considered herein, as the issue of incomplete coverage can potentially be overcome with the aid of online data collection. In light of this observation, we generalize Definition 1 to account for the trade-offs between distributional mismatch and partial coverage.

Definition 2 (Single-policy partial concentrability) . For any σ ∈ [0 , 1] , the single-policy partial concentrability coefficient C glyph[star] ( σ ) of the offline dataset D off is defined as

where

In Definition 2, we allow a fraction of the state-action space reachable by π glyph[star] to be insufficiently covered (as reflected in the definition of G ( σ ) measured by the state-action occupancy distribution) - hence the terminology 'partial'. Intuitively, G h corresponds to a set of state-action pairs that undergo reasonable distribution shift (so that the corresponding density ratio does not rise above C glyph[star] ( σ ) ), whereas the total occupancy density of its complement subset G c h induced by π glyph[star] is under control (i.e., no larger than σ when averaged across steps). As a self-evident fact, C glyph[star] ( σ ) is nonincreasing in σ ; this means that as σ increases, we might incur a less severe distribution shift in a restricted part, at the price of less coverage. In this sense, C glyph[star] ( σ ) reflects certain tradeoffs between distribution shift and coverage. Clearly, C glyph[star] ( σ ) reduces to C glyph[star] in Definition 1 by taking σ = 0 .

Goal. Given a historical dataset D off containing K off sample trajectories, we would like to design an online exploration scheme, in conjunction with the accompanying policy learning algorithm, so as to achieve desirable policy learning (or policy fine-tuning) in a data-efficient manner. Ideally, we would expect a hybrid RL algorithm to harvest provable statistical benefits compared to both purely online RL and purely offline RL approaches.

3 Algorithm

In this section, we propose a new algorithm to tackle the hybrid RL setting. Our algorithm design leverages recent ideas developed in offline RL and reward-agnostic exploration to improve sample efficiency. Our algorithm consists of three stages to be described shortly; informally, the first two stages conduct reward-agnostic exploration to imitate and complement the offline dataset, whereas the third stage invokes a sample-optimal offline RL algorithm to compute a near-optimal policy.

In the sequel, we split the offline dataset D off into two halves:

where D off , 1 (resp. D off , 2 ) consists of the first (resp. last) K off / 2 independent trajectories from D off . As we shall also see momentarily, online exploration in the proposed algorithm - which collects K on trajectories in total - can be divided into three parts, collecting K on prepare , K on imitate and K on explore sample trajectories, respectively. Throughout this paper, for simplicity we choose

3.1 A three-stage algorithm

We now elaborate on the three stages of the proposed algorithm. Due to space limitation, the pseudocode of the complete algorithm is provided in Appendix B.

Stage 1: estimation of the occupancy distributions. As a preparatory step for reward-agnostic exploration, we first attempt to estimate the occupancy distribution induced by any policy as well as the occupancy distribution d off associated with the historical dataset, as described below.

Estimating d π for any policy π . In this step, we would like to sample the environment and collect a set of sample trajectories, in a way that allows for reasonable estimation of the occupancy distribution d π induced by any policy π . For this purpose, we invoke the exploration strategy and the accompanying estimation scheme developed in Li et al. (2023). Working forward (i.e., from h = 1 to H ), this approach collects, for each step h , a set of N sample trajectories in order to facilitate estimation of the occupancy distributions, which amounts to a total number of

sample trajectories collected in this stage. See Algorithm 3 in Appendix D.1 for a precise description of this strategy. Noteworthily, while Algorithm 3 specifies how to estimate ̂ d π for any policy π , we won't need to compute it explicitly unless this policy π is encountered during the subsequent steps of the algorithm; in other words, ̂ d π should be viewed as a sort of 'function handle' that will only be executed when called later.

Estimating d off for the historical dataset D off . In addition, we are in need of estimating the occupancy distribution d off . Towards this end, we propose the following empirical estimate using the K off / 2 sample trajectories from D off , 1 :

for all ( s, a ) ∈ S × A , where c off > 0 is some universal constant. Here, 1 -δ indicates the target success probability, and

In other words, ̂ d off h ( s, a ) is taken to be the empirical visitation frequency of ( s, a ) in D off , 1 if ( s, a ) is adequately visited, and zero otherwise. The cutoff threshold will be made clear in our analysis.

Stage 2: online exploration. Armed with the above estimates of the occupancy distributions, we can readily proceed to compute the desired exploration policies and sample the environment. We seek to devise two exploration strategies, with one strategy selected to imitate the offline dataset, and the other one employed to explore the insufficiently visited territory. As a preliminary fact, if we have a dataset containing K independent trajectories - generated independently from a mixture of deterministic policies with occupancy distribution d b -then it has been shown previously (see, e.g., Li et al. (2023, Section 3.3)) that the model-based offline approach is able to compute a policy ̂ π obeying

This upper bound in (17) provides a guideline regarding how to design a sample-efficient exploration scheme.

Imitating the offline dataset. The offline dataset D off is most informative when it contains expert data, a scenario when the data distribution resembles the distribution induced by the optimal policy π glyph[star] . If this is the case, then it is desirable to find a policy similar to π off in (6) (the mixed policy generating D off ) and employ it to collect new data, in order to retain and further strength the benefits of

such offline data. To do so, we attempt to approximate d π glyph[star] by ̂ d off in (17) when attempting to minimize (17). In fact, we would like to compute a mixture of deterministic policies by (approximately) solving the following optimization problem:

which is clearly equivalent to

In order to solve this minimax problem (19) (note that its objective function is convex in µ ), we resort to the Follow-The-Regularized-Leader (FTRL) strategy from the online learning literature (ShalevShwartz, 2012); more specifically, we perform the following updates iteratively for t = 1 , . . . , T max :

where η denotes the learning rate to be specified later. We shall discuss how to solve the optimization sub-problem (20b) in Appendix C. The output of this step is a mixture of deterministic policies taking the following form:

Exploring the unknown environment. In addition to mimicking the behavior of the historical dataset, we shall also attempt to explore the environment in a way that complements pre-collected data. Towards this end, it suffices to invoke the reward-agnostic online exploration scheme proposed in Li et al. (2023), whose precise description will be provided in Algorithm 5 in Appendix D.2 to make the paper self-contained. The resulting policy mixture is denoted by

with µ explore ∈ ∆(Π) representing the associated weight vector.

With the above two exploration policies (21) and (22) in place, we execute the MDP to obtain sample trajectories as follows:

Execute the MDP K on imitate times using policy π imitate to obtain a dataset containing K on imitate = K on / 3 independent sample trajectories, denoted by D on imitate ;

Execute the MDP K on explore times using policy π explore to obtain a dataset containing K on explore = K on / 3 independent sample trajectories, denoted by D on explore .

Stage 3: policy learning via offline RL. Once the above online exploration process is completed, we are positioned to compute a near-optimal policy on the basis of the data in hand. More precisely,

Let us look at the following dataset

In light of the complicated statistical dependency between D off , 1 and D on imitate ∪D on explore , we only include the second half D off , 2 of the offline dataset D off , so as to exploit the fact that D off , 2 is statistically independent from D on imitate ∪ D on explore .

We invoke the pessimistic model-based offline RL algorithm proposed in Li et al. (2022) to compute the final policy estimate ̂ π ; see Algorithm 6 in Appendix D.3 for more details.

4 Main results

As it turns out, Algorithm 1 is capable of achieving provable sample efficiency, as demonstrated in the following theorem. Here and below, we recall that K = K off + K on .

Theorem 1. Consider δ ∈ (0 , 1) and ε ∈ (0 , H ] . Choose the algorithmic parameters such that

Suppose that

for some large enough constant c 1 > 0 . Then with probability at least 1 -δ , the policy ̂ π returned by Algorithm 1 satisfies

provided that K on and K off both exceed some polynomial poly ( H,S,A,C glyph[star] ( σ ) , log K δ ) (independent of ε ).

The proof is deferred to Appendix E. In a nutshell, Theorem 1 uncovers that our algorithm yields ε -accuracy as long as

ignoring lower-order terms. Several implications of this result are as follows.

Sample complexity benefits compared with pure online or pure offline RL. To make apparent its advantage compared with both pure offline and online RL, we make comparisons with several most relevant works. Discussions of other related works are deferred to Appendix A.

Sample complexity with balanced online and offline data. For the ease of presentation, let us look at a simple case where K off = K on = K/ 2 . The the sample complexity bound (25) in this case simplifies to

Comparisons with pure online RL. We now look at pure online RL, corresponding to the case where K = K on (so that all sample episodes are collected via online exploration). In this case, the minimax-optimal sample complexity for computing an ε -optimal policy is known to be (Azar et al., 2017; Li et al., 2023)

assuming that ε is sufficiently small, which is clearly worse than (26). For instance, if there exists some very small σ glyph[lessmuch] 1 /H obeying C glyph[star] ( σ ) glyph[lessorsimilar] 1 , then the ratio of (26) to (27) is at most

thus resulting in substantial sample size savings.

Comparisons with pure offline RL. In contrast, in the pure offline case where K = K off , the minimax sample complexity is known to be (Li et al., 2022)

for any target accuracy level ε , which is apparently larger than (26) in general. In particular, recognizing that C glyph[star] (0) = ∞ in the presence of incomplete coverage of the state-action space reachable by π glyph[star] , we might harvest enormous sample size benefits (by exploiting the ability of online RL to visit the previously uncovered state-action-step tuples).

Comparison with Wagenmaker and Pacchiano (2022). It is worth noting that Wagenmaker and Pacchiano (2022) also considered policy fine-tuning and proposed a method called FTPedel to tackle linear MDPs. The results therein, however, were mainly instance-dependent, thus making it difficult to compare in general. That being said, we would like to clarify two points:

Wagenmaker and Pacchiano (2022) imposed all-policy concentrability assumptions, requring the combined dataset (i.e., the offline and online data altogether) to cover certain feature vectors for all linear softmax policies (see Wagenmaker and Pacchiano (2022, Definition 4.1)). In contrast, our results only assume single-policy (partial) concentrability, which is much weaker than the all-policy counterpart.

When specializing Wagenmaker and Pacchiano (2022, Corollary 1) to the tabular cases, the sample complexity therein becomes ˜ O ( H 7 S 2 A 2 /ε 2 ) , which is much larger than our result.

Miscellaneous properties of the proposed algorithm. In addition to the sample complexity advantages, the proposed hybrid RL enjoys several attributes that could be practically appealing.

Adaptivity to unknown optimal σ . While we have introduced the parameter σ to capture incomplete coverage, our algorithm does not rely on any knowledge of σ . Take the balanced case described around (26) for instance: our algorithm automatically identifies the optimal σ that minimizes the function f mixed ( σ ) over all σ ∈ [0 , 1] . In other words, Algorithm 1 is able to automatically identify the optimal trade-offs between distribution mismatch and inadequate coverage.

Reward-agnostic data collection. It is noteworthy that the online exploration procedure employed in Algorithm 1 does not require any prior information about the reward function. In other words, it is mainly designed to improve coverage of the state-action space, a property independent from the reward function. In truth, the reward function is only queried at the last step to output the learned policy. This enables us to perform hybrid RL in a reward-agnostic manner, which is particularly intriguing in practice, as there is no shortage of scenarios where the reward functions might be engineered subsequently to meet different objectives.

Strengthening behavior cloning. Another notable feature is that our algorithm does not rely on prior knowledge about the policies generating the offline dataset D off ; in fact, it is capable of finding a mixed exploration policy π imitate that inherits the advantages of the unknown behavior policy π off . This could be of particular interest for behavior cloning, where the offline dataset D off is generated by an expert policy, with C glyph[star] = C glyph[star] (0) ≈ 1 , i.e. the expert policy covers the optimal one. In this situation, the supplement of online data collection improves behavior cloning by lowering the statistical error from √ H 3 SC glyph[star] K off to √ H 3 SC glyph[star] K off + K on , together with an executable learned policy π imitate .

Computational complexity. We now take a moment to discuss the computational cost of the proposed algorithm. In Stage 1, we need to first estimate the transition matrices { P h } , which can be accomplished with runtime O ( K on ) . In the ensuing stages, we call Algorithm 3 to estimate ̂ d π for each π we encounter. When computing π imitate , we need to calculate ̂ d π for T 1 = T max T 2 times, where T 2 denotes the number of iterations for calculating µ t +1 in Eq. (20b); in comparison, the computational cost of estimating d π to yield π explore in Eq. (22) is much smaller. For each ̂ d π , it needs O ( HS 2 A ) computation. With a slight modification on the target Eq. (19) as follows

we can find a good enough π imitate with T max = ˜ O ( H 2 S 2 ) for η glyph[equivasymptotic] 1 H 2 S 2 and O ( H 4 S 4 A 2 ) FrankWolfe updates for α glyph[equivasymptotic] 1 H 3 S 3 A 2 . These taken collectively lead to the following computational complexity for each stage: O ( K on + H 7 S 8 A 3 + K off H ) for Stage 1, O ( H 7 S 7 A 3 ) for Stage 2, and O ( KH ) for Stage 3.

5 Discussion

We have studied the policy fine-tuning problem of practical interest, where one is allowed to exploit pre-collected historical data to facilitate and improve online RL. We have proposed a three-stage algorithm tailored to the tabular setting, which attains provable sample size savings compared with both pure online RL and pure offline RL algorithms. Our algorithm design has leveraged key insights from recent advances in both model-based offline RL and reward-agnostic online RL.

While the proposed algorithm achieves provable sample efficiency, this cannot be guaranteed unless the sample size already surpasses a fairly large threshold (in other words, the algorithm imposes a high burn-in cost). It would be of great interest to see whether one can achieve sample optimality for the entire ε -range. Another issue arises from the computation side: even though the proposed algorithm can be implemented in polynomial time, the computational complexity of the Frank-Wolfe-type subroutine might already be too expensive for solving problems with enormous dimensionality. Can we hope to further accelerate it to make it practically more appealing? Finally, it might also be interesting to study sample-efficient hybrid RL in the presence of low-complexity function approximation, in the hope of further reducing sample complexity.

References

Agarwal, A., Kakade, S., Krishnamurthy, A., and Sun, W. (2020). Flambe: Structural complexity and representation learning of low rank mdps. arXiv preprint arXiv:2006.10814 .

Alon, N. and Spencer, J. H. (2016). The probabilistic method . John Wiley & Sons.

Auer, P. and Ortner, R. (2006). Logarithmic online regret bounds for undiscounted reinforcement learning. Advances in neural information processing systems , 19.

Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 263-272. JMLR. org.

Bai, Y., Xie, T., Jiang, N., and Wang, Y.-X. (2019). Provably efficient q -learning with low switching cost. In Advances in Neural Information Processing Systems , volume 32, pages 8002-8011.

Berner, C., Brockman, G., Chan, B., Cheung, V., D˛ ebiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 .

Bertsekas, D. P. (2017). Dynamic programming and optimal control (4th edition) . Athena Scientific.

Brafman, R. I. and Tennenholtz, M. (2002). R-max-a general polynomial time algorithm for nearoptimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231.

Brambilla, M., Ferrante, E., Birattari, M., and Dorigo, M. (2013). Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence , 7(1):1-41.

Bubeck, S. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends R © in Machine Learning , 8(3-4):231-357.

Cai, C., Cai, T. T., and Li, H. (2022). Transfer learning for contextual multi-armed bandits. arXiv preprint arXiv:2211.12612 .

Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 1042-1051.

Chen, J., Modi, A., Krishnamurthy, A., Jiang, N., and Agarwal, A. (2022a). On the statistical efficiency of reward-free exploration in non-linear rl. arXiv preprint arXiv:2206.10770 .

Chen, Z., Li, C. J., Yuan, A., Gu, Q., and Jordan, M. I. (2022b). A general framework for sampleefficient function approximation in reinforcement learning. arXiv preprint arXiv:2209.15634 .

Cui, Q. and Du, S. S. (2022). When is offline two-player zero-sum Markov game solvable? arXiv preprint arXiv:2201.03522 .

Dann, C., Lattimore, T., and Brunskill, E. (2017). Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 30.

Domingues, O. D., Ménard, P., Kaufmann, E., and Valko, M. (2021). Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory , pages 578-598. PMLR.

Dong, K., Wang, Y., Chen, X., and Wang, L. (2019). Q-learning with UCB exploration is sample efficient for infinite-horizon MDP. arXiv preprint arXiv:1901.09311 .

Du, S. S., Kakade, S. M., Lee, J. D., Lovett, S., Mahajan, G., Sun, W., and Wang, R. (2021). Bilinear classes: A structural framework for provable generalization in rl.

Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018). Deep Q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32.

Huang, R., Yang, J., and Liang, Y. (2022). Safe exploration incurs nearly no additional sample complexity for reward-free RL. arXiv preprint arXiv:2206.14057 .

Jaksch, T., Ortner, R., and Auer, P. (2010). Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research .

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is Q-learning provably efficient? In Advances in Neural Information Processing Systems , pages 4863-4873.

Jin, C., Krishnamurthy, A., Simchowitz, M., and Yu, T. (2020a). Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR.

Jin, C., Liu, Q., and Miryoosefi, S. (2021a). Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Advances in Neural Information Processing Systems , 34.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020b). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR.

Jin, Y., Yang, Z., and Wang, Z. (2020c). Is pessimism provably efficient for offline rl? arXiv preprint arXiv:2012.15085 .

Jin, Y., Yang, Z., and Wang, Z. (2021b). Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096. PMLR.

Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673. PMLR.

Kaufmann, E., Ménard, P., Domingues, O. D., Jonsson, A., Leurent, E., and Valko, M. (2021). Adaptive reward-free exploration. In Algorithmic Learning Theory , pages 865-891. PMLR.

Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS) .

Li, G., Chen, Y., Chi, Y., Gu, Y., and Wei, Y. (2021a). Sample-efficient reinforcement learning is feasible for linearly realizable MDPs with limited revisiting. Advances in Neural Information Processing Systems , 34:16671-16685.

Li, G., Shi, L., Chen, Y ., Chi, Y ., and Wei, Y . (2022). Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 .

Li, G., Shi, L., Chen, Y., Gu, Y., and Chi, Y. (2021b). Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning. Advances in Neural Information Processing Systems , 34:17762-17776.

Li, G., Yan, Y., Chen, Y., and Fan, J. (2023). Minimax-optimal reward-agnostic exploration in reinforcement learning. arXiv preprint arXiv:2304.0727 .

Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. (2020). Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202 .

Ménard, P., Domingues, O. D., Jonsson, A., Kaufmann, E., Leurent, E., and Valko, M. (2021a). Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning , pages 7599-7608. PMLR.

Ménard, P., Domingues, O. D., Shang, X., and Valko, M. (2021b). UCB momentum Q-learning: Correcting the bias without forgetting. In International Conference on Machine Learning , pages 7609-7618.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 .

Munos, R. and Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. In Journal of Machine Learning Research , volume 9, pages 815-857.

Nair, A., Gupta, A., Dalal, M., and Levine, S. (2020). Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 .

Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6292-6299. IEEE.

Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S. (2023). Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479 .

Qiao, D. and Wang, Y.-X. (2022). Near-optimal deployment efficiency in reward-free reinforcement learning with linear function approximation. arXiv preprint arXiv:2210.00701 .

Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2017). Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 .

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716.

Ross, S. and Bagnell, J. A. (2012). Agnostic system identification for model-based reinforcement learning. International Conference on Machine learning .

Shalev-Shwartz, S. (2012). Online learning and online convex optimization. Foundations and Trends R © in Machine Learning , 4(2):107-194.

Shalev-Shwartz, S., Shammah, S., and Shashua, A. (2016). Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 .

Shi, L. and Chi, Y. (2022). Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 .

Shi, L., Li, G., Wei, Y ., Chen, Y ., and Chi, Y . (2022). Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning , pages 19967-20025. PMLR.

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. nature , 529(7587):484-489.

Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. (2022). Hybrid RL: Using both offline and online data can make RL efficient. arXiv preprint arXiv:2210.06718 .

Uehara, M. and Sun, W. (2021). Pessimistic model-based offline RL: PAC bounds and posterior sampling under partial coverage. In arXiv preprint arXiv:2107.06226 .

Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M. (2017). Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 .

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft ii using multiagent reinforcement learning. Nature , 575(7782):350-354.

Wagenmaker, A. and Pacchiano, A. (2022). Leveraging offline data in online reinforcement learning. arXiv preprint arXiv:2211.04974 .

Wagenmaker, A. J., Chen, Y., Simchowitz, M., Du, S., and Jamieson, K. (2022). Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. In International Conference on Machine Learning , pages 22430-22456.

Wang, R., Du, S. S., Yang, L., and Salakhutdinov, R. R. (2020). On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems , 33:17816-17826.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021a). Bellman-consistent pessimism for offline reinforcement learning. arXiv preprint arXiv:2106.06926 .

Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2021b). Policy finetuning: Bridging sampleefficient offline and online reinforcement learning. arXiv preprint arXiv:2106.04895 .

Yan, Y., Li, G., Chen, Y., and Fan, J. (2022). Model-based reinforcement learning is minimax-

optimal for offline zero-sum Markov games. arXiv preprint arXiv:2206.04044 .

Yan, Y., Li, G., Chen, Y., and Fan, J. (2023). The efficacy of pessimism in asynchronous Q-learning. IEEE Transactions on Information Theory .

Yin, M., Bai, Y., and Wang, Y.-X. (2021). Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems , 34:7677-7688.

Zanette, A., Lazaric, A., Kochenderfer, M., and Brunskill, E. (2020). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 1097810989. PMLR.

Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. (2022). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR.

Zhang, W., Zhou, D., and Gu, Q. (2021a). Reward-free model-based reinforcement learning with linear function approximation. Advances in Neural Information Processing Systems , 34:15821593.

Zhang, X., Ma, Y., and Singla, A. (2020a). Task-agnostic exploration in reinforcement learning. Advances in Neural Information Processing Systems , 33:11734-11743.

Zhang, Z., Chen, Y., Lee, J. D., and Du, S. S. (2023). Settling the sample complexity of online reinforcement learning. arXiv preprint arXiv:2307.13586 .

Zhang, Z., Du, S., and Ji, X. (2021b). Near optimal reward-free reinforcement learning. In International Conference on Machine Learning , pages 12402-12412. PMLR.

Zhang, Z., Zhou, Y., and Ji, X. (2020b). Model-free reinforcement learning: from clipped pseudoregret to sample complexity. arXiv preprint arXiv:2006.03864 .

Zhou, D., Gu, Q., and Szepesvari, C. (2021a). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR.

Zhou, Z., Zhou, Z., Bai, Q., Qiu, L., Blanchet, J., and Glynn, P. (2021b). Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339. PMLR.

A Other related works

In this section, we briefly discuss a small set of additional prior works related to the current paper.

(Reward-aware) online RL. In online RL, an agent seeks to find a near-optimal policy by sequentially and adaptively interacting with the unknown environment, without having access to any additional offline dataset. The extensive studies of online RL gravitate around how to optimally trade off exploration against exploitation, for which the principle of optimism in the face of uncertainty plays a crucial role (Auer and Ortner, 2006; Jaksch et al., 2010; Azar et al., 2017; Dann et al., 2017; Jin et al., 2018; Bai et al., 2019; Dong et al., 2019; Zhang et al., 2020b; Ménard et al., 2021b; Li et al., 2021b; Zhang et al., 2023). Information-theoretic regret lower bounds have been established by Domingues et al. (2021); Jin et al. (2018), which are shown to be achievable (up to log factor) by the model-based approach for arbitrary sample sizes (Zhang et al., 2023). A further strand of works extended these studies to the case with function approximation, including both linear function approximation (Jin et al., 2020b; Zanette et al., 2020; Zhou et al., 2021a; Li et al., 2021a) and other more general families of function approximation (Du et al., 2021; Jin et al., 2021a; Foster et al., 2021).

Offline RL. In contrast to online RL, offline RL assumes access to a pre-collected offline dataset and precludes active interactions with the environment. Given the absence of further data collection, the sample complexity of pure offline RL depends heavily upon the quality of the offline dataset at hand, which has often been characterized via some sorts of concentrability coefficients in prior works (Rashidinejad et al., 2021; Zhan et al., 2022). Earlier works (Munos and Szepesvári, 2008; Chen and Jiang, 2019) typically operated under the assumption of all-policy concentrability - namely, the assumption that the dataset covers the visited state-action pairs of all possible policies - thus imposing a stringent requirement for the offline dataset to be highly explorative. To circumvent this stringent assumption, Liu et al. (2020); Kumar et al. (2020); Jin et al. (2021b); Rashidinejad et al. (2021); Uehara and Sun (2021); Li et al. (2022); Yin et al. (2021); Shi et al. (2022); Yan et al. (2023) incorporated the pessimism principle amid uncertainty into the algorithm designs and, as a result, required only single-policy concentrability (so that the dataset only needs to cover the part of the state-action space reachable by the optimal policy). With regards to the basic tabular case, Li et al. (2022) proved that the pessimistic model-based offline algorithm is capable of achieving minimaxoptimal sample complexity for the full ε -range, accommodating both the episodic finite-horizon case and the discounted infinite-horizon analog. Moving beyond single-agent tabular settings, a recent line of works investigated offline RL in the presence of general function approximation (Jin et al., 2020c; Xie et al., 2021a; Zhan et al., 2022), environment shift (Zhou et al., 2021b; Shi and Chi, 2022), and in the context of zero-sum Markov games (Cui and Du, 2022; Yan et al., 2022).

Hybrid RL. While there were a number of empirical works (Rajeswaran et al., 2017; Vecerik et al., 2017; Kalashnikov et al., 2018; Hester et al., 2018; Nair et al., 2018, 2020) suggesting the perfromance gain of combining online RL with offline datasets (compared to pure online or offline learning), rigorous theoretical evidence remained highly limited. Ross and Bagnell (2012); Xie et al. (2021b) attempted to develop theoretical understanding by looking at one special hybrid scenario, where the agent can perform either of the following in each episode: (i) collecting a new online episode; and (ii) executing a prescribed and fixed reference policy to generate a sample episode. In this setting, Xie et al. (2021b) showed that in the minimax sense, combining online learning with samples generated by such a reference policy is not advantageous in comparison with pure online or offline RL. Note that our results do not contradict with the lower bound in Xie et al. (2021b), given that we exploit 'partial' single-policy concentrability that implies additional structure except for the worst case. Akin to the current paper, Song et al. (2022); Wagenmaker and Pacchiano (2022) studied online RL with additional access to an offline dataset. Nevertheless, Song et al. (2022) mainly focused on the issue of computational efficiency, and the algorithm proposed therein does not come with improved sample complexity. In contrast, Wagenmaker and Pacchiano (2022) focused attention on statistical efficiency, although the sample complexity derived therein is highly suboptimal when specialized to the tabular setting.

Reward-free and task-agnostic exploration. Reward-free and task-agnostic exploration, which refer to the scenario where the agent first collects online sample trajectories without guidance of any information about the reward function(s), has garnered much recent attention (Brafman and

Tennenholtz, 2002; Jin et al., 2020a; Zhang et al., 2020a, 2021b; Huang et al., 2022). Focusing on the tabular case, the earlier work Jin et al. (2020a) put forward a reward-free exploration scheme that achieves minimax optimality in terms of the dependency on S , A and 1 /ε , with the horizon dependency further improved by subsequent works (Kaufmann et al., 2021; Ménard et al., 2021a; Li et al., 2023). In particular, the exploration scheme proposed in Li et al. (2023) was shown to achieve minimax-optimal sample complexity when there exist a polynomial number of pre-determined but unseen reward functions of interest, which inspires the algorithm design of the present paper. Moreover, reward-free RL has been extended to account for function approximation, including both linear (Wang et al., 2020; Agarwal et al., 2020; Qiao and Wang, 2022; Zhang et al., 2021a; Wagenmaker et al., 2022) and nonlinear function classes (Chen et al., 2022a).

B Pseudocode of Algorithm 1

In this section, we provide the whole procedure for the proposed hybrid RL algorithm, with several subroutines deferred to Appendix C and Appendix D.

Algorithm 1: The proposed hybrid RL algorithm.

1 Input: offline dataset D off (containing K off trajectories), parameters N,K on , T max , learning rate η .

2 Initialize: π 1 h ( a | s ) = 1 /A for any ( s, a, h ) ; K = K off + K on ; split D off into two halves D off , 1 and D off , 2 .

/* Estimation of occupancy distributions for any policy π

. */

3 Call Algorithm 3, which allows one to specify ̂ d π h ( s, a ) for any deterministic policy π and any ( s, a, h ) .

/* Estimation of occupancy distributions of the historical data. */

4 Use the dataset D off , 1 to compute

for any ( s, a, h ) , where N off h ( s, a ) = ∑ K off / 2 k =1 1 ( s k h = s, a k h = a ) and c off > 0 is some absolute constant.

/* Compute a general sample-efficient online exploration scheme.

*/

5 Call Algorithm 5 with estimators ̂ d π to compute policy π explore and the associated weight µ explore . /* Compute an online exploration scheme tailored to the offline dataset.

*/

6 for t = 1 , · · · , T max do

7 Compute µ t using Algorithm 2.

8 Update π t +1 h ( a | s ) for all ( s, a, h ) ∈ S × A × [ H ] such that:

9 Set µ imitate = 1 T max ∑ T max t =1 µ t and π imitate = E π ∼ µ imitate [ π ] .

/* Sampling using the above two exploration

policies. */

10 Collect K on imitate (resp. K on explore ) sample trajectories using π imitate (resp. π explore ) to form a dataset D on imitate (resp. D on explore ).

/* Run the model-based offline RL algorithm.

11 Apply Algorithm 6 to the dataset D = D off , 2 ∪ D on imitate ∪ D on explore to compute a policy ̂ π .

12 Output: policy ̂ π .

*/

C Subroutine for solving the subproblem (20b)

While (20b) is a convex optimization subproblem, it involves optimization over a parameter space with exponentially large dimensions. In order to solve it in a computationally feasible manner, we propose a tailored subroutine based on the Frank-Wolfe algorithm (Bubeck, 2015).

Before proceeding, recall that when specifying ̂ d π in Algorithm 3, we draw N independent trajectories { s n,h 1 , a n,h 1 , . . . , s n,h h +1 } 1 ≤ n ≤ N and compute an empirical estimate ̂ P h of the probability transition kernel at step h such that

The proposed Frank-Wolfe-type algorithm. With this set of notation in place and with an initial guess taken to be the indicator function µ (1) = 1 π init for an arbitrary policy π init ∈ Π , the k -th iteration of our iterative procedure for solving (20b) can be described as follows.

Computing a search direction. The search direction of the Frank-Wolfe algorithm is typically taken to be a feasible direction that maximizes its correlation with the gradient of the objective function (Bubeck, 2015). When specialized to the current sub-problem (20b), the search direction can be taken to be the Dirac measure δ π ( k ) , where

As it turns out, this optimization problem (32) can be efficiently solved by applying dynamic programming (Bertsekas, 2017) to an augmented MDP M off constructed as follows.

-Introduce an augmented finite-horizon MDP M off = ( S ∪ { s aug } , A , H, ̂ P aug , r off ) , where s aug is an augmented state. We choose the reward function to be

and the probability transition kernel as

Frank-Wolfe updates. We then update the iterate µ ( k +1) as a convex combination of the current iterate and the direction found in the previous step:

where the stepsize is chosen to be

6

7

8

9 Output: the policy mixture µ t +1 = µ ( k ) .

D Useful algorithmic subroutines from prior works

In this section, we provide precise descriptions of several useful algorithmic subroutines that have been developed in recent works. The algorithm procedures are directly quoted from these prior works, with slight modification.

Stopping rule. It is also necessary to specify the stopping rule of the above iterative procedure. Throughout this paper, the above subroutine will terminate as long as

with the final output taken to be µ t +1 = µ ( k ) . We shall justify the feasibility of this stopping rule (namely, the fact that this stopping criterion can be met by some mixed policy) in Appendix F.

Iteration complexity. Encouragingly, the above subroutine in conjunction with the stopping rule (37) leads to an iteration complexity no larger than

The proof of this claim is postponed to Section F.

Algorithm 2: Subroutine for solving the sub-problem (20b).

1

Initialize

:

µ

(1)

=

1

3

Exit for-loop if the following condition is met: // stopping criterion

4

π

for an arbitrary policy

π

init

∈

Π

.

init

/* Find the search direction */

5 Compute the optimal deterministic policy π ( k ) , aug of the MDP

M off = ( S ∪ { s aug } , A , H, ̂ P aug , r off ) , where s aug is an augmented state,

and the augmented probability transition kernel is given by

Let π ( k ) be the corresponding optimal deterministic policy of π ( k ) , aug in the original state space.

D.1 Subroutine: occupancy estimation for any policy π

The first subroutine we'd like to describe is concerned with estimating the occupancy distribution d π induced by any policy π , based on a carefully designed exploration strategy. This algorithm, proposed by Li et al. (2023), seeks to estimate { d π h } step by step (i.e., from h = 1 , . . . , H ). For each h , it computes an appropriate exploration policy π explore ,h to adequately explore what happens between step h and step h +1 , and then collect N sample trajectories using π explore ,h . These turns allow us to estimate the occupancy distribution d π h +1 for step h +1 . See Algorithm 3 for a precise description.

Algorithm 3: Subroutine for estimating occupancy distributions for any policy π (Li et al., 2023).

1 Input: target success probability 1 -δ , threshold ξ = c ξ H 3 S 3 A 3 log( HSA/δ ) .

/* Estimate occupancy distributions for step 1 . */

2 Draw N independent episodes (using arbitrary policies), whose initial states are i.i.d. drawn from s n, 0 1 i . i . d . ∼ ρ (1 ≤ n ≤ N ) . Define the following functions

for any deterministic policy π : S × [ H ] → ∆( A ) and any ( s, a ) ∈ S × A . (Note that these functions are defined for future use and not computed for the moment, as we have not specified policy π .)

/* Estimate occupancy distributions for steps 2 , . . . , H . */ 3 for h = 1 to H -1 do /* Collect N sample trajectories using a suitable exploration policy. */ 4 Call Algorithm 4 to compute an exploration policy π explore ,h and compute an estimate ̂ P h of the true transition kernel P h . /* Specify how to compute ̂ d π h +1 for any policy π . */ 5 For any deterministic policy π : S × [ H ] → ∆( A ) and any ( s, a ) ∈ S × A , define ̂ d π h +1 ( s ) = 〈 ̂ P h ( s | · , · ) , ̂ d π h ( · , · ) 〉 , ̂ d π h +1 ( s, a ) = ̂ d π h +1 ( s ) π h +1 ( a | s ) . (43)

We note, however, that Algorithm 3 requires another subroutine to compute a suitable exploration policy π explore ,h . As it turns out, this can be accomplished by approximately solving the following problem

via the Frank-Wolfe algorithm and returning π explore ,h = E π ∼ ̂ µ h [ π ] . See Algorithm 4 for details.

D.2 Subroutine: reward-agnostic online exploration

Based on the estimated occupancy distributions specified in Algorithm 3, Li et al. (2023) proposed a reward-independent online exploration scheme that proves useful in exploring an unknown environment. In a nutshell, this scheme computes a desired exploration policy by approximately solving the following optimization sub-problem:

again using the Frank-Wolfe algorithm; the resulting policy takes the form of a mixture of deterministic policies, as given by π explore = E π ∼ µ explore [ π ] . This exploration policy is then employed to execute the MDP for a number of times in order to collect enough information about the unknowns. See Algorithm 5 for the whole procedure.

Algorithm 4: Subroutine for computing the exploration policy for step h in occupancy estimation (Li et al., 2023).

1 Initialize : µ (0) = 1 π init for an arbitrary policy π init ∈ Π , T max = glyph[floorleft] 50 SA log( KH ) glyph[floorright] .

2

3

5

6

7

8

9

for

t

= 0

/*

to

T

do the optimal

policy

*/

Compute the optimal deterministic policy π ( t ) , b of the augmented MDP

M h b = ( S ∪ { s aug } , A , H, ̂ P aug ,h , r h b ) , where s aug is an augmented state, glyph[negationslash]

and the augmented probability transition kernel is defined as

Let π ( t ) be the corresponding optimal deterministic policy of π ( t ) , b in the original state space.

Compute // choose the stepsize

Here, ̂ d π h ( s, a ) is computed via (42) for h = 1 , and (43) for h ≥ 2 .

If

g

(

π

(

t

)

,

̂

d, µ

(

t

)

)

2

SA

then exit for-loop

10 Set π explore ,h = E π ∼ µ ( t ) [ π ] with ̂ µ h = µ ( t ) . // The final exploration policy for step

Draw

/*

h

.

samples using

π

explore

,h to

estimate the transition

kernel.

*/

11 Draw N independent trajectories { s n,h 1 , a n,h 1 , . . . , s n,h h +1 } 1 ≤ n ≤ N using policy π explore ,h and compute

12 Output: the exploration policy π explore ,h , the weight ̂ µ h , and the estimated kernel ̂ P h .

D.3 Subroutine: pessimistic model-based offline RL

Given a historical dataset containing a collection of statistically independent sample trajectories, Li et al. (2022) came up with a model-based approach that enjoys provable minimax optimality. This approach first employs a two-fold subsampling trick in order to decouple the statistical dependency across different steps of a single trajectory. After this subsampling step, this approach resorts to the principle of pessimism in the face of uncertainty, which employs value iteration but penalizes the updates via proper variance-aware penalization (i.e., Bernstein-style lower confidence bounds). Details can be found in Algorithm 6.

.

//

stopping rule

3

4

5

6

7

8

9

Algorithm 5: Subroutine for computing the desired online exploration policy (Li et al., 2023).

/* find the optimal policy */

Compute the optimal deterministic policy π ( t ) , b of the MDP

M b = ( S ∪ { s aug } , A , H, ̂ P aug , r b ) , where s aug is an augmented state,

and the augmented probability transition kernel is given by

Let π ( t ) be the corresponding optimal deterministic policy of π ( t ) , b in the original state space.

Compute // choose the stepsize

Here, ̂ d π h ( s, a ) is computed via (42) for h = 1 , and (43) for h ≥ 2 . If g ( π ( t ) , ̂ d, µ ( t ) b ) ≤ 2 HSA then exit for-loop . // stopping rule Update // Frank-Wolfe update

10 Output: the exploration policy π explore = E π ∼ µ ( t ) b [ π ] and the associated weight µ explore = µ ( t ) b .

Algorithm 6: A pessimistic model-based offline RL algorithm (Li et al., 2022).

1 Input: a dataset D ; reward function r . Let K 0 denote the number of sample trajectories in D .

2 Subsampling: run the following procedure to generate the subsampled dataset D trim .

Data splitting. Split D into two halves: D main (which contains the first K 0 / 2 trajectories), and D aux (which contains the remaining K 0 / 2 trajectories); we let N main h ( s ) (resp. N aux h ( s ) ) denote the number of sample transitions in D main (resp. D aux ) that transition from state s at step h .

Lower bounding { N main h ( s ) } using D aux . For each s ∈ S and 1 ≤ h ≤ H , compute

Random subsampling. Let D main ′ be the set of all sample transitions (i.e., the quadruples taking the form ( s, a, h, s ′ ) ) from D main . Subsample D main ′ to obtain D trim , such that for each ( s, h ) ∈ S × [ H ] , D trim contains min { N trim h ( s ) , N main h ( s ) } sample transitions randomly drawn from D main ′ . (We shall also let N trim h ( s, a ) denote the number of samples that visits ( s, a, h ) in D trim .)

3 Run VI-LCB: set D 0 = D trim ; run Algorithm 7 to compute a policy ̂ π .

E Analysis of Theorem 1

In this section, we present the proof for our main result in Theorem 1. Throughout the proof, we let {G h } 1 ≤ h ≤ H denote a sequence of subsets obeying

as motivated by Definition 2. As it turns out, if K on ≥ c 1 H 3 SA ε 2 log K δ for some large enough constant c 1 > 0 , then the claimed result in Theorem 1 follows immediately from the main theory in Li et al. (2023) developed for pure online exploration. As a result, it sufficies to prove the theorem by replacing Condition (24b) with

throughout this section.

On a high level, our proof comprises the following three steps:

Establish the proximity of ̂ d off (resp. ̂ d π ) and d off (resp. d π ).

Show that the mixed policy π imitate is able to mimic and strengthen the offline dataset D off , while the mixed policy π explore is capable of exploring the part of the state-action space that has not been adequately visited by D off .

Derive the sub-optimality of the policy returned by the offline RL algorithm (i.e., Algorithm 6) when applied to the hybrid dataset D = D off , 2 ∪ D on imitate ∪ D on explore .

In the sequel, we shall elaborate on these three steps.

E.1 Step 1: establishing the proximity of ̂ d π (resp. ̂ d off ) and d π (resp. d off )

To begin with, the goodness of the occupancy distribution estimators ̂ d π (cf. Algorithm 3) has been analyzed in Li et al. (2023, Lemma 4), which come with the following performance guarantees.

Lemma 1 (Li et al. (2023)) . Recall that ξ = c ξ H 3 S 3 A 3 log HSA δ for some large enough constant c ξ > 0 . With probability at least 1 -δ , the estimated occupancy distributions specified in Algorithm 3 satisfy

simultaneously for all ( s, a, h ) ∈ S × A × [ H ] and all deterministic policy π ∈ Π , provided that

for some large enough constant C N > 0 . Here, { e π h ( s, a ) } is some non-negative sequence satisfying

(56)

We now turn to the estimator ̂ d off (cf. (15)) for the occupancy distribution of the offline dataset, for which we begin with the following lemma concerning the proximity of d off h and ̂ d off h . The proof of this lemma is deferred to Section G.1.

Lemma 2. Suppose that c off ≥ 48 . With probability at least 1 -δ/ 3 , one has

simultaneously for all ( s, a, h ) ∈ S × A × [ H ] .

This lemma implies that: when d off h ( s, a ) glyph[lessorsimilar] log HSA δ K off + H 4 S 4 A 4 log HSA δ N + SA K on , the estimator ̂ d off h ( s, a ) might be unable to track d off h ( s, a ) in a faithful manner. This motivates us to single out the following two subsets of state-action pairs for which ̂ d off h ( s, a ) might become problematic at step h :

the set G c h (see (11) for the definition of G h ), which corresponds to the set of optimal stateaction pairs that even the true data distribution d off h cannot cover adequately;

another set T small h defined as

comprising those state-action pairs for which ̂ d off h ( s, a ) might not be a faithful estimator of d off h ( s, a ) .

In what follow, we shall adopt the notation:

It is straightforward to demonstrate that:

For any ( s, a ) / ∈ T small h , it is seen from Lemma 2 that

For any ( s, a ) ∈ G h , Condition (52) tells us that

As a consequence, any ( s, a ) / ∈ T h necessarily obeys

Another useful observation that we can readily make is as follows:

Here, the second and the third lines arise from Condition (52), the penultimate line invokes the definition (58) of T small h , whereas the last line is valid since N = K on / (3 H ) (see (14)).

E.2 Step 2: showing that π imitate (resp. π explore ) covers ̂ d off (resp. d π glyph[star] ) adequately

In this step, we aim to demonstrate the quality of the two exploration policies π imitate and π explore , designed for different purposes.

Goodness of π imitate . We begin by assessing the quality of the exploration policy π imitate . Towards this, we first make note of the following crude bound:

In view of the convergence guarantees for FTRL (Shalev-Shwartz, 2012, Corollary 2.16), we see that: if η = √ log A 2 T max L 2 = √ log A 2 T max ( K on H ) 2 , then running FTRL for T max iterations results in

for all s ∈ S and 1 ≤ h ≤ H . Therefore, recalling that µ imitate = 1 T max ∑ T max t =1 µ t and applying Jensen's inequality yield

where the second inequality results from (64). In addition, it follows from the stopping rule (37) that

As a consequence, combining (65) and (66) yields

provided that T max ≥ 2( K on H ) 2 log A . The fact that the left-hand side of (67) is well-controlled suggests that π imitate is able to cover ̂ d off adequately, a crucial fact we shall rely on in the subsequent analysis.

Goodness of π explore . Next, we turn attention to the other exploration policy π explore , computed via Algorithm 5. The following performance guarantees have been established in Li et al. (2023, Section 3.2).

Lemma 3. The distribution µ explore ∈ ∆(Π) returned by Algorithm 5 satisfies

In light of the performance bound (17) for the subsequent offline RL approach, Lemma 3 suggests that π explore is able to explore well with regards to the visitation of any policy π -including the optimal policy π glyph[star] .

E.3 Step 3: establishing the performance of offline RL

Now, we can readily proceed to analyze the performance of the model-based offline procedure described in Algorithm 6. In this subsection, we abuse the notation ̂ P to represent the empirical transition kernel constructed within the offline subroutine in Algorithm 7. Additionally, we introduce a S -dimensional vector d π glyph[star] h := [ d π glyph[star] h ( s )] s ∈S .

E.3.1 Step 3.1: error decomposition

To begin with, we convert the sub-optimality gap of the policy estimate ̂ π into several terms that shall be controlled separately. The following two preliminary facts, which have been established in Li et al. (2022), prove useful for this purpose.

Lemma 4. With probability exceeding 1 -δ/ 3 , one has

and

where b h ( s, a ) is defined in line 7 of Algorithm 7.

In view of Lemma 4, we can derive, for all j ∈ [ H ] ,

Here, the second line comes from (62), whereas the third line is due to Lemma 1 and the basic fact that b h ( s, a ) ≤ H (see line 7 of Algorithm 7). Substituting the definition of b h (see line 7 of Algorithm 7) into the above display and applying Lemma 4, we arrive at

where we recall that c b > 0 is also an absolute constant used to specify b h ( s, a ) .

It is worth noting that the right-hand side of (68) involves a variance term Var ̂ P h ( ·| s,a ) ( ̂ V h +1 ) w.r.t. the empirical model ̂ P . As it turns out, the following lemma established in Li et al. (2022, Lemma 8) makes apparent the intimate connection between Var ̂ P h ( ·| s,a ) ( ̂ V h +1 ) and Var P h ( ·| s,a ) ( ̂ V h +1 ) .

Lemma 5. With probability exceeding 1 -δ/ 3 , we have, for all ( s, a, h ) ∈ S × A × [ H ]

Substituting the result of Lemma 5 into (68) leads to

where

This leaves us with two terms to bound, which we shall accomplish separately in the ensuing two steps.

E.3.2 Step 3.2: controlling γ 1 in (69)

Regarding the first term γ 1 on the right-hand side of (69), let us first define the set I h as follows:

where we remind the reader that ξ = c ξ H 3 S 3 A 3 log HSA δ for some constant c ξ > 0 . Armed with this set, we can deduce that

where the first inequality arises from (62), the penulminate line utilizes the definition (70) of I h , and the last line comes from (67). This in turn allows us to upper bound γ 1 as follows:

where the last line makes use of the elementary fact that min { x y , u w } ≤ x + u y + w for any x, y, u, w > 0 . In addition, note that for any s obeying E imitate [ d π ( s ) ] ≥ ξ/N , we have

π ∼ µ h

where the last line invokes the definition of I h and the choice NH = 1 3 K on . It can then be straightforwardly justified using elementary concentration inequalities (see, e.g., Alon and Spencer (2016, Appendix A.1)) that: with probability exceeding 1 -δ/ 10 ,

holds simultaneously for all ( s, h ) ∈ S × [ H ] , and as a result,

Moreover, for any ( s, a ) ∈ I h (cf. (70)), one can invoke Lemma 2 to obtain

Applying the same concentration of measurement argument as above further reveals that:

any ( s, a ) ∈ I h . Substitution into (71) then gives

By virtue of the Cauchy-Schwarz inequality, we can further derive

To further control this term, we resort to the following lemma, whose proof is deferred to Section G.2.

Lemma 6. With probability at least 1 -δ/ 6 , we have, for all j ∈ [ H ] ,

for some sufficiently large constant c 11 > 0 .

Putting Lemma 6 together with (67) and (73), we obtain

Substitution into (72) results in

Akin to (72) and (75), we can also focus on the offline dataset and obtain

Combine (75) and (76) to arrive at

E.3.3 Step 3.3: controlling γ 2 in (69)

We now turn attention to the term γ 2 on the right-hand side of (69). Akin to (72), we can deduce that

provided that

The Cauchy-Schwarz inequality then tells us that

where ̂ σ is defined in (63). Here, the penultimate line invokes Lemma 3, and the last line is valid since (according to Lemma 1 and (63))

Substitution of the above inequality into (78) yields

E.3.4 Step 3.4: putting all pieces together

To finish up, combining (69),(77) and (79) reveals that: with probability at least 1 -δ , one has

where the last inequality holds true as long as min { K off , K on } glyph[greaterorsimilar] HSA . Taking the right-hand side of (80) to be no larger than ε , we immediately establish Theorem 1 under the sample complexity assumption in this theorem.

F Proof for the stopping criterion and the iteration complexity for solving (20b)

Feasibility of the stopping rule (37) . We first demonstrate that the stopping rule (37) can be satisfied by some mixed policy, namely,

Towards this end, we focus attention on analyzing a specific choice of the mixed policy µ off -the one that represents the mixed policy that generates the offline dataset. Making use of the definition (15) of ̂ d off gives

where the second line relies on (94). This combined with Lemma 1 results in

Moreover, inequality (56) tells us that: when d off h ( s, a ) ≥ 2 3 c off ( H 4 S 4 A 4 log HSA δ N + SA K on ) for some large enough constant c off > 0 , we have

In turn, this implies that

Consequently, combine (83) and (85) to yield

which clearly validates the claim (81) (with an even better pre-constant).

Before moving forward, we single out one useful property that arises from the above arguments:

To prove the validity of this claim (87), it suffices to make the following two observations:

When d off h ( s, a ) ≥ 2 3 c off ( H 4 S 4 A 4 log HSA δ N + SA K on ) , it has been shown in (83) and (84) in conjunction with (82) that

When d off h ( s, a ) < 2 3 c off ( H 4 S 4 A 4 log HSA δ N + SA K on ) , one sees from (82) that ̂ d off h ( s, a ) = 0 , and hence (87) holds true trivially.

Iteration complexity. Suppose that the stopping criterion (37) is not yet met in the k -th iteration, namely,

It can be easily seen that

where the first inequality follows from the choice (32) of π ( k ) , the second inequality is a consequence of the relation (87), and the last line makes use of (89). This in turn allows one to demonstrate that

where the last line results from (90) and the condition (89). We can then readily derive

where the third line relies on the update rule (35), and the last line utilizes the choice (36) of α .

In summary, the above argument reveals that: before the stopping criterion is met, each iteration is able to make progress at least as large as in (91). Recognizing the crude bound

that holds for any µ ∈ ∆(Π) , one can combine this with (91) to conclude that: the proposed procedure terminates within O ( K 4 H 4 S 2 ) iterations, as claimed.

G Proofs of technical lemmas

G.1 Proof of Lemma 2

The Bernstein inequality combined with the union bound tells us that, with probability at least 1 -δ/ 3 ,

holds simultaneously for all ( s, a, h ) ∈ S × A × [ H ] , where the last line invokes the AM-GM inequality. This in turn reveals that

As a result, we can show that:

If instead N off h ( s,a ) K off < c off ( log HSA δ K off + H 4 S 4 A 4 log HSA δ N + SA K on ) , then one has ̂ d off h ( s, a ) = 0 , and therefore,

Taken collectively, these inequalities demonstrate that

provided that c off > 0 is sufficiently large.

G.2 Proof of Lemma 6

Before embarking on the proof, let us introduce several notation. For each 1 ≤ h ≤ H , define P π glyph[star] h ∈ R S × S and d π glyph[star] h ∈ R S such that: for all s ∈ S ,

holds for any j ∈ [ H ] .Note that we have bounded γ 2 in (79). We then need to bound γ 3 . With regards to the term γ 3 : invoking similar arguments as for (72) leads to

where the second step invokes the Cauchy-Schwartz inequality, and the last line comes from (67). Similarly, repeating the above argument but focusing on the offline dataset, we can derive (which we omit for the sake of brevity)

where the last line makes use of (86).

Combining (98), (99) and (79) with (97), we can show that

for all 1 ≤ j ≤ H , with the proviso that

for some sufficiently small constant c 10 > 0 . As a consequence, we can demonstrate that

for all 1 ≤ j ≤ H . Here, the third line in (101) applies the following fact

where the second identity comes from the Bellman equation; the third relation uses the fact that V glyph[star] h ( s ) ≤ H , and the penultimate line holds since ( d π glyph[star] h ) glyph[latticetop] P π glyph[star] h = ( d π glyph[star] h +1 ) glyph[latticetop] ; the penultimate step in (101) is due to (100); and the line in (101) holds true since ( d π glyph[star] h ) glyph[latticetop] P π glyph[star] h = ( d π glyph[star] h +1 ) glyph[latticetop] .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors prove that augmenting an offline dataset with online data collection leads to improved sample complexity compared to online and offline RL in the tabular setting, under the condition of single-policy partial concentrability. This is an extension of previous work on single-policy concentrability, relaxing the requirement that the offline dataset cover the full state-action space of the optimal policy. The authors propose a three step algorithm, given a budget of $K^{on}$ online data collection steps - spend the first third estimating occupancy distributions, the next two thirds on imitating the provided dataset and an exploration focused policy respectively, and finally finding a policy using standard offline RL on the original and collected datasets. This algorithm can be run in polynomial time using a Frank-Wolfe approximation to find the optimal policies in the second step.\n\nSTRENGTHS:\n- The algorithm and proof are both novel to the best of my knowledge, and address an important question in RL theory. \n- The result is a general improvement over previous work, relaxing the coverage requirements without imposing new constraints or significant additional computation.  \n- The paper is generally well written without many typos or errors.\n\nWEAKNESSES:\n- Some of the choices in the algorithm could be better justified in the main text. Without referring to the Appendix the motivation for the imitation and exploration datasets is difficult to understand. \n- While the Frank-Wolfe updates proposed include a polynomial stopping rule, it's not clear how feasible this bound would be in any realistic task.\n\nMinor:\n- Section 2 (Line 123): noation -> notation\n\nQUESTIONS:\n- What is the definition of $f_{mixed}$ in section 4?\n- Intuitively, what do the terms $\\gamma_1$ and $\\gamma_2$ represent?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new three step algorithm in the hybrid RL setting for tabular MDPs that improves in terms of sample complexity on the pure online and as well as the pure offline setting. The notion of single-policy partial concentrability is introduced which offers an intuitive concept that connects the sample complexity of pure online and pure offline RL. The algorithm leverages results from reward agnostic exploration to explore the parts of the state-action space that are missing from the offline dataset and then applies an offline learning algorithm to  find an $\\epsilon$-suboptimal policy. The sample complexity result is compared to existing results and the algorithms advantages and limitations are discussed.\n\nSTRENGTHS:\nThe paper is well written and reads itself easily, the algorithm and the sample complexity result are introduced in a digestible manner without loss in rigor.\n\nThe concept of single-policy partial concentrability captures the two problems with offline datasets well (lack of coverage and mismatch with the state-action distribution of the optimal policy) and provides a neat connection between sample complexity results of pure online and pure offline approaches. \n\nFor the tabular setting without function approximation the sample complexity result is SOTA. \n\nThe advantages and limitations of the algorithms are extensively discussed.\n\nWEAKNESSES:\nThe motivation/intuition why the number of samples coming from the offline dataset distribution needs to be increased with the imitation exploration policy is not clear to me. Why is the offline dataset and the dataset collected with $\\pi^{explore}$ not enough ?\n\nI find the notation used for the single-policy partial concentrability coefficient slightly confusing: \n\n- In equation 10 I would write $\\\\{G_{h}\\\\}_{1 \\leq h \\leq H} \\in G(\\sigma)$\n- in equation 11 I would write $G(\\sigma) \\coloneqq \\\\{ \\\\{ G_{h}\\\\}_{1 \\leq h \\leq H} | \\forall 1 \\leq h \\leq H , G_h \\subseteq S \\times A, ... \\\\} $\n\nQUESTIONS:\nHow does the sample complexity result from \"Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient\" (Song et al., 2023) compare to the presented result? The mentioned paper looks at bilinear models so similarly to Wagenmaker and Pacchiano is not specifically designed for the tabular case.\n\nLIMITATIONS:\nThe limitations are discussed. An empirical comparison of the presented algorithm with other hybrid RL algorithms would have been interesting but seems out of scope of the presented work.  \n\nNegative Societal Impact - Not Applicable", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies hybrid reinforcement learning under tabular MDPs, where the agent has access to both a given offline dataset and online interaction with environment. This paper first proposes an interesting notion of single-policy partial concentrability, which relaxs previous concentrability assumption and accounts for the trade offs between distributional mismatch and partial converage. Then this paper proposes a novel three stage algorithm, which can automatically identify the optiaml trade offs. Then the paper provides a sample complexity upper bound. The sample complexity result shows the benefit of hybrid RL compared with pure online or offline RL.

STRENGTHS:
1. The notion of single-policy partial concentrability is interesting and reasonable to me and relaxs previous concentrability assumption. 

2. This paper proposes a novel three-stage algorithm for hybrid RL, which can automatically identify the optiaml trade offs between distributional mismatch and partial converage. Intuitively, the agent can imitate the offline dataset to collect partial informative data, and explore the unknown data which is not covered by offline dataset.

3. When compared to pure online or offline results, the samplexity complexity result shows the benefit of hybrid RL.

WEAKNESSES:
1. This paper mentions Frank-Wolfe-type subroutine in introduction and conclusion in main body. However the Frank-Wolfe-type subroutine is even not discussed in main body. Also, pseudocode of algorithm 1 is deferred in Appendix, which make it not easy to read and follow the idea of algorithm.

2. This paper only provides theoretical results without experiment results.

QUESTIONS:
As listed in weakness.

LIMITATIONS:
There is no negative impact in this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper investigates tabular reinforcement learning (RL) in a hybrid setting, where both an offline dataset and online interactions with an unknown environment are available. The main question addressed is how to effectively use online data to enhance and complement the offline dataset for policy fine-tuning. The authors propose a three-stage hybrid RL algorithm that outperforms pure offline RL and pure online RL in terms of sample complexity. The algorithm does not require reward information during data collection.

STRENGTHS:
- Innovative approach: The paper introduces a novel three-stage hybrid RL algorithm that combines online and offline data to improve sample complexity in tabular RL.
- Theoretical contributions: The authors develop a new notion, single-policy partial concentrability, which captures the trade-off between distribution mismatch and lack of coverage. They provide theoretical results that demonstrate the statistical benefits of hybrid RL compared to pure online and offline RL.

WEAKNESSES:
- The paper provides no empirical evaluations or demonstrations of the proposed algorithm. Neither does it shed light on the design of practical algorithms.
- There remain some issues unsolved in the paper. See the questions for details.

QUESTIONS:
- Empirical estimation of the occupancy distribution in Sec. 3.1 can have high variance. How does the proposed approach handle this issue?
- Why do we need to imitate the offline dataset? What if the dataset is collected with random policies?
- The analyses are conducted in the tabular case? How to extend them to continuous state/action spaces?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a tabular reinforcement learning setting where the agent tries to learn the optimal policy by both offline data (obtained by exploiting the optimal policy) and online exploration. A three-stage algorithm is proposed to obtain such a policy, which has lower sample complexity compared to pure online and pure offline reinforcement learning algorithms.\n\nSTRENGTHS:\nThis paper proposes the first algorithm for the hybrid RL setting that can achieve lower sample complexity than pure online and pure offline algorithms, which is innovative and inspiring. Overall, The writing is clear.\n\nWEAKNESSES:\nThere are a few crucial technique details that are not clear enough, which should be well justified. Please see my questions below.\n\nQUESTIONS:\nLine 120: what do the authors mean by a \"unknown\" initial state distribution? Does its value depend on specific applications?\n\nLine 126-127: What is the difference between a mixed deterministic policy and a randomized one? Could the author provide a small example to elucidate their difference?\n\nDefinition 1: I think the definition of $C^\\star$ is associated with some policy $\\pi^\\star$. If so, the authors should specify this.\n\nLine 195: How should one decide which samples belong to the first/second half? Or can we divide the two halves arbitrarily?\n\nIn the \"Imitating the offline dataset\" step, the authors appear to assume that the offline dataset is obtained by executing an optimal policy. (1) If so, I think the authors should formalize this assumption. (2) Also, this assumption is strong (where the information of the optimality is leaked to the decision maker). Could the authors justify this assumption?\n\nIt feels fishy that authors claimed that \"Towards this end, it suffices to invoke the reward-agnostic online exploration scheme\" in line 250, while they assumed that \"the reward function is fully revealed upon completion of online data collection\" in line 118.\n\nLIMITATIONS:\nPlease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies hybrid RL (online RL with offline data) in the specialized setting of tabular MDPs, while providing tighter bounds than prior works. Namely, they seem to propose a novel and weaker concentrability notion called single-policy partial concentrability. The algorithm takes three stages: 1. estimate occupancy distribution, 2. perform online exploration, 3. invoke offline RL for policy learning.\n\nSTRENGTHS:\n1. A weaker notion of partial coverage for hybrid RL is proposed in Def 2.\n2. While I'm not familiar with all previous hybrid RL works, this paper seems to be one of the first that obtains better sample complexity in online RL, in the tabular setting.\n\nWEAKNESSES:\n1. While the paper is reasonably clear, I think the presentation is a bit incomplete and can be made more succinct. For instance, the main algorithm of the paper is not even in the main paper and requires reading the appendix.\n2. The algorithm runs in three stages and is quite complicated for solving the tabular MDP problem. Also, I feel uneasy about Stage 1 of the algorithm, since estimating occupancy distributions is a hard problem in practice, eg. the hardness of DICE algorithms in offline RL is precisely due to this.\n3. Can you discuss the computationally complexity of each stage, compared to optimal tabular methods like UCBVI? In Line 340, it says the FW subroutine might be expensive, but can you be more concrete and derive a big-Oh running time? My worry is that this paper, while obtaining better instance-dependent statistical bounds sacrifice the practicality of the algorithm. There are also no experiments to validate the practicality of the new coverage assumption.\n\nQUESTIONS:\n1. How is f_{mixed} is defined in Eq. 26 and 27? I couldn't find a clear definition.\n2. For Eq. 27, isn't the minimax-optimal result by Azar supposed to be $H^2 SA/\\epsilon^2$ trajectories? \n3. Where does the current analysis break when extending to more general MDPs, like linear MDPs?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}