Title: Hybrid Reinforcement Learning Breaks Sample Size Barriers In Linear MDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: hybrid RL

reinforcement learning

linear MDPs

minimax
Abstract: Hybrid Reinforcement Learning (RL), where an agent learns from both an offline dataset and online explorations in an unknown environment, has garnered significant recent interest. A crucial question posed by Xie et al. (2022) is whether hybrid RL can improve upon the existing lower bounds established in purely offline and purely online RL without relying on the single-policy concentrability assumption. 
While Li et al. (2023) provided an affirmative answer to this question in the tabular PAC RL case, the question remains unsettled for both the regret-minimizing RL case and the non-tabular case. In this work, building upon recent advancements in offline RL and reward-agnostic exploration, we develop computationally efficient algorithms for both PAC and regret-minimizing RL with linear function approximation, without requiring concentrability on the entire state-action space. We demonstrate that these algorithms achieve sharper error or regret bounds that are no worse than, and can improve on, the optimal sample complexity in offline RL (the first algorithm, for PAC RL) and online RL (the second algorithm, for regret-minimizing RL) in linear Markov decision processes (MDPs), regardless of the quality of the behavior policy. To our knowledge, this work establishes the tightest theoretical guarantees currently available for hybrid RL in linear MDPs.

Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs

Abstract

Hybrid Reinforcement Learning (RL), where an agent learns from both an offline dataset and online explorations in an unknown environment, has garnered significant recent interest. A crucial question posed by Xie et al. (2022b) is whether hybrid RL can improve upon the existing lower bounds established for purely offline or online RL without requiring that the behavior policy visit every state and action the optimal policy does. While Li et al. (2023b) provided an affirmative answer for tabular PAC RL, the question remains unsettled for both the regretminimizing and non-tabular cases. In this work, building upon recent advancements in offline RL and reward-agnostic exploration, we develop computationally efficient algorithms for both PAC and regret-minimizing RL with linear function approximation, without requiring concentrability on the entire state-action space. We demonstrate that these algorithms achieve sharper error or regret bounds that are no worse than, and can improve on, the optimal sample complexity in offline RL (the first algorithm, for PAC RL) and online RL (the second algorithm, for regret-minimizing RL) in linear Markov decision processes (MDPs), regardless of the quality of the behavior policy. To our knowledge, this work establishes the tightest theoretical guarantees currently available for hybrid RL in linear MDPs.

1 Introduction

Reinforcement learning (RL) holds great promise in attaining reliable decision-making in adaptive environments for a broad range of modern applications. Typical RL algorithms often require an enormous number of training samples, motivating a line of recent efforts to study the sample efficiency of RL algorithms. There are two mainstream paradigms of RL, distinguished by how samples are collected: online RL and offline RL. In online RL, an agent learns in a real-time manner, exploring the environment to maximize her cumulative rewards by executing a sequence of adaptively chosen policies (e.g. Azar et al. (2017); Jin et al. (2018); Sutton and Barto (2018); Zhang et al. (2023)). Whereas, in offline RL, an agent has only access to a pre-collected dataset, and tries to figure out how to perform well in a different environment without ever experiencing it (e.g. Jin et al. (2021b); Lange et al. (2012); Levine et al. (2020); Li et al. (2024)). Online methods are often sample-hungry, but offline methods often impose stringent requirements on the quality of the pre-collected data.

To address the limitations of both, the setting of hybrid RL (Song et al., 2023; Xie et al., 2022b) has recently received considerable attention from both theoretical and practical perspectives (Amortila et al., 2024; Ball et al., 2023; Kausik et al., 2024; Li et al., 2023b; Nair et al., 2020; Nakamoto et al., 2023; Song et al., 2023; Tan and Xu, 2024; Vecerik et al., 2017; Wagenmaker and Pacchiano, 2023; Zhou et al., 2023). In hybrid RL, an agent learns from a combination of both offline and online data, extracting information from offline data to enhance online exploration. Theoretical guarantees for hybrid RL algorithms can be categorized on: (1) the type of function approximation considered, (2) the level of coverage required by the behavior policy, (3) whether it improves on the minimax lower bounds for online-only and offline-only learning, and (4) whether they minimize regret or obtain a PAC guarantee. We elaborate below, and summarize the prior art in Table 1.

While much of the prior literature (Amortila et al., 2024; Nakamoto et al., 2023; Song et al., 2023; Tan and Xu, 2024; Zhou et al., 2023) tackles general function approximation in hybrid RL, they either require stringent concentrability assumptions on behavior policy quality, or fail to obtain tight theoretical guarantees. Under such single-policy concentrability assumptions (explained below), Xie et al. (2022b) show the optimal RL algorithm is either a purely offline reduction or a purely online algorithm if the agent can choose the ratio of offline to online samples, rendering the benefits of hybrid RL questionable. Without this assumption, Li et al. (2023b) show guarantees for PAC RL that improve over lower bounds for offline-only and online-only RL, but only for tabular MDPs.

This paper focuses on obtaining sharper theoretical guarantees in the setting of linear function approximation in linear MDPs . First proposed in Jin et al. (2019); Yang and Wang (2019), linear MDPs parameterize the transition probability kernel and reward function by linear functions of known features (e.g. pre-trained neural embeddings). It has been extensively studied due to its benefits in dimension reduction and mathematical tractability in both the online and offline settings (Du et al., 2019; Duan and Wang, 2020; He et al., 2023; Hu et al., 2023; Jin et al., 2019; Li et al., 2021; Min et al., 2021; Qiao and Wang, 2022; Xiong et al., 2023; Yang and Wang, 2019; Yin et al., 2022; Zanette et al., 2021). Despite these efforts, hybrid RL algorithms for linear MDPs (Amortila et al., 2024; Nakamoto et al., 2023; Song et al., 2023; Tan and Xu, 2024; Wagenmaker and Pacchiano, 2023) have suboptimal worst-case guarantees (Table 2), which raises the question:

Is it possible to develop sample efficient RL algorithms in the setting of hybrid RL that are provably better than online-only and offline-only algorithms for linear MDPs?

1.1 Hybrid RL: two approaches

To answer the question above, we introduce two types of approaches widely-adopted in hybrid RL.

The offline-to-online approach: Most of the current literature (e.g. Amortila et al. (2024); Nakamoto et al. (2023); Song et al. (2023); Tan and Xu (2024)) initializes the online dataset with offline samples to perform regret-minimizing online RL. We refer to this as the offline-to-online approach. This method is simple and natural, and as the algorithm optimizes the reward during each online episode, it is suitable when the agent has to perform well during online exploration.

The online-to-offline approach: However, if our goal is to output a near-optimal policy, especially in real-world situations in medicine and defense, randomizing between policies can be suboptimal or even unethical. Recently, Wagenmaker and Pacchiano (2023) and Li et al. (2023b) propose using reward-agnostic online exploration to explore parts of the state space unseen by the behavior policy, to construct a dataset that is especially amenable to leverage the sharp performance guarantees of offline RL. We refer to this as the online-to-offline approach . While this approach does not optimize the 'true reward' during online exploration, it avoids the need to deploy mixed policies to achieve a PAC bound, allowing for the deployment of fixed, and thus more interpretable, policies.

1.2 Our contributions

We propose an online-to-offline method called Reward-Agnostic Pessimistic PAC Exploration-initialized Learning (RAPPEL) in Algorithm 1. It employs reward-agnostic online exploration to enhance the offline dataset, then learns a policy through a pessimistic offline RL algorithm. Algorithm 1 significantly improves upon the sample complexity of the only dedicated hybrid RL algorithm for linear MDPs (Wagenmaker and Pacchiano, 2023) by a factor of at least H 3 . This performs no worse than the offline-only minimaxoptimal error bound from Xiong et al. (2023), with the potential of significant gains from online data. This is the first work to explore the online-to-offline approach in linear MDPs.

In addition, we propose an offline-to-online method called Hybrid Regression for UpperConfidence Reinforcement Learning (HYRULE) in Algorithm 2, where one warm-starts an

online RL algorithm with parameters estimated from offline data. In addition to improving the ambient dimension dependence, this algorithm enjoys a regret (or sample complexity) bound that is no worse than the online-only minimax optimal bound, with the potential of significant gains if the offline dataset is of high quality (Agarwal et al., 2022; He et al., 2023; Hu et al., 2023; Zhou et al., 2021). Our result demonstrates the provable benefits of hybrid RL in scenarios where offline samples are much cheaper or much easier to acquire.

To the best of our knowledge, we are the first to show improvements over the aforementioned lower bounds of hybrid RL algorithms (in the same vein as Li et al. (2023b)) in the presence of function approximation, without any explicit requirements on the quality of the behavior policy, and with both the offline-to-online and online-to-offline approaches. Our results are also, at the point of writing, the best bounds available in the literature for hybrid RL in linear MDPs (see Table 2).

Technical contributions. In this work, we build on recent advancements in offline and online RL, demonstrating that intuitive modifications suffice to achieve state-of-the-art sample complexity for hybrid RL in linear MDPs. At a high level, our sample efficiency gains are achieved by decomposing the error of interest into offline and online partitions, and optimizing them respectively, following the same idea in Tan and Xu (2024). Below, we summarize our specific technical contributions.

We sharpen the dimensional dependence from d to d on and c off p X off q via projections onto those partitions. The former is accomplished in Algorithm 1 by Kiefer-Wolfowitz in Lemma 1, and in Algorithm 2 by proving a sharper variant of Lemma B.1 from Zhou and Gu (2022) in Lemma 18, using this in Lemma 14 to reduce the dimensional dependence in the summation of bonuses, which helps achieve the desired result.

We maintain a H 3 dependence for the error or regret for both algorithms, which is nontrivial, in Algorithm 1 and for the offline partition in Algorithm 2 by combining the total variance lemma with a novel truncation argument for 'bad' trajectories in Lemma 17.

2 Preliminaries

2.1 Basics of Markov decision processes

An episodic MDP is a tuple M ' ` S , A , H, p P h q H h ' 1 , p r h q H h ' 1 ˘ , where S is the state space, A the action space, H the horizon, p P h q H h ' 1 the collection of transition probability kernels P h : S ˆ A Ñ ∆ p S q , and p r h q H h ' 1 the collection of reward functions r h : S ˆ A Ñr 0 , 1 s . ∆ p¨q is the collection of distributions over a set. At each h P r H s ' t 1 , ..., H u , an agent observes the current state s h P S , takes an action a h P A according to π h : S Ñ ∆ p A q , and observes the reward r h and next state s h ` 1 ' P h p¨ | s h , a h q . We write Π for the set of policies π ' t π h u H h ' 1 , with value and Q-functions

π ˚ ' t π ˚ u H h ' 1 is the optimal policy attaining the highest value and Q-functions, and we write V ˚ ' t V ˚ h u H h ' 1 and Q ˚ ' t Q ˚ h u H h ' 1 for the optimal value and Q-functions. We consider the setting of hybrid RL, where an agent has access to two sources of data:

N off independent episodes of length H collected by a behavior policy π b where the n -th sample trajectory is a sequence of data p s p n q 1 , a p n q 1 , r p n q 1 , ..., s p n q H , a p n q H , r p n q H , s p n q H ` 1 q ;

N on sequential episodes of online data, where at each episode n ' 1 , ..., N on , the algorithm has knowledge of the N off offline episodes and the previous online episodes 1 , ..., n ´ 1 .

The quality of the behavior policy π b is measured by the all-policy and single-policy concentrability coefficients proposed by Xie et al. (2023); Zhan et al. (2022):

Definition 1 (Occupancy Measure) . For a policy π ' t π h u H h ' 1 , its occupancy measure d π ' t d π h u H h ' 1 corresponds to the collection of distributions over states and actions induced by running π within M , where for some initial distribution ρ and s 1 ' ρ , we have

Definition 2 (Concentrability Coefficient) . The all-policy and single-policy concentrability coefficients of π with regard to the occupancy measure µ ' t µ h u H h ' 1 of a behavior policy π b are

Policy learning and regret minimization. Hybrid RL aims to either learn an ϵ -optimal policy p π such that V ˚ ´ V p π ď ϵ with high probability , or to minimize the regret. Here, the regret of an online algorithm L : H Ñ Π is Reg L p T q : ' E r ř T t ' 1 p V ˚ 1 p s p t q 1 q ´ ř H h ' 1 r p t q h qs . We write T ' N on interchangeably for the number of episodes taken by a regret-minimizing online RL algorithm.

2.2 Linear MDPs

Throughout this paper, we study linear MDPs, where the transition probabilities and rewards are linearly parametrizable as functions of known features. This was first proposed by Jin et al. (2019); Yang and Wang (2019), and further studied in He et al. (2023); Hu et al. (2023); Wagenmaker and Jamieson (2023); Wagenmaker and Pacchiano (2023); Xiong et al. (2023); Zanette et al. (2021).

Assumption 1 (Linear MDP, Jin et al. (2019)) . There exists a known feature map ϕ : S ˆ A Ñ R d , d unknown signed measures µ h ' p µ p 1 q h , ¨ ¨ ¨ , µ p d q h q over S for each h , and an unknown vector θ h P R d , such that for any s, a, h we have P h p¨ | s, a q ' x ϕ p s, a q , µ h p¨qy , r h p s, a q ' x ϕ p s, a q , θ h y . Assume } ϕ p s, a q} ď 1 for all s, a , and max t} µ h p S q} , } θ h }u ď ? d for all h .

This allows for sample-efficient RL for a few reasons. Firstly, linear MDPs are Bellman complete (Jin et al., 2021a), a common assumption for sample-efficient RL in the literature (Duan and Wang, 2020; Fan et al., 2020; Munos and Szepesv´ ari, 2008). Secondly, the value and Q-functions are linearly parametrizable in the features, allowing one to learn them via ridge regression. This allows for sample-efficient online (He et al., 2023; Hu et al., 2023) and offline (Xiong et al., 2023; Yin et al., 2022) RL with function approximation. However, existing guarantees for hybrid RL in linear MDPs (Wagenmaker and Pacchiano, 2023) are loose (Li et al., 2023b), inspiring our work.

Further notation. Write ϕ n,h ' ϕ p s p n q h , a p n q h q for the feature vector at episode n and horizon h . Let Λ h ' ř N n ' 1 ϕ n,h ϕ J n,h ` λ I and Λ off ,h ' ř N off n ' 1 ϕ n,h ϕ J n,h ` λ I be the covariance matrices of the entire dataset and the offline dataset respectively, and Ω the set of all covariates. We consider two kinds of variance-weighted covariance matrices, namely Σ ˚ n,h ' ř N n ' 1 ϕ n,h ϕ J n,h { ' V h V ˚ h ` 1 ‰ p s τ h , a τ h q ` λ I and Σ n,h ' ř N n ' 1 ¯ σ ´ 2 n,h ϕ n,h ϕ J n,h ` λ I , where ' V h V ˚ h ` 1 ‰ p s τ h , a τ h q ' max ␣ 1 , ' Var h V ˚ h ` 1 ‰ p s, a q ( is the truncated variance of the optimal value function (where s, a are random variables) and ¯ σ ´ 2 n,h is the variance estimator from He et al. (2023).

2.3 Exploring the state-action space

We aim to develop efficient hybrid RL algorithms for linear MDPs that do not rely on single-policy concentrability over the entire state-action space, which entails that the behavior policy covers every state-action pair that π ˚ visits. A natural idea from Li et al. (2023b) is to partition this space into a component that is well-covered by the behavior policy, which we call the offline partition X off , and a component requiring further exploration, which we call the online partition X on . Based on this partition, similarly to Tan and Xu (2024), the estimation error or regret of a hybrid RL algorithm can be analyzed on each component separately. We define X on Y X off ' r H s ˆ S ˆ A , with their images under the feature map Φ off ' Span p ϕ p X off ,h qq h Pr H s Ď R d and Φ on ' Span p ϕ p X on ,h qq h Pr H s Ď R d being subspaces of dimension d off and d on respectively. Write P off , P on for the orthogonal projection operators onto these subspaces respectively. Let λ k p M q denote the k -th largest eigenvalue

Algorithm 1 Reward-Agnostic Pessimistic PAC Exploration-initialized Learning (RAPPEL)

1: Input: Offline dataset D off , samples sizes N on , N off , feature maps ϕ h , tolerance parameter for reward-agnostic exploration τ .

3: for horizon h ' 1 , ..., H do

2: Initialize: D p 0 q h ÐH @ h P r H s , λ ' 1 { H 2 , β 2 ' ˜ O p ? d q .

4: Run an exploration algorithm (OPTCOV, Wagenmaker and Jamieson (2023)) to collect covariates Λ h such that max ϕ h P Φ ϕ J h p Λ h ` λ I ` Λ off ,h q ´ 1 ϕ h ď τ.

5: end for

6: Output: p π from running a pessimistic offline RL algorithm (LinPEVI-ADV+, Xiong et al. (2023)) with hyperparameters λ, β 2 on the combined dataset D off Yt D p N on q h u h Pr H s .

of a symmetric matrix M. We borrow the definition of partial offline all-policy concentrability, 1

from Tan and Xu (2024), where we use the convention that c off pHq ' 0 . This corresponds to the inverse of the d off -th largest eigenvalue of the covariance matrix of the projected feature maps. Similarly, the partial all-policy analogue of the coverability coefficient from Xie et al. (2022a) is c on p X on q : ' inf max 1 L λ d on p E d π rp P on ϕ h qp P on ϕ h q J sq . (6)

As we shall see, these quantities characterize the estimation error of our proposed algorithms.

3 Algorithms and main results

We provide two algorithms with improved statistical guarantees to tackle the unsolved (Table 2) problem of achieving sharp guarantees with hybrid RL in linear MDPs, with different approaches:

Performing reward-agnostic online exploration (Wagenmaker and Pacchiano, 2023) to augment the offline data, then invoking offline RL (Xiong et al., 2023) to learn an ϵ -optimal policy on the combined dataset, in the same vein of Li et al. (2023b). This is Algorithm 1.

Warm-starting an online RL algorithm (He et al., 2023) with parameters estimated from an offline dataset to minimize regret, as in Song et al. (2023), with details in Algorithm 2.

3.1 Offline RL after online exploration

Algorithm 1 collects online samples informed by the degree of coverage (or lack thereof) of the offline dataset D off with a reward-agnostic online exploration algorithm called OPTCOV from Wagenmaker and Jamieson (2023). OPTCOV explores so that the smallest eigenvalue of the covariance matrix, λ min p Λ h q , is no smaller than a tolerance parameter 1 { τ . We then learn a policy from the combined dataset using a minimax-optimal pessimistic offline RL algorithm from Xiong et al. (2023), LinPEVI-ADV+. To employ OPTCOV, one requires a modified analogue of the full-rank covariate assumption from Wagenmaker and Pacchiano (2023) that ensures that the MDP is 'explorable' enough. This assumption is only imposed for Algorithm 1.

Informally, this states that for any partition, there exists some 'optimal exploration policy' that ensures that the projected covariates onto the online partition have the same rank as its dimension at every timestep. In practice, this is achievable for any linear MDP via projecting the features onto the eigenspace corresponding to the nonzero singular values. We can then establish the following:

Lemma1 (Partial Coverability Is Bounded In Linear MDPs) . For any partition X off , X on , it satisfies that c on p X on q ď d on . Also, there exists at least one partition such that c off p X off q ' O p d q .

The proof of this lemma is deferred to Appendix D. This result allows us to bound the error on the offline and online partitions by the dimensionality of the partitions, instead of the coverability coefficient. Define α off : ' N off { N , α on : ' N on { N , and the minimal online samples for exploration

We now have, with full proof in Appendix B and sketch at the end of the subsection, the following:

Theorem 1 (Error Bound for RAPPEL, Algorithm 1) . For every δ P p 0 , 1 q and any partition X off , X on , when choosing τ ď ˜ O p max t d on { N on , c off p X off q{ N off uq , RAPPEL achieves w.p. 1 ´ δ :

given N ě max ␣ α 4 on d ´ 4 on , α 4 off c off p X off q ´ 4 ( max t N ˚ p τ q , poly p d, H, c off p X off q , log 1 { δ qu . This result, when applied to tabular MDPs with finite states and actions, yields:

In sum, Theorem 1 shows that with a poly p d, H q burn-in cost that is no smaller than N ˚ (the minimal online samples for any algorithm to achieve our choice of OPTCOV tolerance), we require only

trajectories to learn an O p ϵ q -optimal policy. N ˚ , from Wagenmaker and Pacchiano (2023), is essentially unavoidable in reward-agnostic exploration for linear MDPs. To compare with prior literature, our result leads to a better worst-case guarantee than the error bound of a d 2 H 7 { N attained in Wagenmaker and Pacchiano (2023) (by at least a factor of H 3 { 2 ), the only other work on hybrid RL in linear MDPs thus far. While we employ the same online exploration procedure, we combine our exploration phase with an offline learning algorithm LinPEVI-ADV+ from Xiong et al. (2023) and conduct a careful analysis. When comparing with the offline-only and online-only settings, Theorem 1 improves upon the offline-only minimax-optimal error bound of ? d ř H h ' 1 E π ˚ || ϕ p s h , a h q|| Σ ˚´ 1 off ,h from Xiong et al. (2023) as a consequence of Σ ˚ off ,h ` Σ ˚ on ,h ľ Σ ˚ off ,h ; the best offline-only error bound is a d 2 H 4 { N off obtained under the 'well-covered' assumption (Corollary 4.6, Jin et al. (2021b)) that λ min p Λ h, off q ě Ω p 1 { d q , Theorem 1 enjoys better dimension and horizon dependence as there is always a partition such that d on , c off p X off q ď d and d on H 3 min t d on , H u ď d 2 H 4 .

The literature has experienced considerable difficulty in sharpening the horizon dependence to H 3 in offline RL for linear MDPs. While Yin et al. (2022) and Xiong et al. (2023) provide minimaxoptimal algorithms for offline RL in linear MDPs, both only manage to achieve a H 3 horizon dependence in the special case of tabular MDPs, even under the aforementioned 'well-covered' assumption. We provide the same result in Corollary 1 with proof deferred to Appendix C, but encouragingly, hybrid RL lets us bypass the 'well-covered' assumption. In Appendix B and G, we use a novel truncation argument and the total variance lemma (Lemma C.5 of Jin et al. (2018)) to improve the dependence on H , but our result falls slightly short of a c off p X off q dH 3 { N off ` a d on dH 3 { N on .

Computational efficiency. In terms of computational efficiency, Algorithm 1 inherits the computational costs of the previous proposed algorithms OPTCOV and LinPEVI-ADV+ (Wagenmaker and Jamieson (2023); Xiong et al. (2023). OPTCOV runs in polynomial time poly p d, H, c on p X on q , log 1 { δ q , and LinPEVI-ADV+ runs in ˜ O p d 3 HN | A |q time when the action space is discrete. Algorithm 1 therefore remains computationally efficient in this case.

Requirement of choosing d on . There is the caveat that we require the user to choose the tolerance for OPTCOV. In practice, one can achieve this by performing SVD on the offline dataset and looking at the plot of eigenvalues. One can also choose a tolerance of O p d { min t N off , N on uq , but this would not achieve the reduction in the dependence on dimension from d 2 to c off p X off q d, d on d .

Practical benefits of the online-to-offline approach. Algorithm 1 outputs a fixed policy satisfying a PAC bound. This enables policies to be deployed in critical real-world applications, such as in medicine or defense, where randomized policies from regret minimization are unacceptable.

Algorithm 1 works for reward-agnostic hybrid RL. The use of reward-agnostic online exploration in Algorithm 1 enables one to use the hybrid dataset D to learn policies for different reward functions offline. As the online exploration is not influenced by any single reward function, the collected data satisfies good coverage for any possible reward function even if it is revealed only after exploration, enabling one to use a single dataset to achieve success on many different tasks. This therefore also serves as an algorithm for the related setting of reward-agnostic hybrid RL , where the reward function is unknown during online exploration and only revealed to the agent after it.

Algorithm 2 Hybrid Regression for Upper-Confidence Reinforcement Learning (HYRULE)

1: Input: Offline dataset D off , samples sizes N on , N off , feature maps ϕ h . Regularization parameter λ ą 0 , confidence radii β, ¯ β, ˜ β , t last ' 0 .

2: Initialize: For h P r H s , estimate p w 1 ,h , q w 1 ,h , Q 1 ,h , q Q 1 ,h , σ 1 ,h , ¯ σ 1 ,h from D off , and assign Σ 0 ,h ' Σ 1 ,h ' Σ off ` λ I ' ř N off n ' 1 ¯ σ ´ 2 n,h ϕ n,h ϕ J n,h ` λ I .

3: for episodes t ' 1 , ..., T do

4: Update optimistic and pessimistic weights p w t,h , q w t,h for all h .

5: if there exists a stage h 1 P r H s such that det p Σ t,h 1 q ě 2 det p Σ t last ,h 1 q then

6: Update optimistic and pessimistic Q-functions Q t,h p s, a q , q Q t,h p s, a q , set t last ' t . 7: end if

8: for horizon h ' 1 , ..., H do

9: Play action a p t q h Ð arg max a Q t,h p s p t q h , a q , receive reward r p t q h , next state s p t q h ` 1

10: Estimate σ t,h , ¯ σ t,h Ð max t σ t,h , ? H, 2 d 3 H 2 || ϕ p s p t q h , a p t q h q|| 1 { 2 Σ ´ 1 t,h u 2 , update Σ t ` 1 ,h .

11: end for

12: end for

13: Output: Greedy policy p π ' π Q T,h , Unif p π Q 1 ,h , ..., π Q T,h q for PAC guarantee.

Proof sketch. The relation (7) in Theorem 1 follows from invoking Theorem 2 from Xiong et al. (2023) with N ą Ω p d 2 H 6 q , λ ' 1 { H 2 , β 1 ' O p ? d q . To establish (8), we first bound

As Σ ˚´ 1 h ĺ H 2 Λ ´ 1 h (see Xiong et al. (2023)), it therefore boils down to controlling max ϕ h P Φ ϕ J h Λ ´ 1 h ϕ h . Towards this, first, we make the observation that Lemma 1 suggests that c on p X on q ď d on . If we run OPTCOV with tolerance ˜ O p max t d on { N on , c off p X off q{ N off uq on partitions where the above hold, in Lemma 5, we prove that max ϕ h P Φ ϕ J h Λ ´ 1 h ϕ h À max t c off p X off q{ N off , d on { N on u . This yields the c off p X off q dH 4 , d on dH 4 result.

To tighten the horizon dependence to H 3 , we employ an useful truncation argument. More specifically, from the total variance lemma (Lemma C.5 of Jin et al. (2018)), the average variance V h V ˚ h ` 1 is asymptotically on the order of H . We therefore define the sets of trajectories E h p δ h q ' t τ P D : ' V h V ˚ h ` 1 ‰ p s τ h , a τ h q ě H 1 ` δ h u . The cardinality of each set can be bounded by | E h p δ h q| À NH 1 ´ δ h , and so truncating at the level where NH 1 ´ δ h « min p N off c off p X off q , N on d on q leads to min ϕ h P Φ ϕ J h Σ ‹ h ϕ h Á 1 NH 2 min p N off c off p X off q , N on d on q 2 . Putting things together yields the last c off p X off q 2 dH 3 , d 2 on dH 3 result needed, and the theorem then follows.

3.2 Online regret minimization

Thus far, we described an online-to-offline strategy which collects online samples to augment the offline dataset. However, in certain critical cases, such as with a doctor treating patients, performanceagnostic online exploration is untenable. One may wish to minimize the regret of the online actions taken while learning a policy. We therefore explore another approach inspired by the work of Song et al. (2023); Tan and Xu (2024) - that of warm-starting an online RL algorithm with parameters estimated from an offline dataset. We describe this in Algorithm 2, and show that hybrid RL enables provable gains over minimax-optimal online-only regret bounds in the offline-to-online case as well.

In order to warm-start an online RL algorithm with an offline dataset, we modify LSVI-UCB++ from He et al. (2023) by estimating its parameters from D off with the same formulas it would use as if it had experienced the N off offline episodes itself. As Tan and Xu (2024) suggest, this can be understood as including the offline episodes in the 'experience replay buffer' that the algorithm uses to learn parameters. The full version can be found in Appendix E as Algorithm 4. Doing so allows us prove a regret bound depending on the partial all-policy concentrability coefficient. Below we state our theoretical guarantees for this algorithm. The proof of this result is deferred to Appendix E, and a brief proof sketch is provided at the end of this subsection.

Theorem 2 (Regret Bound for HYRULE, Algorithm 2) . Given any δ P p 0 , 1 q , for every partition X off , X on , if N on , N off ' ˜ Ω p d 13 H 14 q , the regret of HYRULE is bounded w.p. at least 1 ´ δ by

Corollary 2. By the regret-to-PAC conversion, Algorithm 2 achieves a sub-optimality gap w.p. 1 ´ δ :

To understand this result, we first note that bounding the regret over all possible partitions yields an improvement over the ? d 2 H 3 N on regret bound obtained by He et al. (2023), as we can take X on ' X , X off ' H to recover the ? d 2 H 3 N on bound. In the scenario where offline samples are abundant (where N off " N on ), it is possible to achieve significant improvements over online-only learning. Furthermore, in view of Lemma 1, there always exists a partition such that c off p X off q , d on ď d . This result therefore yields provable improvements over the minimax-optimal online regret bound in linear MDPs (Agarwal et al., 2022; He et al., 2023; Hu et al., 2023; Zhou et al., 2021).

Additionally, Theorem 2 shows that Algorithm 2 attains the best known regret bound in hybrid RL for linear MDPs, as we illustrate in Table 2. The current best known result is that of Tan and Xu (2024), with a dependence of a c off p X off q dH 5 N 2 on { N off ` ? d on dH 5 N on . Notably, we achieve the same a reduction in the dimension dependence on the online partition from d 2 to d on d that Tan and Xu (2024) do by proving a sharper variant of Lemma B.1 from Zhou and Gu (2022) in Lemma 18, using this in Lemma 14 to reduce the dimensional dependence in the summation of bonuses. Song et al. (2023) and Amortila et al. (2024), on the other hand, have bounds on the order of C ˚ ? d 2 H 6 N on and a p C ˚ ` c on p X qq d 3 H 6 N on respectively. We produce a better bound than Amortila et al. (2024); Song et al. (2023); Tan and Xu (2024) by at least a factor of H 2 by combining the total variance lemma and a novel truncation argument that rules out 'bad' trajectories in Lemma 17, which allows us to maintain a desirable H 3 dependence on both partitions.

Computational efficiency. When the action space is finite and of cardinality | A | , the computational complexity of Algorithm 2 is of order r O ` d 4 H 3 N | A | ˘ , as outlined in He et al. (2023). Algorithm 2 is therefore computationally efficient and runs in polynomial time in this case. When the action space is continuous, one may need to solve an optimization problem over the continuous action space, making the computational complexity highly problem-dependent.

Algorithm 2 is unaware of the partition. Unlike Algorithm 1, Algorithm 2 is fully unaware of the choice of partition, and there is therefore no need to estimate d on or any relevant analogue to the choice of tolerance for OPTCOV. The regret bound therefore automatically adapts to the best possible partition, even though Algorithm 2 is unaware of it.

Practical benefits of the offline-to-online approach. While Algorithm 2 only satisfies a PAC bound with a randomized policy, it minimizes the regret of the actions it takes. This enables the algorithm to be deployed in situations where its performance during online exploration is of critical importance, e.g. in applications like mobile health (Nahum-Shani et al., 2017).

Technical challenges. Although Algorithm 2 is a straightforward generalization of LSVI-UCB++ in He et al. (2023), with Σ 0 initialized with the offline dataset, we had to decompose the regret into the regret on the offline and online partitions to achieve the regret guarantee in Theorem 2. In the process, we faced the following challenges:

Bounding the regret on the offline partition was challenging, as the argument of He et al. (2023) was not applicable. Instead, we used a truncation argument in Lemma 17 to bound the maximum eigenvalue of Σ ´ 1 off ,h , maintaining a H 3 dependence on the offline partition.

Bounding the regret on the online partition allowed us to use an analysis that was close to that of He et al. (2023). However, directly following their argument would have left us with a d 2 H 3 dependence. To reduce the dimensional dependence to d on d , we prove a sharper variant of Lemma B.1 from Zhou and Gu (2022) in Lemma 18. We use this in Lemma 14 to reduce the dimensional dependence in the sum of bonuses, achieving the desired result.

Without the above two techniques, one could have used a simpler analysis to achieve a far looser a c off p X off q 2 d 6 H 8 N 2 on { N off ` ? d 2 H 3 regret bound by using the maximum magnitude of the variance weights for the offline partition and the analysis from He et al. (2023) verbatim for the online partition, but this would not have yielded the same improvement.

We accordingly provide a proof sketch below.

✶ ✶ It then boils down to controlling the second and third terms separately. We prove in Lemma 12 that the sum of bonuses on the offline partition can be bounded by ř h b dN on N on N off max ϕ h P Φ off ϕ J h ¯ Σ ´ 1 off ,h ϕ h . Further, ř h max ϕ h P Φ off b ϕ J h ¯ Σ ´ 1 off ,h ϕ h À c off p X off q 2 H 3 by Lemma 13. Putting things together, the second term can be controlled as

✶ With respect to the third term, Lemma 14 (a sharpened version of Lemma E.1 in He et al. (2023)), combined with the Cauchy-Schwartz inequality, yields

✶ By the total variance lemma (Appendix B, He et al. (2023)), ř h,t σ 2 t,h ď r O ` H 2 T ` d 10 . 5 H 16 ˘ . Taking everything collectively establishes the desired result.

4 Numerical experiments

To demonstrate the benefits of hybrid RL in the offline-to-online and online-to-offline settings, we implement Algorithms 1 and 2 on a scaled-down Tetris environment (as in Tan and Xu (2024)). For the purposes of brevity, we defer the details of the environment to Appendix H. 3

Figure 1 depicts the coverage (defined by 1 { λ min p Λ q , 1 { λ d off p Λ off q , 1 { λ d on p Λ on q ) achieved by the reward-agnostic exploration algorithm, OPTCOV, when initialized with 200 trajectories from (1) a uniform behavioral policy, (2) an adversarial behavior policy obtained by the negative of the weights of a fully-trained agent under Algorithm 1, and (3) no offline trajectories at all. Although hybrid RL with the uniform behavior policy achieves the best coverage throughout as expected, hybrid RL with even adversarially collected offline data achieves better coverage than online-only exploration. This demonstrates the potential of hybrid RL as a tool for taking advantage of poor quality offline data.

Figure 2 shows the benefits of hybrid RL in the online-to-offline setting when the behavior policy is of poor quality. When applying LinPEVI-ADV to the hybrid dataset of 200 trajectories and 100 online trajectories, 300 trajectories of adversarially collected offline data, and 300 trajectories of online data under reward-agnostic exploration, we see that the hybrid dataset is most conducive for learning. Additionally, without a warm-start from offline data, online-only reward-agnostic exploration performs worse than the adversarially collected offline data due to significant burn-in costs. Hybrid RL, in this instance, performs better than both offline-only and online-only learning alone. Figure 3 compares the performances of LSVI-UCB++ and Algorithm 2. Initializing a regret-minimizing online algorithm (LSVI-UCB++, (He et al., 2023)) with an offline dataset as in Algorithm 2 yields lower regret than LSVI-UCB++ without an offline dataset. This shows that even a nearly minimaxoptimal online learning algorithm can stand to benefit from offline data.

Coverage Over Feature Space,Reward-AgnosticExploration

Entire Space

Offline Partition

Online Partition

1015

1013-.

1018-.

Coefficient/Episodes

m

1011-.

1015 -.

1012

109 -.

1012 -

109

Hybrid (Uniform)

107-

Hybrid (Uniform)

109 -

Concentrability

106-

Hybrid(Adversarial)

Online-Only

105.

Hybrid (Adversarial)

Online-Only

106 -

103-.

103

103 -

101 -

Hybrid (Uniform)

100

10-1_

100-

Hybrid (Adversarial)

Online-Only

0

25

50

75

100

0

1

25

50

75

100

0

25

50

75

100

Online Episodes

Online Episodes

Online Episodes

Hybrid Exploration

Offline-OnlyLearning

Online-OnlyExploration

-65

-60

-55

-50

-45

-40

-35

-30

-25

Value

CumulativeRegretOverOnlineTimesteps

60

Hybrid

Online Only

CumulativeRegret

50

40

30

20

10

20

40

60

80

100

Online Episodes

AveragePer-EpisodeRewardOverOnlineTimesteps

0.4

Online Only

0.6

Average Reward

0.8

1.0

1.2

-1.4

1.6

20

40

60

80

100

Online Episodes

5 Discussion, limitations and future work

In this paper, we develop two hybrid RL algorithms for linear MDPs with desirable statistical guarantees for the online-to-offline and offline-to-online settings. Both algorithms demonstrate provable gains over the minimax-optimal rates in offline or online-only reinforcement learning, and provide the sharpest worst-case bounds for the performance of hybrid RL in linear MDPs thus far.

Throughout this paper, we have used both optimism and pessimism in our algorithm design. Other work in hybrid RL (Amortila et al., 2024; Li et al., 2023b; Nakamoto et al., 2023; Song et al., 2023; Tan and Xu, 2024; Wagenmaker and Pacchiano, 2023) uses optimism, pessimism, or sometimes even neither. We conjecture that optimism is still helpful in aiding online exploration within hybrid RL and that pessimism helps in hybrid RL when learning from a combined dataset. However, determining if or when optimism or pessimism is beneficial in hybrid RL remains an open question.

Achieving a H 3 horizon dependence in offline RL for linear MDPs has proven challenging. Even under strong coverage assumptions, Yin et al. (2022) and Xiong et al. (2023) only manage to achieve a H 3 horizon dependence for tabular MDPs. Obtaining a a d 2 H 3 { N bound is an open problem.

Aresult depending on a partial single-policy concentrability coefficient would be desirable, but may provide only limited benefits as we take the infimum over partitions. A good offline partition for the partial all-policy concentrability contains the portion of the state-action space well-covered by the offline dataset, while the same for the partial single-policy concentrability would be well-covered by both the offline dataset and the optimal policy. The smaller size of the latter offline partition may be offset by the larger size of the latter's online partition, and as such any gains may be limited.

Furthermore, while Algorithm 1 improves upon the offline-only error lower bound in Xiong et al. (2023) and Algorithm 2 improves upon the online-only regret lower bound in Zhou et al. (2021), we still desire a single algorithm that improves upon both the best possible offline-only and online-only rates at once. Additionally, the burn-in costs for Algorithms 1 and 2 are nontrivial. The former is inherited from OPTCOV (Wagenmaker and Jamieson, 2023), while the latter is inherited from He et al. (2023) and the truncation argument. Improving the former by devising new reward-agnostic exploration algorithms for linear MDPs, perhaps in the vein of Li et al. (2023a), would be welcome.

While we tackle the setting of linear MDPs, it remains a first step towards showing that hybrid RL breaks minimax-optimal barriers in the presence of function approximation. Further work in this vein on other types of function approximation would be an interesting contribution to the literature.

References

Abbasi-yadkori, Y., P´ al, D., and Szepesv´ ari, C. (2011). Improved algorithms for linear stochastic bandits. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K., editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc.

Agarwal, A., Jin, Y., and Zhang, T. (2022). Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation.

Amortila, P., Foster, D. J., Jiang, N., Sekhari, A., and Xie, T. (2024). Harnessing density ratios for online reinforcement learning.

Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In International conference on machine learning , pages 263-272. PMLR.

Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. (2023). Efficient online reinforcement learning with offline data. In International Conference on Machine Learning , pages 1577-1594. PMLR.

Du, S. S., Kakade, S. M., Wang, R., and Yang, L. F. (2019). Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint arXiv:1910.03016 .

Duan, Y. and Wang, M. (2020). Minimax-optimal off-policy evaluation with linear function approximation.

Fan, J., Wang, Z., Xie, Y., and Yang, Z. (2020). A theoretical analysis of deep q-learning.

He, J., Zhao, H., Zhou, D., and Gu, Q. (2023). Nearly minimax optimal reinforcement learning for linear markov decision processes.

Hu, P., Chen, Y., and Huang, L. (2023). Nearly minimax optimal reinforcement learning with linear function approximation.

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is q-learning provably efficient? Advances in neural information processing systems , 31.

Jin, C., Liu, Q., and Miryoosefi, S. (2021a). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2019). Provably efficient reinforcement learning with linear function approximation.

Jin, Y., Yang, Z., and Wang, Z. (2021b). Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR.

Kausik, C., Tan, K., and Tewari, A. (2024). Leveraging offline data in linear latent bandits. arXiv preprint arXiv:2405.17324 .

Lange, S., Gabel, T., and Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement learning: State-of-the-art , pages 45-73. Springer.

Lattimore, T., Szepesvari, C., and Weisz, G. (2020). Learning with good feature representations in bandits and in rl with a generative model.

Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems.

Li, G., Chen, Y., Chi, Y., Gu, Y., and Wei, Y. (2021). Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. Advances in Neural Information Processing Systems , 34:16671-16685.

Li, G., Shi, L., Chen, Y ., Chi, Y ., and Wei, Y . (2024). Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics , 52(1):233-260.

Li, G., Yan, Y., Chen, Y., and Fan, J. (2023a). Minimax-optimal reward-agnostic exploration in reinforcement learning.

Li, G., Zhan, W., Lee, J. D., Chi, Y., and Chen, Y. (2023b). Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. arXiv preprint arXiv:2305.10282 .

Min, Y., Wang, T., Zhou, D., and Gu, Q. (2021). Variance-aware off-policy evaluation with linear function approximation. Advances in neural information processing systems , 34:7598-7610.

Munos, R. and Szepesv´ ari, C. (2008). Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(27):815-857.

Nahum-Shani, I., Smith, S. N., Spring, B. J., Collins, L. M., Witkiewitz, K. A., Tewari, A., and Murphy, S. A. (2017). Just-in-time adaptive interventions (jitais) in mobile health: Key components and design principles for ongoing health behavior support. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine , 52:446 - 462.

Nair, A., Gupta, A., Dalal, M., and Levine, S. (2020). Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 .

Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S. (2023). Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.

Qiao, D. and Wang, Y.-X. (2022). Near-optimal deployment efficiency in reward-free reinforcement learning with linear function approximation. arXiv preprint arXiv:2210.00701 .

Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. (2023). Hybrid rl: Using both offline and online data can make rl efficient.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction . The MIT Press, second edition.

Tan, K. and Xu, Z. (2024). A natural extension to online algorithms for hybrid RL with limited coverage. Reinforcement Learning Journal , 1.

Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Roth¨ orl, T., Lampe, T., and Riedmiller, M. (2017). Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 .

Wagenmaker, A., Chen, Y., Simchowitz, M., Du, S. S., and Jamieson, K. (2022). First-order regret in reinforcement learning with linear function approximation: A robust estimation approach.

Wagenmaker, A. and Jamieson, K. (2023). Instance-dependent near-optimal policy identification in linear mdps via online experiment design.

Wagenmaker, A. and Pacchiano, A. (2023). Leveraging offline data in online reinforcement learning.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2023). Bellman-consistent pessimism for offline reinforcement learning.

Xie, T., Foster, D. J., Bai, Y ., Jiang, N., and Kakade, S. M. (2022a). The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157 .

Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2022b). Policy finetuning: Bridging sampleefficient offline and online reinforcement learning.

Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. (2023). Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game.

Yang, L. F. and Wang, M. (2019). Sample-optimal parametric q-learning using linearly additive features.

Yin, M., Duan, Y., Wang, M., and Wang, Y.-X. (2022). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism.

Zanette, A., Wainwright, M. J., and Brunskill, E. (2021). Provable benefits of actor-critic methods for offline reinforcement learning.

Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. D. (2022). Offline reinforcement learning with realizability and single-policy concentrability.

Zhang, Z., Chen, Y., Lee, J. D., and Du, S. S. (2023). Settling the sample complexity of online reinforcement learning. arXiv preprint arXiv:2307.13586 .

Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps.

Zhou, D., Gu, Q., and Szepesvari, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes.

Zhou, Y., Sekhari, A., Song, Y., and Sun, W. (2023). Offline data enhanced on-policy policy gradient with provable guarantees.

A Unabridged versions of our algorithms

Algorithm 3 Reward-Agnostic Exploration-initialized Pessimistic PAC Learning (RAPPEL, Full)

1: Input: Offline dataset D off , samples sizes N on , N off , feature maps ϕ h , , tolerance parameter for reward-agnostic exploration τ .

Exploration Phase: Run an exploration algorithm (OPTCOV, Wagenmaker and Jamieson (2023)) to collect covariates Λ h such that max ϕ h P Φ ϕ J h p Λ h ` λ I ` Λ off ,h q ´ 1 ϕ h ď τ.

4: Set the number of iterates T i Ð 2 i , episodes per iterate K i Ð 2 i .

5: Play any policy for K i episodes to collect covariates Γ 0 and data D 0 .

6: Initialize covariance matrix Λ 1 Ð Γ 0 { K .

7: for t ' 1 , ..., T i do

8:

9:

10: end if

11: Run FORCE (Wagenmaker et al., 2022) or another regret-minimizing algorithm on the exploration-focused synthetic reward g p t q h p s, a q9 tr p´ ∇ Λ f i p Λ q| Λ ' Λ t ϕ p s,a q ϕ p s,a q J q .

12:

Collect covariates

Γ

, data

14: end for

D

.

17: if f i p x Λ i q ď K i T i τ then

18: break

19: end if

20: end for

Planning Phase: Estimate p π using a pessimistic offline RL algorithm (LinPEVI-ADV+, Xiong et al. (2023)) with hyperparameters λ, β 2 on the combined dataset D off Yt D p N on q h u h Pr H s . 21: Split the dataset D off Yt D p N on q h u h Pr H s into D and D 1 . Estimate, on D 1 ,

22: for h ' 1 , ..., H do

23: Compute covariance matrix Σ h ' ř τ P D ϕ p s τ h , a τ h q ϕ p s τ h , a τ h q J { p σ 2 h p s τ h , a τ h q ` λ I d .

25: Compute pessimistic penalty Γ h p¨ , ¨q Ð β 2 } ϕ p¨ , ¨q} Σ ´ 1 h .

28: end for

29: Output: p π .

Algorithm 4 Hybrid Regression for Upper-Confidence Reinforcement Learning (HYRULE, Full)

1: Input: Offline dataset D off , samples sizes N on , N off , feature maps ϕ h . Regularization parameter λ ą 0 , confidence radii β, ¯ β, ˜ β , t last ' 0 .

2: Initialize: For h P r H s , estimate p w 1 ,h , q w 1 ,h , Q 1 ,h , q Q 1 ,h , σ 1 ,h , ¯ σ 1 ,h from D off with the same formulas outlined below, and assign Σ 0 ,h ' Σ 1 ,h ' Σ off ` λ I ' ř N off n ' 1 ¯ σ ´ 2 n,h ϕ n,h ϕ J n,h ` λ I .

3: for episodes t ' 1 , ..., T do

4: Receive the initial state s p t q 1 .

5: for horizon h ' 1 , ..., H do

8: if there exists a stage h 1 P r H s such that det p Σ t,h 1 q ě 2 det p Σ t last ,h 1 q then

11:

12:

13:

14:

15:

16:

17:

18:

Set the last updating episode

t

last

'

t

.

end for for

horizon

h

'

Play action

1

a

, ..., H

p

t

do

q

h

p

t

h

Ð

a

Q

t,h

p

s

, a

q

.

22: Receive reward r p t q h , next state s p t q h ` 1 .

23: end for

24: end for

25: Output: Greedy policy p π ' π Q T,h , Unif p π Q 1 ,h , ..., π Q T,h q for PAC guarantee.

q

B Proofs for Theorem 1

The proof of Theorem 1 follows from a series of distinct results, presented as three lemmas below. The first lemma demonstrates that RAPPEL achieves no higher error than LinPEVI-ADV+ itself, the second produces a d on dH 4 error bound, while the third produces a d 2 on dH 3 error bound via a slightly different truncation argument. We will prove Equation 7 in Lemma 2, which act as a general statistical guarantee for RAPPEL. We show the validity of the instance-dependent bound developed from Equation 7 in Lemmas 3 and 4. We observe that Theorem 1 follows immediately after.

Lemma 2 (General Statistical Guarantee for RAPPEL, Algorithm 1) . For every δ P p 0 , 1 q and any partition X off , X on , with probability at least 1 ´ δ , RAPPEL achieves

Proof. Before we proof the desired result, we first recall that

Then, by invoking Theorem 2 from Xiong et al. (2023) with N ą Ω p d 2 H 6 q , λ ' 1 { H 2 , β 1 ' O p ? d q , we see that

as Σ h ' Σ ˚ off ,h ` Σ ˚ on ,h . Noting that Σ ˚ on ,h is positive semi-definite, it then follows Σ ˚ off ,h ĺ Σ ˚ off ,h ` Σ ˚ on ,h . Therefore,

and the inequality holds.

Lemma 3 (First Error Bound for RAPPEL, Algorithm 1) . For every δ P p 0 , 1 q and any partition X off , X on , with probability at least 1 ´ δ , RAPPEL achieves

N ě max ␣ α 4 on d ´ 4 on , α 4 off c off p X off q ´ 4 ( max t N ˚ , poly p d, H, c off p X off q , log 1 { δ qu , where we define the quantities α off ' N off N , α on ' N on N , and the minimal samples for coverage is

Proof. Let X off , X on be an arbitrary partition of S ˆ A ˆr H s . Let us leave the choice of OPTCOV tolerance unspecified for the moment, and simply assume for now that we have data D collected under the success event of Lemma 16.

We now invoke Theorem 2 from Xiong et al. (2023) on this dataset. As we choose N ą Ω p d 2 H 6 q , λ ' 1 { H 2 and β 1 ' O p ? d q , we obtain the suboptimality gap decomposition below:

This decomposition can be further decomposed into the sum of bonuses on the offline and online partitions X off and X on , respectively:

We can further upper bound the above expectations under the optimal policy π ˚ by taking the maximum of the quadratic form over each partition, yielding

From Xiong et al. (2023), as ' V h V ˚ h ` 1 ‰ p¨ , ¨q P ' 1 , H 2 ‰ , the weighted covariance matrix is uniformly upper bounded by the unweighted covariance matrix in the following manner:

which leads to our conclusion that

We now further bound the above two quadratic forms over the online and offline partitions respectively. By Lemma 1, the partial online coverage coefficient is bounded by the dimensionality of the online partition:

As we have N on online episodes, the optimal covariates for online exploration would then yield

Conversely, we also have access to N off episodes of offline data with the following guarantee that follows from an application of Matrix Chernoff:

Therefore, by Lemma 5, we can conclude that on its success event, running OPTCOV with tolerance ˜ O p max t d on { N on , c off p X off q{ N off uq , provides us covariates such that

yielding the desired result.

It now remains to work out the burn-in cost from running OPTCOV. The following quantity of the minimal online samples any algorithm requires to establish coverage was first proposed in Wagenmaker and Pacchiano (2023):

We can use this as follows. Invoking Lemma 16, we see that OPTCOV incurs

episodes of online exploration, for an overall burn-in cost of

episodes, where α off ' N off N off ` N on and α on ' N on N off ` N on .

Note that the more even the proportion of offline to online samples, the smaller α off , α on are. In fact, as α 4 off , α 4 on P r 0 . 0625 , 1 s , this term contributes no more than a constant factor that is no greater than 1 to the final sample complexity.

We then have that

This incurs

episodes of online exploration, for an overall burn-in cost of

with probability at least 1 ´ δ , when N ě max ! α 4 on d 4 on , α 4 off c off p X off q 4 ) max t N ˚ , poly p d, H, c off p X off q , log 1 { δ qu .

Lemma 4 (Second Error Bound for RAPPEL, Algorithm 1) . For every δ P p 0 , 1 q and any partition X off , X on , with probability at least 1 ´ δ , RAPPEL achieves

N ě max ␣ α 4 on d ´ 4 on , α 4 off c off p X off q ´ 4 ( max t N ˚ , poly p d, H, c off p X off q , log 1 { δ qu , we define the quantities α off ' N off N , α on ' N on N , and the minimal samples for coverage is

Proof. First, we set up some preliminaries. Following the same argument as the proof of Lemma 3, we can establish that, for arbitrary partition X ' X on Y X off , we have

and running OPTCOV with tolerance ˜ O p max t d on { N on , c off p X off q{ N off uq , yields:

episodes.

To tighten the horizon dependence even further from the result of Lemma 3, we turn to the total variance lemma. i.e. Lemma C.5 in Jin et al. (2018), indicating that

Then, we directly apply Lemma 17 with γ ' max t d on { N on , c off p X off q{ N off u and ¯ σ ' H ` H 2 { N , we will then obtain that

which leads to our final result:

C Proof of Corollary 1

Proof. In tabular case, we set ϕ p s, a q ' 1 s,a and d ' | S | ¨ | A | . Let N h p s, a q be the number of visits to a specific state-action pair p s, a, h q . As the exploration algorithm OPTCOV ensures that

we bound the error in the following way follows from Lemma 2,

where the last inequality follows from the fact that Σ ‹ h ' diag ` N h p s, a q{ ' V h V ˚ h ` 1 ‰ p s, a q ˘ s P S ,a P A . Wewill then decompose the state-action space into X off and X on , and bound the two parts seperately based on the tolerance level of OPTCOV,

As the optimal policy π ‹ executes a deterministic action π ‹ p s q for any state s , the inequality can be further bounded as

where the last inequality follows from the proof of Lemma C.5. in Jin et al. (2018).

D On concentrability and coverability

Lemma 1. For any partition X off , X on , we have that c on p X on q ď d on . Similarly, there exists a partition such that c off p X off q ' O p d q .

Proof. This proof follows a similar strategy to that of Lemma B.10 in Wagenmaker and Jamieson (2023), except that we exploit the projections onto d on to get a bound that depends on d on ď d , instead of d . We wish to bound

P on P R d ˆ d has rank d on ď d , so we can decompose this with the thin SVD into P on ' U on U J on , where U on P R d ˆ d on . It then holds that

and from Lemma 20 that

Apply Jensen's inequality to find that for any v h P Φ on ,

Then, we can bound

By Kiefer-Wolfowitz (Lattimore et al., 2020), this is bounded by d on .

Similarly,

where the upper bound is achieved when, for instance, we choose X off such that Φ off ' Span ` p v h, 1 , ..., v h,k h q h Pr H s ˘ , where v h,i is the i -th largest eigenvector of E µ r ϕ h ϕ J h s « 1 N off ř τ P D off ϕ h p s τ h , a τ h q ϕ h p s τ h , a τ h q J , and v h,k h is the eigenvector corresponding to the largest eigenvalue λ h,k h ě Ω p 1 { k h q . The largest eigenvalue λ h, 1 is always Ω p 1 { d q for non-null features, so there always exists such a partition where d off is at least 1.

Informally, one can choose the offline partition to be the span of the large eigenvectors of the covariance matrix, so the smallest eigenvalue of the projected covariance matrix, i.e. the partial all policy concentrability coefficient, is no larger than the dimension of the partition.

Lemma 5 (Maximum Eigenvalue Bound with OPTCOV) . On any partition X off , X on , if we run OPTCOV with tolerance ˜ O p max t d on { N on , c off p X off q{ N off uq , on this partition we also have that

Proof. By Lemma 1, for any partition, we have that

Applying Matrix Chernoff, we have that with probability at least 1 ´ δ ,

and similarly for c on p X on q we also have that

As Λ h, off ` Λ h, on ' Λ h , we have

where the last step follows from the choice of partition. So it suffices to run OPTCOV with tolerance ˜ O p max t d on { N on , c off p X off q{ N off uq , to find that there exists at least one partition such that

Lemma6 (Coverability Coefficient Is Bounded In Tabular MDPs) . If the underlying MDP is tabular, for any partition X off , X on , we have that c on p X on q ď d on .

Proof. First, we write the concentrability coefficient in terms of densities.

✶ By the same trick that Xie et al. (2022a) use in their Lemma 3,

E Proofs for Algorithm 2

E.1 Setup

Weconsider the same state-action space splitting framework of Tan and Xu (2024). Let X on Y X off ' r H s ˆ S ˆ A . Then, their images under the feature map Φ off ' Span p ϕ p X off ,h qq h Pr H s Ď R d and Φ on ' Span p ϕ p X on ,h qq h Pr H s Ď R d are subspaces of X with dimension d off and d on , respectively. We denote P off , P on as the orthogonal projection operators onto these subspaces respectively. The partial offline all-policy concentrability coefficient

is bounded by the inverse of the d off -th largest eigenvalue of the covariance matrix of the projected feature maps onto the offline partition, where λ k is the k -th largest eigenvalue. Write ✶ X on as shorthand for pp s, a, h q P X on q , and similarly for X off .

✶ ✶ Now, we work through the analysis of He et al. (2023) to ensure that their result holds in our setting, where the regret decomposes into online part } Σ ´ 1 { 2 t,h ϕ h p s p t q h , a p t q h q ✶ X on } 2 and offline part } Σ ´ 1 { 2 t,h ϕ h p s p t q h , a p t q h q ✶ X off } 2 respectively, instead of } Σ ´ 1 { 2 t,h ϕ h p s p t q h , a p t q h q} 2 . E.2 High-probability events

We define several 'high probability' events which are similar to those defined in He et al. (2023).

We define r w t,h as the solution of the weighted ridge regression problem for the squared value function

We define E as the event where the following inequalities hold for all s, a, t, h P S ˆ A ˆ r T s ˆ r H s :

b

This is the 'coarse event' as mentioned in their paper, where concentration holds for the value and squared value function with all three estimators.

We define r E h as the event that for all episodes t P r T s , stages h ď h 1 ď H and state-action pairs p s, a q P S ˆ A , the weight vector p w t,h satisfies

where

Furthermore, let r E ' r E 1 denotes the event that (19) holds for all stages h P r H s . This is the fine event where concentration for ˆ w is tighter than that required in (16) to (18).

Equipped with these definitions, we recall the following lemmas from He et al. (2023):

Lemma 7 (Lemma B.1, He et al. (2023)) . E holds with probability at least 1 ´ 7 δ .

Lemma 8 (Lemma B.2, He et al. (2023)) . On the event E and r E h ` 1 , for each episode t P r T s and stage h , the estimated variance satisfies

Lemma 9 (Lemma B.3, He et al. (2023)) . On the event E and r E h ` 1 , for any episode t and i ą t , we have

Lemma 10 (Lemma B.4, He et al. (2023)) . On the event E and r E h , for all episodes t P r T s and stages h ď h 1 ď H , we have Q t,h p s, a q ě Q ˚ h p s, a q ě q Q t,h p s, a q . In addition, we have V t,h p s q ě V ˚ h p s q ě q V t,h p s q .

Lemma 11 (Lemma B.5, He et al. (2023)) . On event E , event r E holds with probability at least 1 ´ δ .

E.3 Regret decomposition

From He et al. (2023), based on Lemma B.4 of their paper, Q t,h p s p t q h , a p t q h q ' V t,h p s p t q h q ě V ˚ h p s p t q h q , i.e. optimism holds for all episodes and timesteps. Therefore,

Accordingly, given a partition X off , X on of r H s ˆ S ˆ A , we can further decompose this into the fraction of episodes where each partition is visited,

He et al. (2023) define the events

which they show that by Azuma-Hoeffding, both hold with probability 1 ´ δ each. As such, we have that

Reg off p T q ' ÿ h,t β › › Σ ´ 1 { 2 t,h ϕ h p s p t q h , a p t q h q ✶ X off › › 2 , Reg on p T q ' ÿ h,t β › › Σ ´ 1 { 2 t,h ϕ h p s p t q h , a p t q h q ✶ X on › › 2 , as the offline regret and online regret, respectively.

E.4 Offline regret control

Now, we bound the regret on the offline partition. We first perform a similar argument to that in Tan and Xu (2024); Xie et al. (2022a) to show that the sum of bonuses can be controlled by the maximum eigenvalue of the inverse weighted average covariance matrix in Lemma 12. We will then show that the maximum eigenvalue can be nicely bounded in Lemma 13.

Lemma 12 (Sum of Bonuses on Offline Partition) . For any partition X off , X on , we can bound the sum of bonuses on the offline partition with the following:

where ¯ Σ off ,h ' p Σ off ,h ` λ I q{ N off and T ' N on .

Proof. It is sufficient to show the following holds true

d

then the desired inequality directly follows. With a direct calculation, one may observe that

✶ where the last inequality holds as Σ off ,h ĺ Σ t,h . As a result, we are able to bound the desired inequality with the maximum eigenvalue of the inverse weighted matrix,

where ¯ Σ off ,h ' p Σ off ,h ` λ I q{ N off . As β ' ˜ O p ? d q , we obtain the bound we desired:

Lemma 13 (Partial Concentrability Bound) . For any partition X off , X on , we have that

when N on , N off ě ˜ Ω p d 13 H 14 q , where we define ¯ Σ off ,h ' p Σ off ,h ` λ I q{ N off .

Proof. Similar to the definition of ¯ Σ off ,h , we define ¯ Λ off ,h ' p Λ off ,h ` λ I q{ N off in a similar way. Then, one may observe that

where the last line holds by an application of the Matrix Chernoff inequality. Then, we may further bound the quantity with the partial offline all-policy concentrability coefficient,

To tighten the dependence of the regret of the offline partition on H , we again employ a truncation argument that used in Lemma 4. Recall that in Section B of the appendix in He et al. (2023), by the total variance lemma of Jin et al. (2019), it holds that

Again, recall that we have

Here, I 2 roughly correspond to the 'bad' set of trajectories where there exists some timestep h such that ¯ σ 2 n,h ą max t σ 2 n,h , H u , and I 1 to be the 'good' set of trajectories where the monotonic variance estimator is controlled.

We need to bound the cardinality of the latter before employing our truncation argument on the estimated variances. As we note that for all n P I 2 we have that max h Pr H s b ϕ J n,h Σ ´ 1 n,h ϕ n,h ě 1 {p 4 d 6 H 2 q , which indicates that

and so we can conclude that

by Lemma D.5 of Zhou and Gu (2022) and the fact that || ϕ n,h { ¯ σ n,h || 2 ď 1 { H 2 . As we require in Theorem 2 that N on , N off ' ˜ Ω p d 13 H 14 q , we come to the following result

Informally, this means that the proportion of trajectories in the 'bad set' I 2 is asymptotically zero, and the proportion in the 'good set' I 1 is asymptotically one. As for every n P I 1 we have that for any h P r H s ,

Now we invoke the total variance lemma. Recall that in Section B of the appendix in He et al. (2023), by the total variance lemma of Jin et al. (2019), if N off ě ˜ Ω p d 10 . 5 H 14 q , it holds that

With a direct application of Lemma 17, as we set T ' ˜ O p H q and γ ' c off p X off q{ N off , we will then get to

which indicates that

Now, from Lemmas 12 and 13, for any partition X off , X on , the offline regret satisfies

E.5 Online regret control

We will then bound the online term, Reg on p T q . He et al. (2023) show in Lemma E.1 that it is possible to use Cauchy-Schwarz to bound this by

and in Section B of the appendix, state that by the total variance lemma of Jin et al. (2019),

We will seek to use the online partition to tighten the dimensional dependence in the first result accordingly.

Lemma 14 (Modified Lemma E.1 in He et al. (2023)) . For any parameters β 1 ě 1 and C ě 1 , and any partition X off , X on , the summation of bonuses on the online partition is upper bounded by

where ι ' log p 1 ` N {p dλ qq .

Proof. For each horizon h P r H s , we first note that the summation can be bounded by the sum of two terms, where the first term is tight-bounded and the second term stands for a tail event where ϕ T Σ ´ 1 ϕ gets large.

We first bound ř T t ' 1 β 1 min ˜c ϕ ´ s p t q h , a p t q h ¯ J Σ ´ 1 t,h ϕ ´ s p t q h , a p t q h ¯ ✶ X on , 1 ¸ , using a variant of Lemma B.1 from Zhou and Gu (2022) in Lemma 18. With this, we have that

From this, it suffices to follow the rest of the proof of Lemma E.1 from He et al. (2023) to bound the remaining term by

As a result, we obtain the following bound for the online regret

E.6 Putting everything together

Combining our results in E.4 and E.5, we come to the bound of total regret that

When we set N on , N off ' ˜ Ω p d 13 H 14 q and choose X off , X on be the partition that minimize the right hand side, we have

proving Theorem 2.

F OPTCOV from Wagenmaker and Jamieson (2023)

We lean on the OPTCOV algorithm from Wagenmaker and Pacchiano (2023) for reward-agnostic exploration , first proposed in Wagenmaker and Jamieson (2023), as well as the Frank-Wolfe subroutine used, for completeness.

Algorithm 5 Collection of Optimal Covariates (OPTCOV), Wagenmaker and Pacchiano (2023)

The algorithm essentially performs the doubling trick to determine how many samples to collect, terminating when the minimum eigenvalue of the covariance matrix is above the set tolerance.

Wagenmaker and Pacchiano (2023) then prove the following guarantee for OPTCOV in the hybrid setting:

Lemma 15 (Termination of OPTCOV, Lemma C.2 (Wagenmaker and Pacchiano, 2023)) . Let

for some Λ 0 ,i satisfying Λ 0 ,i ľ Λ 0 for all i , and η i ' 2 2 i { 5 . Let p β i , M i q denote the smoothness and magnitude constants for f i . Let p β, M q be some values such that β i ď η i β, M i ď M for all i . Then, if we run OPTCOV on p f i q i with constraint tolerance ϵ and confidence δ , we have that with probability at least 1 ´ δ , it will run for at most

episodes, and will return data t ϕ τ u N τ ' 1 with covariance p Σ N ' ř N τ ' 1 ϕ τ ϕ J τ such that

where p i is the iteration on which OPTCOV terminates.

We use this to obtain a modified guarantee for OPTCOV that does not require a call to the CONDITIONEDCOV algorithm of Wagenmaker and Jamieson (2023).

Lemma 16 (Modified Bound on OPTCOV, Theorem 4, Wagenmaker and Pacchiano (2023)) . Consider running OPTCOV with some ϵ exp ą 0 and functions f i as defined in Lemma 15, instantiated with the regularization ¯ λ ě 0 . Then with probability 1 ´ δ , this procedure will collect at most

episodes, and will produce covariates p Σ such that

Proof. This is essentially the proof of Theorem 4 in Wagenmaker and Pacchiano (2023), except where we chase around a few terms that differ in the analysis. By Lemma D.5 of Wagenmaker and Jamieson (2023), it suffices to bound the smoothness constants of f i p Λ q by

Assume that the termination condition of OPTCOV is met for p i satisfying

We assume this holds and justify it at the conclusion of the proof. For notational convenience, define

Given this upper bound on p i , set

With this choice of L, M, β , we have L i ď L, M i ď M,β i ď η i β for all i ď p i .

Now apply Lemma 15 with Λ 0 ' ¯ λ ¨ I and get that, with probability at least 1 ´ δ , OPTCOV terminates after at most

episodes, and returns data t ϕ τ u N τ ' 1 with covariance p Σ ' ř N τ ' 1 ϕ τ ϕ J τ such that

where p i is the iteration on which OPTCOV terminates.

By Lemma D.1 of Wagenmaker and Jamieson (2023) we have

and the upper bound on the tolerance follows from Lemma D.8 of Wagenmaker and Jamieson (2023).

It remains to justify the bound on p i . We do so with the same argument that Wagenmaker and Pacchiano (2023) use. Note that by the definition of OPTCOV, if we run for a total of ¯ N episodes, we can bound p i ď 1 4 log 2 p ¯ N q . However, we see that the bound on p i given above upper bounds 1 4 log 2 p ¯ N q for ¯ N the upper bound on the number of samples collected by OPTCOV stated above. Thus, the bound on ˆ i is valid.

G Miscellanous lemmas

Lemma 17. Let Φ Ă R d be a linear subspace. Suppose t ϕ h,n u h Pr H s ,n Pr N s P Φ be a collection of unit vectors and t σ h,n u h Pr H s ,n Pr N s P R ` be a collection of positive real numbers with mean ¯ σ ' p NH q ´ 1 ř h,n σ h,n . Suppose it holds that max h Pr H s max ϕ h P Φ p ϕ T h Λ ´ 1 h ϕ h q ď γ, then the following result satisfies

with

Proof. First, we denote ¯ σ h ' N ´ 1 ř n σ h,n . Informally, this implies that most individuals of σ h, ¨ is asymptotically on the order of ¯ σ h , with only a small amount of individuals being higher in order. To rule out the effect of the 'large' ones, we group them into the following collection of sets:

Here, we leave the choice of the truncation level C h open for now, but note that we allow the truncation levels C h vary across different timesteps h and related to ¯ σ h . It follows by definition that ř H h ' 1 ¯ σ h ' H ¯ σ. From an application of Markov's Inequality, the cardinality of set E h p C h q can be upper bounded as

We now choose the truncation level C h . To do so, we follow the steps below to quantify the effect induced by the trajectories with high variance (i.e. those that belong to E h p C h q ):

Finally, we note that

Lemma18 (Modified Lemma B.1 from Zhou and Gu (2022)) . Let X off , X on be a partition of S ˆ A ˆ r H s , such that their images under the feature map, Φ off , Φ on are subspaces of dimension d off , d on respectively. Let t σ k , β k u k ě 1 be a sequence of non-negative numbers, α, γ ą 0 , t x k u k ě 1 Ă R d and } x k } 2 ď L . Let t Z k u k ě 1 and t ¯ σ k u k ě 1 be recursively defined as follows: Z 1 ' λ I ` Z off for some symmetric matrix Z off , where N ' N off ` K , and we have

Let ι ' log ` 1 ` NL 2 { ` dλα 2 ˘˘ . Then we have

Proof. The proof roughly follows that of Lemma B.1 in Zhou and Gu (2022), except that we have to make modifications as necessary to tighten the dimension dependence to d on and incorporate the offline data.

We now utilize a basic matrix inequality that for any matrix A,B , we have

which allows us to further bound min ϕ h P Φ ϕ J h Σ ‹ h ϕ h as

This leads to the following result:

where the first equality holds because Λ h is a linear transformation on the subspace Φ . Equivalently, this holds from the variational characterization of the eigenvalues and the fact that the largest absolute eigenvalue is equal to the inverse of the smallest absolute eigenvalue of the inverse. As a result, in order to rule out the effect of the 'high variance trajectories', we select the truncation level δ h such that N { C h ' Θ p γ ´ 1 q , implying C h ' Θ p Nγ q . Hence, we obtain the following lower bound:

Decompose the set r K s into a union of two disjoint subsets r K s ' I 1 Y I 2 ,

Then the following upper bound of | I 1 | holds, where the projector P on onto Φ on has the decomposition P on ' U on U J on by the thin SVD, and we write u k ' U J on x k :

!

)

By Lemma 20, we can take the U on inside the inverse and conclude that

Writing u n ' U J on x n , and invoking Lemma D.5 of Zhou and Gu (2022) (which is a restatement of Lemma 11 of Abbasi-yadkori et al. (2011)) and the fact that } x k { ¯ σ k } 2 ď L { α , it holds that

Intuitively, this is because all the ✶ X on U J on x n and ✶ X on x n are both in Φ on , and in that case the projection is just the identity.

as desired, and conclude that | I 1 | ď 2 d on ι.

The rest of the proof follows Zhou and Gu (2022) more closely. By the same argument that Zhou and Gu (2022) use,

Decompose I 2 ' J 1 Y J 2 , where

Similar to Zhou and Gu (2022),

and as for k P J 2 we have that ¯ σ k ' γ 2 } x k { ¯ σ k } Z ´ 1 X on ,

Therefore,

Lemma 19 (Modified Version of Theorem 4.3, Zhou and Gu (2022)) . Let t G n u N n ' 1 be a filtration, and t x n , η n u N n ' 1 be a stochastic process such that x n P R d is G n -measurable and η n P R is G n ` 1 -measurable. Let L, σ, λ, ϵ ą 0 , µ ˚ P R d . Arrange the datapoints from the offline and online samples as follows, 1 , ..., N off , N off ` 1 , ..., N off ` N on . For n ' 1 , ..., N , let y n ' x µ ˚ , x n y ` η n and suppose that η n , x n also satisfy

For n ' 1 , ..., N , let Z n ' λ I ` ř n i ' 1 x i x J i , b n ' ř n i ' 1 y i x i , µ n ' Z ´ 1 n b n , and

Then, for any 0 ă δ ă 1 , we have with probability at least 1 ´ δ that,

Proof. The proof is merely a small wrapper over Theorem 4.3 of Zhou and Gu (2022), where we adapt this to our setting in the same way that Tan and Xu (2024) do in Lemma 1 of their paper. That is, we pre-append the offline data to the online data, and generate the Z n , b n , µ n , β n accordingly.

As in Lemma 1 of Tan and Xu (2024), let N ' N off ` N on . Order the N off offline episodes arbitrarily, to form episodes 1 , ..., N off , and then begin the online episodes from episode N off ` 1 , ..., N . Then, we can directly apply Theorem 4.3 of Zhou and Gu (2022) to recover the desired result.

Lemma 20. Suppose that W ' R m and V ' R n , where n ă m . Let U : W ÞÑ V be a linear transformation and that S ' p U J U q W . As v , v 1 , . . . , v n P S , we have

Proof. For projection matrix U , there exists orthogonal matrix Q P R m ˆ m and diagonal matrix D ' p I n , 0 n ˆp m ´ n q q such that U ' DQ . We further define u ' Uv , ˜ v ' Qv , u i ' Uv i and ˜ v i ' Qv i for 1 ď i ď n . Then, we note that as v P S , we have v ' U J Uv ' Q J Λ Qv , where Λ ' diag p I n , 0 m ´ n q , which is equivalent to ˜ v ' Λ˜ v . As a result, we may conclude that ˜ v J ' p u J , 0 m ´ n q .

Therefore, with a direct calculation, one will see that

This will establish our desired conclusion

H Further details on the numerical experiments

The environment, as in Tan and Xu (2024), is a 6 -piece wide Tetris board with pieces no larger than 2 ˆ 2 , where the action space consists of four actions, differentiated by the degree of rotation in 90 degree intervals and the reward is given by penalizing any increases in the height of the stack from a tolerance of 2 blocks. As in Tan and Xu (2024), we generate feature vectors by projecting the 640 -dimensional one-hot state-action encoding onto a 60 -dimensional subspace spanned by the top 60 eigenvectors of the feature covariance matrix under 200 trajectories from the uniform behavior policy.

All experiments were run on a single computer with an Intel i9-13900k CPU, 128 GB of RAM, and a NVIDIA RTX3090 GPU, in no more than a couple of hours.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that our algorithms achieve sharper error or regret bounds that are no worse than, and can improve on, (1) the optimal sample complexity in offline RL (the first algorithm, for PAC RL) and online RL (the second algorithm, for regret-minimizing RL) in linear Markov decide processes (MDPs), and (2) that this work establishes the tightest theoretical guarantees currently available for hybrid RL in linear MDPs. Theorems 1 and 2, and the accompanying discussion, support claim (1) by being no worse than the associated bounds from Xiong et al. (2023) and He et al. (2023) respectively. Table 2 supports claim (2) by showcasing our improvement over existing hybrid RL methods for linear MDPs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a discussion of the limitations of our algorithms and analysis in Section 5 and in the discussion immediately after Theorems 1 and 2. The latter also contains a discussion on the computational efficiency of our algorithms.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our assumptions and setting are outlined in Assumptions 1 and 2, and Section 2. Any further conditions are outlined in the statement of our theorems and lemmas. Though we provide proof sketches in the main body, the proof of Theorem 1 is found in Appendix B, and the proof of Theorem 2 is found in Appendix E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is available on GitHub at github.com/hetankevin/hybridlin. The full details are in Section 4 and Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available on GitHub at github.com/hetankevin/hybridlin.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full details are in Section 4 and Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: One standard deviation error bars are displayed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were run on a single computer with an Intel i9-13900k CPU, 128 GB of RAM, and a NVIDIA RTX3090 GPU, in no more than a couple of hours.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and reviewed the code of ethics. Our paper is a theoretical contribution to the field of reinforcement learning theory, and as such conforms to the guidelines outlined.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper is a theoretical contribution to the field of reinforcement learning theory, and as such any societal impact will at the very least be second-order impacts not directly tied to our work. Having said that, we discuss that Algorithm 1 does not randomize over policies, allowing learned policies to be deployed in critical real-world applications, and Algorithm 2 minimizes regret over the online samples, allowing it to be used in situations where performance-agnostic online exploration is untenable, such as in medicine.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies hybrid RL in linear MDPs, aiming to address the problem of whether hybrid RL can improve upon the existing lower bounds established in purely offline and purely online settings, without relying on the single-policy concentrability assumption. By combining offline dataset with online interaction, authors introduce computationally efficient algorithms which achieve sharper error bound and regret bound in offline and online settings.\n\nSTRENGTHS:\nExisting theoretical works of hybrid RL under function approximation mainly focus on PAC guarantees which are not tight and require stringent concentrability assumptions on the quality of the behavior policy. Motivated by the question of whether hybrid RL is useful (i.e. improving upon the existing lower bounds established in purely offline and purely online RL without relying on the single-policy concentrability assumption, which is raised by Xie et al. (2022b)), authors provide computationally efficient algorithms for both PAC and regret-minimizing RL for linear MDPs, without relying on single-policy concentrability.\n\n\nIn particular, this work exhibits several interesting findings:\n1.  Algorithmically, two types of hybrid RL algorithms are introduced: an online-to-offline algorithm, which involves reward-agnostic online exploration followed by a pessimistic offline algorithm to learn an optimal policy; and additionally, an offline-to-online method that utilizes offline data to warm start an online algorithm.\n\n2. Theoretically, authors show both algorithms improve upon existing sample complexity, which is measured with respect to either the PAC framework or the regret.\n\nThis work demonstrates clear rationale and compelling motivation, while clearly articulates its main ideas throughout the draft. Thorough discussion of related works have been provided. It develops a better theoretical understanding of hybrid RL in linear MDPs, which potentially benefits future works for hybrid RL with functional approximation in this context.\n\nWEAKNESSES:\nBelow are several potential improvements that authors are suggested to consider:\n\n1. It is better to summarize and highlight the main technical novelties in the main text. There exist extensive theoretical studies in either linear MDPs or in hybrid RL. It is still not quite clear throughout the main text what the main technical challenges are in achieving the optimal policy in the studied context and what the technical novelties are in improving existing theoretical results. It seems to me that the proofs closely follow Xiong et al. (2023) in offline RL,  Wagenmaker and Pacchiano (2023) in reward-agnostic exploration and He et al. (2023) in linear MDPs. As a result, it is important to point out how the studied settings can be technically challenging, and what novel arguments are developed compared to existing works.\n\nQUESTIONS:\n1. Could you highlight the main technical techniques / reason why Alg. 2 can achieve better regret compared to Tan and Xu, 2024 and Amortila et al., 2024 for the linear MDP case?\n\n2. In the studied setting, if we consider general function approximation or a more general class of MDPs with linear structure, do you envision the current analysis and results can be utilized to improve existing bounds?\n\n3. Could you comment on the optimality of the provided bounds, whether dependence on the parameters involved can be further improved?\n\n4. Could you provide an intuitive explanation of concentrability coefficient? And what is the range that indicates good concentrability? In line 135, does $d^*$ represent the occupancy measure of the optimal policy?\n\nLIMITATIONS:\nThis is a theoretical work, no negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies the hybrid reinforcement learning problem in the linear MDP setting. It provides two algorithms (one focused on improving the offline error and the other on improving the online error) with theoretical analysis on their sample complexity. Though the algorithms are not optimal in terms of sample complexity with respect to some problem parameters, it appears to be the state-of-the-art in the literature for the linear setting.

STRENGTHS:
The paper is clearly written; the problem formulation and assumptions are clearly stated. The theoretical results seem sound.

WEAKNESSES:
Aside from the suboptimality in $d$ and $H$ already pointed out by the authors, another weakness is that Algorithm 2 has a pretty large minimum requirement for the size of the offline dataset.

QUESTIONS:
It would be nice to discuss how Algorithm 1 compares with Algorithm 2, aside from the difference in approach (e.g., why isn’t one clearly better than the other).

Why is $|A|$ in Line 303? If that is not a typo, then it seems to be another weakness of Algorithm 2, since the action space might not be discrete.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors develop sample and computationally efficient hybrid RL algorithms that are provably better than online-only and offline-only algorithms for linear MDPs. Without relying on the single-policy concentrability assumption, the authors take both online-to-offline and offline-to-online approaches to achieve no worse than optimal sample complexity, regardless of the quality of the behavior policy.\n\nSTRENGTHS:\nThis work demonstrates a thorough discussion of all relevant literature, while the results are extensively compared with existing methods.\n\nThis work provides no worse than optimal sample complexity for two types of hybrid RL methods, which is non-trivial.\n\nThe presentation is generally clear, and the appendix is well-organized. The proof appears sound from a quick skim.\n\nWEAKNESSES:\nThe assumption regarding Full Rank Projected Covariates seems to implicitly impose some constraints. \n\nNo experimentation or code is included with the work, making it difficult to examine whether the proposed algorithms are efficient in practice.\n\nQUESTIONS:\nQ: I am not familiar with Assumption 2: Full Rank Projected Covariates. Is it a common assumption? How practical is it in downstream applications?\n\nQ: HYRULE seems to be a straightforward generalization of existing algorithms. Could you please list the challenges in proving its regret guarantee?\n\nQ: It would be helpful if there is an experimental plan to verify the algorithms. A simple toy experiment plan should suffice.\n\nMinor suggestion:\n\nAs mentioned in line 241, OPTCOV requires tolerance parameter. Should this parameter also listed in the input of RAPPEL (Algorithm 1)?\n\nLIMITATIONS:\nSame as weakness. This work focuses on the setting of linear MDPs, where the techniques may not be generalizable to other types of function approximations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper presents studies Hybrid Reinforcement Learning for linear MDPs, where Hybrid RL addresses the limitations of purely offline and online methods by combining offline data and online exploration. The paper introduces two specific algorithms: Reward-Agnostic Pessimistic PAC Exploration-initialized Learning (RAPPEL) and Hybrid Regression for Upper-Confidence Reinforcement Learning (HYRULE). RAPPEL is a online-to-offline approach where one first perform reward agnostic exploration to increase the coverage of the existing data and then perform offline RL method, and HYRULE performs offline-to-online method by starting LSVI_UCB++ with the offline data. Both methods shows improvement over preivous methods, and no worse than offline minimax and online minimax rate, respectively.

STRENGTHS:
1. The paper is well-organized, and the literature review is thorough. 

2. The proposed method clearly improves on the previous hybrid RL method for linear MDPs, and the online part of the rate indeed of Alg.2 matches the online minimax rate of linear mdps. Overall the improvement of the result is solid. 

3. The improvement of the result leverages existing algorithms that are (relatively) well-known as subroutines with intuitive modifications (plus sharper analysis at certain part), which are easy to relate to the literature and intuitive.

WEAKNESSES:
1. There is no clear new techniques introduced in the paper. 

2. The offline rates match the minimax rate only up to the coverage term - the minimax coveage is the single policy coverage, while the proposed algorithm and analysis depends on the all policy coverage.

QUESTIONS:
see above

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13