Title: Kernelized Reinforcement Learning with Order Optimal Regret Bounds
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Kernel ridge regression

Gaussian processes

LSVI
Abstract: Modern reinforcement learning (RL) has shown empirical success in various real world settings with complex models and large state-action spaces. The existing analytical results, however, typically focus on settings with a small number of state-actions or simple models such as linearly modeled state-action value functions. To derive RL policies that efficiently handle large state-action spaces with more general value functions, some recent works have considered nonlinear function approximation using kernel ridge regression. We propose $\pi$-KRVI, an optimistic modification of least-squares value iteration, when the action-value function is represented by an RKHS. We prove the first order-optimal regret guarantees under a general setting. Our results show a significant polynomial in the number of episodes improvement over the state of the art. In particular, with highly non-smooth kernels (such as Neural Tangent kernel or some Matérn kernels) the existing results lead to trivial (superlinear in the number of episodes) regret bounds. We show a sublinear regret bound that is order optimal in the cases where a lower bound on regret is known (which includes the kernels mentioned above).

Kernelized Reinforcement Learning with Order Optimal Regret Bounds

Abstract

Reinforcement learning (RL) has shown empirical success in various real world settings with complex models and large state-action spaces. The existing analytical results, however, typically focus on settings with a small number of state-actions or simple models such as linearly modeled state-action value functions. To derive RL policies that efficiently handle large state-action spaces with more general value functions, some recent works have considered nonlinear function approximation using kernel ridge regression. We propose π -KRVI, an optimistic modification of least-squares value iteration, when the state-action value function is represented by a reproducing kernel Hilbert space (RKHS). We prove the first order-optimal regret guarantees under a general setting. Our results show a significant polynomial in the number of episodes improvement over the state of the art. In particular, with highly non-smooth kernels (such as Neural Tangent kernel or some Matérn kernels) the existing results lead to trivial (superlinear in the number of episodes) regret bounds. We show a sublinear regret bound that is order optimal in the case of Matérn kernels where a lower bound on regret is known.

1 Introduction

Reinforcement learning (RL) in real world often has to deal with large state action spaces and complex unknown models. While RL policies using complex function approximations have been empirically effective in various fields including gaming (Silver et al., 2016; Lee et al., 2018; Vinyals et al., 2019), autonomous driving (Kahn et al., 2017), microchip design (Mirhoseini et al., 2021), robot control (Kalashnikov et al., 2018), and algorithm search (Fawzi et al., 2022), little is known about theoretical performance guarantees in such settings. The analysis of RL algorithms has predominantly focused on simpler cases such as tabular or linear Markov decision processes (MDPs). In a tabular setting, a regret bound of ˜ O ( √ H 3 |S × A| T ) has been shown for optimistic state-action value learning algorithms (e.g., see, Jin et al., 2018), where H is the length of episodes, T is the number of episodes, and S and A are finite state and action spaces. This bound does not scale well when the size of state-action space grows large. Furthermore, when the model (the state-action value function or the transitions) admits a d -dimensional linear representation in some state-action features, a regret bound of ˜ O ( √ H 3 d 3 T ) is established (Jin et al., 2020), that scales with the dimension of the linear model rather than the cardinality of the state-action space.

Several recent studies have explored the utilization of complex models with large state-action spaces. A very general model entails representing the state-action value function using a reproducing kernel Hilbert space (RKHS). This approach allows using kernel ridge regression to obtain confidence intervals, which facilitate the design and analysis of RL algorithms. The most significant contribution

Julia Olkhovskaya

TU Delft ∗

to this general RL problem is Yang et al. (2020a), 2 that provides regret guarantees for an optimistic least-squares value iteration (LSVI) algorithm, referred to as kernel optimistic least-squares value iteration (KOVI). The main assumption is that the state-action value function can be represented using the RKHS of a known kernel k . The regret bounds reported in Yang et al. (2020a) scale as ˜ O ( H 2 √ (Γ( T ) + log N ( ϵ )) Γ( T ) T ) , with ϵ = H T , where Γ( T ) and N ( ϵ ) are two kernel related complexity terms, respectively, referred to as maximum information gain and ϵ -covering number of the class of state-action value functions. The definitions are given in Section 4. Both complexity terms are determined using the spectrum of the kernel. While for smooth kernels, characterized by exponentially decaying Mercer eigenvalues, such as Squared Exponential kernel, Γ( T ) and log N ( H T ) are logarithmic in T , for more general kernels with greater representation capacity, these terms may grow polynomially in T , possibly making the regret bound trivial (superlinear).

To have a better understanding of the existing result, let { σ m > 0 } ∞ m =1 denote the Mercer eigenvalues of the kernel k in a decreasing order. Also, let { ϕ m } ∞ m =1 denote the corresponding eigenfeatures. Refer to Section 2.2 for details. The kernel k is said to have a polynomial eigendecay when σ m decay at least as fast as m -p for some p > 1 . The polynomial eigendecay profile satisfies for many kernels of practical and theoretical interest such as Matérn family of kernels (Borovitskiy et al., 2020) and the Neural Tangent (NT) kernel (Arora et al., 2019). For a Matérn kernel with smoothness parameter ν on a d -dimensional domain, p = 2 ν + d d (e.g., see, Janz et al., 2020). For a NT kernel with s -1 times differentiable activations, p = 2 s -1+ d d (Vakili et al., 2021b). In Yang et al. (2020a), the regret bound is specialized for the class of kernels with polynomially decaying eigenvalues, by bounding the complexity terms based on the kernel spectrum. However, the reported regret bound is sublinear in T only when the kernel eigenvalues decay very fast. In particular, let ˜ p = p (1 -2 η ) , where for η ≥ 0 , σ η m ϕ m is uniformly bounded. Then, Yang et al. (2020a), Corollary 4 . 4 reports a regret bound of ˜ O ( T ξ ∗ + κ ∗ + 1 2 ) , with

The regret bound ˜ O ( T ξ ∗ + κ ∗ + 1 2 ) is sublinear only when p and ˜ p are sufficiently large. That, at least, requires 2 ξ ∗ < 1 2 , implying p > d +2 , when ˜ p is also sufficiently large. For instance, for Matérn kernels, this requirement can be expressed as ν > d ( d +1) 2 , when (2 ν + d )(1 -2 η ) d is sufficiently large.

Special case of bandits. A similar issue existed in the simpler problem of kernelized bandits, corresponding to the special case where H = 1 , |S| = 1 . Specifically, the ˜ O (Γ( T ) √ T ) regret bounds reported for optimistic sampling (Srinivas et al., 2010a, GP-UCB), as well as for Thompson sampling (Chowdhury and Gopalan, 2017, GP-TS) are also trivial (superlinear) when Γ( T ) grows faster than √ T . It remains an open problem whether the suboptimal performance guarantees for these two algorithms is a fundamental shortcoming or an artifact of the proof. This observation is formalized as an open problem on the online confidence intervals for RKHS elements in Vakili et al. (2021d). For the kernelized bandits problem, Scarlett et al. (2017) proved lower bounds on regret in the case of Matérn family of kernels. In particular, they proved an Ω( T ν + d 2 ν + d ) lower bound on regret of any bandit algorithm. Several recent algorithms, different from GP-UCB and GP-TS, have been developed to alleviate the suboptimal and superlinear regret bounds in kernelized bandits and obtain an ˜ O ( √ Γ( T ) T ) regret bound (Li and Scarlett, 2022; Salgia et al., 2021), that matches the lower bound in the case of the Matérn family of kernels, up to logarithmic factors. The Sup variations of the UCB algorithms also obtain the optimal regret bound in the contextual kernel bandit setting with finite actions (Valko et al., 2013).

Main contribution. The RL setting presents a greater level of complexity compared to the bandit setting due to the Markovian dynamics. None of the solutions in Li and Scarlett (2022); Salgia et al. (2021); Valko et al. (2013) seem appropriate in the presence of MDP dynamics, thereby leaving the question of order optimal regret bounds largely open. In this work, we leverage the scaling of the kernel spectrum with the size of the domain to improve the regret bounds. We consider kernels with polynomial eigendecay on a hypercubical domain with side length ρ , where eigenvalues scale with ρ α for some α > 0 . See Definition 1. This encompasses a large class of common kernels, including the Matérn family, for which, α = 2 ν . The hypercube domain assumption is a technical formality

that can be relaxed to other regular compact subsets of R d . In Section 3, we propose a domain partitioning kernel ridge regression based least-squares value iteration policy ( π -KRVI) that achieves sublinear regret of ˜ O ( H 2 T d + α/ 2 d + α ) for kernels introduced in Definition 1. This is the first sublinear regret bound under such a general stetting. Moreover, with Matérn kernels, our regret bound matches the Ω( T ν + d 2 ν + d ) lower bound reported in in Scarlett et al. (2017) for the special case of kernelized bandits, up to a logarithmic factor.

Our proposed policy, π -KRVI, is based on least-squares value iteration (similar to KOVI, Yang et al. (2020a)). However, in order to effectively utilize the confidence intervals from kernel ridge regression, π -KRVI creates a partitioning of the state-action domain and builds the confidence intervals only based on the observations within the same partition element. The domain partitioning allows us to leverage the scaling of the kernel eigenvalues with respect to the domain size. The inspiration for this idea is drawn from π -GP-UCB algorithm introduced in Janz et al. (2020) for kernelized bandits. In comparison to Janz et al. (2020), π -KRVI and its analysis present greater complexity due to the Markovian dynamics in the MDP setting. Furthermore, we provide a finer analysis that significantly improves the results compared to Janz et al. (2020). Although Janz et al. (2020) obtained sublinear regret guarantees of ˜ O ( T 2 ν + d (2 d +3) 4 ν + d (2 d +4) ) in the kernelized bandit setting with Matérn kernel, there still remained a polynomial in T gap between their regret bounds and the lower bound reported in Scarlett et al. (2017). As a consequence of our results, we also close this gap.

There are several novel contributions in our analysis that lead to the improved and order optimal regret bounds. We establish confidence intervals for kernel ridge regression that apply uniformly to all functions in the state-action value function class (Theorem 1). A similar confidence interval was given in Yang et al. (2020a). We however provide flexibility with respect to setting the parameters of the confidence interval, that eventually contributes to the improved regret bounds, with a proper choice of parameters. We also derive bounds on the maximum information gain (Lemma 2) and the function class covering number (Lemma 3), taking into consideration the size of the state-action domain. These bounds are important for the analysis of our domain partitioning policy which effectively controls the number of observations utilized in kernel ridge regression by partitioning the domain into subdomains of diminishing size. These intermediate results may also be of general interest in similar problems.

The π -KRVI policy enjoys an efficient runtime, polynomial in T , and linear in |A| , similar to the runtime of KOVI (Yang et al., 2020a). The dependency of the runtime on |A| limits the scope of the policy to finite A , while allowing a continuous S (with |S| infinite). The assumption of finite A can be relaxed, provided there is an efficient optimizer of a certain state-action value function. See the details in Section 3.2.

Other related work. There is an extensive literature on the analysis of RL policies which do not rely on a generative model or an exploratory behavioral policy. The literature has primarily focused on the tabular setting (Jin et al., 2018; Auer et al., 2008; Bartlett and Tewari, 2012). The domain of potential applications for this setting is very limited, as in many real world problems, the state-action space is very large or even infinite. In response to this, recent literature has placed a notable emphasis on employing function approximation in RL, particularly within the context of generalized linear settings. This approach involves representing the value function or transition model through a linear transformation to a well-defined feature mapping. Important contributions include the work of Jin et al. (2020); Yao et al. (2014), as well as subsequent studies by Russo (2019); Zanette et al. (2020a,b); Neu and Pike-Burke (2020); Yang and Wang (2020). Furthermore, there have been several efforts to extend these techniques to a kernelized setting, as explored in Yang et al. (2020a); Yang and Wang (2020); Chowdhury and Gopalan (2019); Yang et al. (2020c); Domingues et al. (2021). These works are also inspired by methods originally designed for linear bandits (Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013), as well as kernelized bandits (Srinivas et al., 2010b; Valko et al., 2013; Chowdhury and Gopalan, 2017). However, all known regret bounds in the RL setting (Yang et al., 2020a; Yang and Wang, 2020; Chowdhury and Gopalan, 2019; Yang et al., 2020c; Domingues et al., 2021) are not order optimal. We compare our regret bounds with the state of the art reported in Yang et al. (2020a). A similar issue existed for classic kernelized bandit algorithms. A detailed discussion can be found in Vakili et al. (2021d). The authors in Yang and Wang (2020) considered finite state-actions under a kernelized MDP model where the transition model can be directly estimated. That is different from the setting considered in our work and Yang et al. (2020a).

2 Preliminaries and Problem Formulation

In this section, we overview the background on episodic MDPs and kernel ridge regression.

2.1 Episodic Markov Decision Processes

An episodic MDP can be described by the tuple M = ( S , A , H, P, r ) , where S is the state space, A is the action space, the integer H is the length of each episode, r = { r h } H h =1 are the reward functions and P = { P h } H h =1 are the transition probability distributions. 2 We use the notation Z = S × A to denote the state-action space. For each h ∈ [ H ] , the reward r h : Z → [0 , 1] is the reward function at step h , which is supposed to be deterministic for simplicity, and P h ( ·| s, a ) is the transition probability distribution on S for the next state from state-action pair ( s, a ) . The choice of deterministic rewards allows us to concentrate on the core complexities of the problem, and should not be regarded as a limitation. Both the framework and results can be readily extended to a setting with random rewards.

A policy π = { π h } H h =1 , at each step h , determines the (possibly random) action π h : S → A taken by the agent at state s . At the beginning of each episode t = 1 , 2 , · · · , the environment picks an arbitrary state s t 1 . The agent determines a policy π t = { π t h } H h =1 . Then, at each step h ∈ [ H ] , the agent observes the state s t h ∈ S , picks an action a t h = π t h ( s t h ) and observes the reward r h ( s t h , a t h ) . The new state s t h +1 then is drawn from the transition distribution P h ( ·| s t h , a t h ) . The episode ends when the agent receives the final reward r H ( s t H , a t H ) .

The goal is to find a policy π that maximizes the expected total reward in the episode, starting at step h , i.e., the value function defined as

where the expectation is taken with respect to the randomness in the trajectory { ( s h , a h ) } H h =1 obtained by the policy π . It can be shown that under mild assumptions (e.g., continuity of P h , compactness of Z , and boundedness of r ) there exists an optimal policy π ⋆ which attains the maximum possible value of V π h ( s ) at every step and at every state (e.g., see, Puterman, 2014). We use the notation V ⋆ h ( s ) = max π V π h ( s ) , ∀ s ∈ S , h ∈ [ H ] . By definition V π ⋆ h = V ⋆ h . For a value function V : S → [0 , H ] , we define the following notation

We also define the state-action value function Q π h : Z → [0 , H ] as follows.

where the expectation is taken with respect to the randomness in the trajectory { ( s h , a h ) } H h =1 obtained by the policy π . The Bellman equation associated with a policy π then is represented as

where the expectation is taken with respect to the randomness in the policy π . The Bellman optimality equation is also given as Q ⋆ h ( s, a ) = r h ( s, a ) + [ P h V ⋆ h +1 ]( s, a ) , V ⋆ h ( s ) = max a Q ⋆ h ( s, a ) , V ⋆ H +1 := 0 . The performance of a policy π t is measured in terms of the loss in the value function, referred to as regret , denoted by R ( T ) in the following definition

Recall that π t is the policy executed by the agent at episode t , where s t 1 is the initial state in that episode determined by the environment.

2.2 Kernel Ridge Regression

We assume that the state-action value functions belong to a known reproducing kernel Hilbert space (RKHS). See Assumption 1 and Lemma 1 for the formal statement. This is a very general assumption, considering that the RKHS of common kernels can approximate almost all continuous functions on the compact subsets of R d (Srinivas et al., 2010a). Consider a positive definite kernel k : Z × Z → R . Let H k be the RKHS induced by k , where H k contains a family of functions defined on Z . Let ⟨· , ·⟩ H k : H k ×H k → R and ∥ · ∥ H k : H k → R denote the inner product and the norm of H k , respectively. The reproducing property implies that for all f ∈ H k , and z ∈ Z , ⟨ f, K ( · , z ) ⟩ H k = f ( z ) . Without loss of generality, we assume k ( z, z ) ≤ 1 for all z . Mercer theorem implies, under certain mild conditions, k can be represented using an infinite dimensional feature map:

where σ m > 0 , and √ σ m ϕ m ∈ H k form an orthonormal basis of H k . In particular, any f ∈ H k can be represented using this basis and wights w m ∈ R as

where ∥ f ∥ 2 H k = ∑ ∞ m =1 w 2 m . A formal statement and the details are provided in Appendix A. We refer to σ m and ϕ m as (Mercer) eigenvalues and eigenfeatures of k , respectively.

Kernel-based models provide powerful predictors and uncertainty estimators which can be leveraged to guide the RL algorithm. In particular, consider a fixed unknown function f ∈ H k . Consider a set Z t = { z i } t i =1 ⊂ Z of t inputs. Assume t noisy observations { Y ( z i ) = f ( z i ) + ε i } t i =1 are provided, where ε i are independent zero mean noise terms. Kernel ridge regression provides the following predictor and uncertainty estimate, respectively (see, e.g., Schölkopf et al., 2002),

where k Z t ( z ) = [ k ( z, z 1 ) , . . . , k ( z, z t )] ⊤ is a t × 1 vector of the kernel values between z and observations, K Z t = [ k ( z i , z j )] t i,j =1 is the t × t kernel matrix, Y Z t = [ Y ( z 1 ) , . . . , Y ( Z t )] ⊤ is the t × 1 observation vector, I is the identity matrix of dimensions t , and λ > 0 is a free regularization parameter. The predictor and uncertainty estimate could be interpreted as posterior mean and variance of a surrogate centered Gaussian process (GP) model with covariance k , and zero mean Gaussian noise with variance λ 2 (e.g., see, Williams and Rasmussen, 2006).

2.3 Technical Assumption

We assume that the reward functions { r h } H h =1 and the transition probability distributions P h ( s ′ |· , · ) belong to the 1 -ball of the RKHS. We use the notation B k,R = { f : ∥ f ∥ H k ≤ R } to denote the R -ball of the RKHS.

Assumption 1 We assume

This is a mild assumption considering the generality of RKHSs, that is also supposed to hold in Yang et al. (2020a). Similar assumptions are made in linear MDPs which are significantly more restrictive (e.g., see, Jin et al., 2020).

An immediate consequence of Assumption 1 is that for any integrable V : S → [0 , H ] , r h + [ P h V h +1 ] ∈ B k,H +1 . This is formalized in the following lemma.

Lemma 1 Consider any integrable V : S → [0 , H ] . Under Assumption 1, we have r h +[ P h V ] ∈ B k,H +1 . (11)

See (Yeh et al., 2023, Lemma 3 ) for a proof.

3 Domain Partitioning Least-Squares Value Iteration Policy

A standard policy in episodic MDPs is the least-squares value iteration (LSVI), which computes an estimate ̂ Q t h for Q ⋆ h at each step h of episode t , by recursively applying Bellman equation as discussed in the previous section. In addition, an exploration bonus term b t h : Z → R is typically added leading to

The term ̂ Q t h + βb t h is an upper confidence bound on the state-action value function, that is inspired by the principle of optimism in the face of uncertainty . Since the rewards are assumed to be at most 1 , the state-action value function at step h is also bounded by H -h +1 . In episode t , then π t is the greedy policy with respect to Q t = { Q t h } H h =1 . Under Assumption 1, the estimate ̂ Q t h , the parameter β and the exploration bonus b t h can all be designed using kernel ridge regression. Specifically, having the Bellman equation in mind, ̂ Q t h is the (kernel ridge) predictor for r h +[ P h V t h +1 ] using (possibly some of) the past t -1 observations { r h ( z τ h ) + V t h +1 ( s τ h +1 ) } t -1 τ =1 at points { z τ h } t -1 τ =1 . Recall that E [ r h ( z τ h ) + V t h +1 ( s τ h +1 ) ] = r h ( z τ h ) + [ P h V t h +1 ]( z τ h ) , where the expectation is taken with respect to P h ( ·| z τ h ) . The observation noise V t h +1 ( s τ h +1 ) -[ P h V t h +1 ]( z τ h ) is due to random transitions and is bounded by H -h ≤ H .

3.1 Domain Partitioning

To overcome the suboptimal performance guarantees rooted in the online confidence intervals in kernel ridge regression, we introduce domain partitioning kernel ridge regression based least-squares value iteration ( π -KRVI). The proposed policy partitions the state-action space Z into subdomains and builds kernel ridge regression only based on the observations within each subdomain. By doing so, we obtain tighter confidence intervals, ultimately resulting in a tighter regret bound. To formalize this procedure, we consider the state-action space Z ⊂ [0 , 1] d . Let S t h , h ∈ [ H ] , t ∈ [ T ] be sets of hypercubes overlapping only at edges, covering the entire [0 , 1] d . For any hypercube Z ′ ∈ S t h , we use ρ Z ′ to denote the length of any of its sides, and N t h ( Z ′ ) to denote the number of observations at step h in Z ′ up to episode t :

For all h ∈ [ H ] , we initialize S 1 h = { [0 , 1] d } . At each episode t , for each step h , after observing a sample from r h + [ P h V t h +1 ] at ( s t h , a t h ) , we construct a new cover S t h as follows. We divide every element Z ′ ∈ S t -1 h that satisfies ρ -α Z ′ < | N t h ( Z ′ ) | + 1 , into two equal halves along each side, generating 2 d hypercubes. The parameter α > 0 in the splitting rule is a constant specified in Definition 1. Subsequently, we define S t h as the set of newly created hypercubes and the elements of S t -1 h that were not split.

The construction of the cover sets described above ensures the number N t h ( Z ′ ) of observations within each cover element Z ′ remains relatively small with respect to the size of Z ′ , while also controlling the total number |S t h | of cover elements. The key parameter managing this tradeoff is α , which is carefully chosen to achieve an appropriate width for the confidence interval, as shown in Section 4.

3.2 π -KRVI

Our proposed policy, π -KRVI, is derived by adopting the precise structure of an optimistic LSVI, as described previously, where the predictor and the exploration bonus are designed based on kernel ridge regression only on cover elements. In particular, for z ∈ Z , let Z t h ( z ) ∈ S t h be the cover element at step h of episode t containing z . Define Z t h ( z ) = { ( s τ h , a τ h ) ∈ Z t h ( z ) , τ < t } to be the set of past observations belonging to the same cover element as z . We then set

where k Z t h ( z ) = [ k ( z, z ′ )] ⊤ z ′ ∈ Z t h ( z ) is the kernel values between z and all observations z ′ in Z t h ( z ) , K Z t h ( z ) = [ k ( z ′ , z ′′ )] z ′ ,z ′′ ∈ Z t h ( z ) is the kernel matrix for observations in Z t h ( z ) , and Y Z t h ( z ) =

[ r h ( z ′ )+ V t h +1 ( s ′ h +1 )] ⊤ z ′ ∈ Z t h ( z ) , where s ′ h +1 is drawn from the transition distribution P h ( ·| z ′ ) , denotes the observation values for the observation points z ′ ∈ Z t h ( z ) . The vectors k Z t h ( z ) and Y Z t h ( z ) are N t -1 h ( Z t h ( z )) dimensional column vectors, and K Z t h ( z ) and I are N t -1 h ( Z t h ( z )) × N t -1 h ( Z t h ( z )) dimensional matrices.

The exploration bonus is determined based on the uncertainty estimate of the kernel ridge regression model on cover elements defined as

The policy π -KRVI then is the greedy policy with respect to

Specifically, at step h of episode t , the following action is chosen, after observing s t h ,

A pseudocode is provided in Algorithm 1.

Algorithm 1 The π -KRVI Policy. 1: Input: λ , β T ( δ ) , k , M = ( S , A , H, P, r ) . 2: For all h ∈ [ H ] , let S 1 h = { [0 , 1] d } . 3: for Episode t = 1 , 2 , . . . , T , do 4: Receive the initial state s t 1 . 5: Set V t H +1 ( s ) = 0 , for all s . 6: for step h = H,.. . , 1 do 7: Obtain value functions Q t h ( z ) as in (16). 8: end for 9: for step h = 1 , 2 , . . . , H do 10: Take action a t h ← arg max a ∈A Q t h ( s t h , a ) . 11: Observe the reward r h ( s t h , a t h ) and the next state s t h +1 . 12: Split any element Z ′ ∈ S t -1 h , for which ρ -α Z ′ < | N t h ( Z ′ ) | +1 along the middle of each side, and obtain S t h . 13: end for 14: end for

The predictor ̂ Q t h , the confidence interval width multiplier β T ( δ ) and the exploration bonus b t h are all designed using kernel ridge regression limited to the observations within cover elements given above. The parameter β T ( δ ) , in particular, is designed in a way that Q t h is a 1 -δ upper confidence bound on r h +[ P h V t h +1 ] . Using Theorem 1 on the confidence intervals, we show that a choice of β T ( δ ) = Θ( H √ log( TH δ )) satisfies this requirement.

Figure 1 demonstrates the domain partitioning used in π -KRVI on a 2 -dimensional domain. The colors represent the value of the target function. The observation points are expected to concentrate around the areas where the target function has a high value. As a result the domain is partitioned to smaller squares in that region.

Runtime complexity. The π -KRVI policy is also runtime efficient with a polynomial runtime complexity. In particular, an upper bound on the runtime of π -KRVI is O ( HT 4 + H |A| T 3 ) , that is similar to KOVI (Yang et al., 2020a). However, analogous to (Janz et al., 2020), we expect an improved runtime for π -KRVI in practice. In addition, the runtime can further improve in terms of T , utilizing sparse approximations of kernel ridge predictor and uncertainty estimate (e.g., see, Vakili et al., 2022). The dependency of the runtime on |A| is due to the step given in Equation (17). If this optimization can be done efficiently over continuous domains, π -KRVI (also KOVI) could handle infinite number of actions. The assumption that the upper confidence bound index can be efficiently optimized over continuous domains is often made in the kernelized bandits (e.g., see, Srinivas et al., 2010a).

1.0

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

10

4 Main Results and Regret Analysis

In this section, we present our main results. In Theorem 2, we establish an ˜ O ( H 2 T d + α/ 2 d + α ) regret bound for π -KRVI, for the class of kernels with polynomial eigendecay. We first prove bounds on maximum information gain and covering number of state-action value function class. Those enable us to present our uniform confidence interval for state-action value functions (Theorem 1), and subsequently the regret bound (Theorem 2).

Definition 1 (Polynomial Eigendecay) Consider the Mercer eigenvalues { σ m } ∞ m =1 of k : Z×Z → R , given in Equation (7) , in a decreasing order, as well as the corresponding eigenfeatures { ϕ m } ∞ m =1 . Assume Z is a d -dimensional hypercube with side length ρ Z . For some C p , α > 0 , p > 1 , the kernel k is said to have a polynomial eigendecay, if for all m ∈ N , σ m ≤ C p m -p ρ α Z . In addition, for some η ≥ 0 , m -pη ϕ m ( z ) is uniformly bounded over all m and z . We use the notation ˜ p = p (1 -2 η ) .

The polynomial eigendecay profile encompasses a large class of common kernels, e.g., the Matérn family of kernels. For a Matérn kernel with smoothness parameter ν , p = 2 ν + d d and α = 2 ν (e.g., see, Janz et al., 2020). Another example is the NT kernel (Arora et al., 2019). It has been shown that the RKHS of the NT kernel, when the activations are s -1 times differentiable, is equivalent to the RKHS of a Matérn kernel with smoothness ν = s -1 2 (Vakili et al., 2021b). For instance, the RKHS of an NT kernel with ReLU activations is equivalent to the RKHS of a Matérn kernel with ν = 1 2 (also known as the Laplace kernel). In this case, p = 1 + 1 d and α = 1 . The hypercube domain assumption is a technical formality that can be relaxed to other regular compact subsets of R d . The uniform boundedness of m -pη ϕ m ( z ) for some η > 0 , also holds for a broad class of kernels, including the Matérn family, as discussed in (Yang et al., 2020a). Several works including (Vakili et al., 2021b; Kassraie and Krause, 2022), have employed an averaging technique over subsets of eigenfeatures, demonstrating that, for the bounds on information gain, the effective value of η can be considered as 0 in the case of Matérn and NT kernels.

4.1 Confidence Intervals for State-Action Value Functions

Confidence intervals are an important building block in the design and analysis of bandit and RL algorithms. For a fixed function f in the RKHS of a known kernel, 1 -δ confidence intervals of the form | f ( z ) -µ t,f ( z ) | ≤ β ( δ ) b t ( z ) are established in several works (Srinivas et al., 2010a; Chowdhury and Gopalan, 2017; Abbasi-Yadkori, 2013; Vakili et al., 2021a) under various assumptions. In our setting of interest, however, these confidence intervals cannot be directly applied. This is due to the randomness of the target function itself. Specifically, in our case, the target function is r h +[ P h V t h +1 ] , which is not a fixed function due to the temporal dependence within an episode. An argument based on the covering number of the state-action value function class was used in Yang et al. (2020a) to establish uniform confidence intervals over all z ∈ Z and all f in a specific function class. In Theorem 1, we prove a different confidence interval that offers flexibility with respect to setting the parameters of the confidence interval. Our approach leads to a more refined confidence interval,

which, with a proper choice of parameters, contributes to the improved regret bound achieved by our policy.

We first give a formal definition of the two complexity terms: maximum information gain and the covering number of the state-action value function class, which appear in our confidence intervals.

Definition 2 (Maximum Information Gain) In the kernel ridge regression setting described in Section 2.2, the following quantity is referred to as maximum information gain: Γ k,λ ( t ) = max Z t ⊂Z 1 2 log det ( I + 1 λ 2 K Z t ) .

Upper bounds on maximum information gain based on the spectrum of the kernel are established in Janz et al. (2020); Srinivas et al. (2010a); Vakili et al. (2021c). Maximum information gain is closely related to the effective dimension of the kernel. While the feature representation of common kernels is infinite dimensional, with a finite observation set, only a finite number of features have a significant impact on kernel ridge regression, that is referred to as the effective dimension. It has been shown that information gain and effective dimension are the same up to logarithmic factors (Calandriello et al., 2019). This observation offers an intuitive understanding of information gain.

State-action value function class: Let us use Q k,h ( R,B ) to denote the class of state-action value functions. In particular for a set of observations Z , let b h ( z ) be the uncertainty estimate obtained from kernel ridge regression as given in (9). We define

Definition 3 (Covering Set and Number) Consider a function class F . For ϵ > 0 , we define the minimum ϵ -covering set C ( ϵ ) as the smallest subset of F that covers it up to an ϵ error in l ∞ norm. That is to say, for all f ∈ F , there exists a g ∈ C ( ϵ ) , such that ∥ f -g ∥ l ∞ ≤ ϵ . We refer to the size of C ( ϵ ) as the ϵ -covering number.

We use the notation N k,h ( ϵ ; R,B ) to denote the ϵ -covering number of Q k,h ( R,B ) , that appears in the confidence interval.

In Lemmas 2 and 3, we establish bounds on Γ k,λ ( t ) and N k,h ( ϵ ; R,B ) , respectively.

Lemma 2 (Maximum information gain) Consider a positive definite kernel k : Z × Z → R , with polynomial eigendecay on a hypercube with side length ρ Z . The maximum information gain given in Definition 2 satisfies

Lemma 3 (Covering Number of Q k,h ( R,B ) ) Recall the class of state-action value functions Q k,h ( R,B ) , where k : Z × Z → R satisfies the polynomial eigendecay on a hypercube with side length ρ Z . We have

Our bound on maximum information gain is stronger than the ones presented in Yang et al. (2020a); Janz et al. (2020); Srinivas et al. (2010a) and is similar to the one given in Vakili et al. (2021c), in terms of dependency on T . Our bound on function class covering number is similar to the one given in Yang et al. (2020a), in terms of dependency on T . Both Lemmas 2 and 3 given in this work are, however, novel in terms of dependency on the domain size ρ Z , and are required for the analysis of our domain partitioning algorithm.

We next present the confidence interval. Proofs are given in the appendix.

Theorem 1 (Confidence Interval) Let ̂ Q t h and b t h denote the kernel ridge predictor and uncertainty estimate of r h +[ P h V t h +1 ] , using t observations { V t h +1 ( s τ h +1 ) } t τ =1 at Z t h = { z τ h } t τ =1 ⊂ Z , where s τ h +1 is the next state drawn from P h ( ·| z τ h ) . Let R T = 2 H √ Γ k,λ ( T ) . For ϵ, δ ∈ (0 , 1) , with probability, at least 1 -δ , we have, ∀ z ∈ Z , h ∈ [ H ] and t ∈ [ T ] ,

where β t h ( δ, ϵ ) is set to any value satisfying

4.2 Regret of π -KRVI

A key step in the analysis of π -KRVI is to apply the confidence interval in Theorem 1 to a subdomain Z ′ ∈ S t h . By design of the splitting rule, we can prove that the maximum information gain corresponding to Z ′ satisfies Γ k,λ ( N T h ( Z ′ )) = O (log( T )) . In addition, we choose ϵ = H √ log( TH δ ) √ N t h ( Z ′ ) , when applying the confidence interval at step h of episode t on this subdomain. That ensures log N k,h ( ϵ ; R N T h ( Z ′ ) , β t h ( δ, ϵ )) = O (log( T )) . From these, and by applying a probability union bound over all subdomains Z ′ created in π -KRVI, we can deduce that the choice of β T ( δ ) = Θ( H √ log( TH δ )) with a sufficiently large constant, satisfies the requirements for confidence interval widths based on Theorem 1. The details are provided in the proof of Theorem 2 in Appendix D. Then, using standard tools from the analysis of optimistic LSVI algorithms, we arrive at the following regret bound.

Theorem 2 (Regret of π -KRVI) Consider the π -KRVI policy described in Section 3.2, with β T ( δ ) = Θ( H √ log( TH δ )) with a sufficiently large constant implied in the Θ notation. Under Assumption 1, for kernels given in Definition 1, with probability at least 1 -δ , the regret of π -KRVI satisfies

The regret bound of π -KRVI presented in Theorem 2 is sublinear in T when α > 0 , in contrast to the state of the art regret bound in Yang et al. (2020a). The O notation used in the expression above hides constants that depend on p, α and d . See Appendix D for more details. When specialized to the Matérn family of kernels, replacing p = 2 ν + d d and α = 2 ν , the regret bound becomes

In terms of T scaling, this matches the lower bound for the special case of kernelized bandits (Scarlett et al., 2017), up to a log( T ) factor.

5 Conclusion

The analysis of RL algorithms has predominantly focused on simple settings such as tabular or linear MDPs. Several recent studies have considered more general models, including representing the state-action value functions using RKHSs. Notably, the work in Yang et al. (2020a) derives regret bounds for an optimistic LSVI policy. However, the regret bounds in Yang et al. (2020a) are sublinear only when the eigenvalues of the kernel decay rapidly. In this work, we leveraged a domain partitioning technique, a uniform confidence interval for state-action value functions, and bounds on complexity terms based on the domain size to propose π -KRVI, which attains a sublinear regret bound for a general class of kernels. Moreover, our regret bounds match the lower bound derived for Matérn kernels in the special case of kernelized bandits, up to logarithmic factors. It remains an open problem whether the suboptimal regret bounds in the case of standard optimistic LSVI policies (such as KOVI, Yang et al., 2020a) represent a fundamental shortcoming or an artifact of the proof.

Funding Disclosure

This work was funded by MediaTek Research.

References

Abbasi-Yadkori, Y. (2013). Online learning for linearly parametrized control problems. PhD Thesis, University of Alberta .

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 24.

Agrawal, S. and Goyal, N. (2013). Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR.

Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems , 32.

Auer, P., Jaksch, T., and Ortner, R. (2008). Near-optimal regret bounds for reinforcement learning. In Koller, D., Schuurmans, D., Bengio, Y., and Bottou, L., editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc.

Bartlett, P. L. and Tewari, A. (2012). REGAL: A regularization based algorithm for reinforcement learning in weakly communicating mdps. CoRR , abs/1205.2661.

Borovitskiy, V., Terenin, A., Mostowsky, P., et al. (2020). Matérn Gaussian processes on Riemannian manifolds. In Advances in Neural Information Processing Systems , volume 33, pages 1242612437.

Calandriello, D., Carratino, L., Lazaric, A., Valko, M., and Rosasco, L. (2019). Gaussian process optimization with adaptive sketching: scalable and no regret. In Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , Phoenix, USA. PMLR.

Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853. PMLR.

Chowdhury, S. R. and Gopalan, A. (2019). Online learning in kernelized Markov decision processes. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3197-3205. PMLR.

Christmann, A. and Steinwart, I. (2008). Support Vector Machines . Springer New York, NY.

Domingues, O. D., Ménard, P., Pirotta, M., Kaufmann, E., and Valko, M. (2021). Kernel-based reinforcement learning: A finite-time analysis. In International Conference on Machine Learning , pages 2783-2792. PMLR.

Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., Novikov, A., R Ruiz, F. J., Schrittwieser, J., Swirszcz, G., et al. (2022). Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53.

Hoeffding, W. (1994). Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding , pages 409-426.

Janz, D., Burt, D., and González, J. (2020). Bandit optimisation of functions in the Matérn kernel RKHS. In International Conference on Artificial Intelligence and Statistics , pages 2486-2495. PMLR.

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is Q-learning provably efficient? Advances in neural information processing systems , 31.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR.

Kahn, G., Villaflor, A., Pong, V ., Abbeel, P., and Levine, S. (2017). Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182 .

Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673. PMLR.

Kassraie, P. and Krause, A. (2022). Neural contextual bandits without regret. In International Conference on Artificial Intelligence and Statistics , pages 240-278. PMLR.

Lee, K., Kim, S.-A., Choi, J., and Lee, S.-W. (2018). Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling. In International Conference on Machine Learning, , pages 2937-2946. PMLR.

Li, Z. and Scarlett, J. (2022). Gaussian process bandit optimization with few batches. In International Conference on Artificial Intelligence and Statistics .

Mercer, J. (1909). Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 209:415-446.

Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J. W., Songhori, E., Wang, S., Lee, Y.-J., Johnson, E., Pathak, O., Nazi, A., et al. (2021). A graph placement methodology for fast chip design. Nature , 594(7862):207-212.

Neu, G. and Pike-Burke, C. (2020). A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems , 33:1392-1403.

Pozrikidis, C. (2014). An introduction to grids, graphs, and networks . Oxford University Press.

Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons.

Russo, D. (2019). Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems , 32.

Salgia, S., Vakili, S., and Zhao, Q. (2021). A domain-shrinking based Bayesian optimization algorithm with order-optimal regret performance. Conference on Neural Information Processing Systems , 34.

Scarlett, J., Bogunovic, I., and Cevher, V . (2017). Lower bounds on regret for noisy Gaussian process bandit optimization. In Conference on Learning Theory , pages 1723-1742. PMLR.

Schölkopf, B., Smola, A. J., Bach, F., et al. (2002). Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press.

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489.

Srinivas, N., Krause, A., Kakade, S., and Seeger, M. (2010a). Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML 2010 - Proceedings, 27th International Conference on Machine Learning , pages 1015-1022.

Srinivas, N., Krause, A., Kakade, S., and Seeger, M. (2010b). Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning , pages 1015-1022.

Vakili, S., Bouziani, N., Jalali, S., Bernacchia, A., and Shiu, D.-s. (2021a). Optimal order simple regret for Gaussian process bandits. Advances in Neural Information Processing Systems , 34:2120221215.

Vakili, S., Bromberg, M., Garcia, J., Shiu, D.-s., and Bernacchia, A. (2021b). Uniform generalization bounds for overparameterized neural networks. arXiv preprint arXiv:2109.06099 .

Vakili, S., Khezeli, K., and Picheny, V. (2021c). On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90. PMLR.

Vakili, S., Scarlett, J., and Javidi, T. (2021d). Open problem: Tight online confidence intervals for RKHS elements. In Conference on Learning Theory , pages 4647-4652. PMLR.

Vakili, S., Scarlett, J., Shiu, D.-s., and Bernacchia, A. (2022). Improved convergence rates for sparse approximation methods in kernel-based learning. In International Conference on Machine Learning , pages 21960-21983. PMLR.

Valko, M., Korda, N., Munos, R., Flaounas, I., and Cristianini, N. (2013). Finite-time analysis of kernelised contextual bandits. In Uncertainty in Artificial Intelligence .

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354.

Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning . MIT press Cambridge, MA.

Yang, L. and Wang, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning , pages 10746-10756. PMLR.

Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. (2020a). Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems , 33:13903-13916.

Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. I. (2020b). On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622 .

Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. I. (2020c). On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622 .

Yao, H., Szepesvári, C., Pires, B. A., and Zhang, X. (2014). Pseudo-MDPs and factored linear action models. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) , pages 1-9. IEEE.

Yeh, S.-Y., Chang, F.-C., Yueh, C.-W., Wu, P.-Y., Bernacchia, A., and Vakili, S. (2023). Sample complexity of kernel-based q-learning. In International Conference on Artificial Intelligence and Statistics , pages 453-469. PMLR.

Zanette, A., Brandfonbrener, D., Brunskill, E., Pirotta, M., and Lazaric, A. (2020a). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR.

Zanette, A., Lazaric, A., Kochenderfer, M., and Brunskill, E. (2020b). Learning near optimal policies with low inherent Bellman error. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 10978-10989. PMLR.

A Mercer Theorem and the RKHSs

Mercer theorem (Mercer, 1909) provides a representation of the kernel in terms of an infinite dimensional feature map (e.g., see, Christmann and Steinwart, 2008, Theorem 4 . 49 ). Let Z be a compact metric space and µ be a finite Borel measure on Z (we consider Lebesgue measure in a Euclidean space). Let L 2 µ ( Z ) be the set of square-integrable functions on Z with respect to µ . We further say a kernel is square-integrable if

Theorem 3 (Mercer Theorem) Let Z be a compact metric space and µ be a finite Borel measure on Z . Let k be a continuous and square-integrable kernel, inducing an integral operator T k : L 2 µ ( Z ) → L 2 µ ( Z ) defined by

where f ∈ L 2 µ ( Z ) . Then, there exists a sequence of eigenvalue-eigenfeature pairs { ( σ m , ϕ m ) } ∞ m =1 such that σ m > 0 , and T k ϕ m = σ m ϕ m , for m ≥ 1 . Moreover, the kernel function can be represented as

where the convergence of the series holds uniformly on Z × Z .

According to the Mercer representation theorem (e.g., see, Christmann and Steinwart, 2008, Theorem 4 . 51 ), the RKHS induced by k can consequently be represented in terms of { ( σ m , ϕ m ) } ∞ m =1 .

Theorem 4 (Mercer Representation Theorem) Let { ( σ m , ϕ m ) } ∞ i =1 be the Mercer eigenvalue eigenfeature pairs. Then, the RKHS of k is given by

Mercer representation theorem indicates that the scaled eigenfeatures { √ σ m ϕ m } ∞ m =1 form an orthonormal basis for H k .

B Proof of Theorem 1 (Confidence Interval)

Confidence bounds of the form given in Theorem 1 have been established for a fixed function f with bounded RKHS norm and sub-Gaussian observation noise in several works including Abbasi-Yadkori (2013); Chowdhury and Gopalan (2017); Vakili et al. (2021a). In the RL setting, however, we apply the confidence interval to f = r h +[ P h V t h +1 ] . Although the RKHS norm of this target function is bounded by H +1 , this is not a fixed function as it depends on V t h +1 . In addition the observation noise terms V h +1 ( s t h +1 ) -[ P h V t h +1 ]( s t h , a t h ) also depend on V t h +1 . To handle this setting, we prove a confidence interval that holds for all possible V t h +1 : S → [0 , H ] . For this purpose, we use a probability union bound and a covering set argument over the function class of V t h +1 .

We first recall the confidence interval for a fixed function and noise sequence given in (Chowdhury and Gopalan, 2017, Theorem 2 ). See also (Abbasi-Yadkori, 2013, Corollary 3 . 15 ).

Lemma 4 Let { z t ∈ Z} T t =1 be a stochastic process predictable with respect to the filtration {F t } T t =0 . Let { ε t } T t =1 be a real valued F t measurable stochastic process with a σ sub-Gaussian distribution conditioned on F t -1 . Let µ t,f and b t be the kernel ridge predictor and uncertainty estimate of f using t noisy observations of the form { f ( z τ ) + ε τ } t τ =1 . Assume f ∈ B k,R .Then with probability at least 1 -δ , for all z ∈ Z and t ≥ 1 ,

As discussed above, we cannot directly use this confidence interval on r h + [ P h V t h +1 ] in the RL setting. Instead, we need to prove a new confidence interval that holds true for all possible V t h +1 . We thus define V to be the function class of V t h +1 as follows.

For simplicity of presentation, we specify the parameters R and B later.

Let C V k,h ( ϵ ; R,B ) be the smallest ϵ -covering set of V k,h ( R,B ) in terms of l ∞ norm. That is to say for all V ∈ V k,h ( R,B ) , there exists some V ∈ C V k,h ( ϵ ; R,B ) such that ∥ V -V ∥ l ∞ ≤ ϵ . Let N V k,h ( ϵ ; R,B ) denote the ϵ covering number of V k,h ( R,B ) . By definition N V k,h ( ϵ ; R,B ) = |C V k,h ( ϵ ; R,B ) | .

We can create a confidence bound for all V ∈ C V k,h ( ϵ ; R,B ) , using Lemma 4 and a probability union bound over C V k,h ( ϵ ; R,B ) . Fix h ∈ [ H ] and t ∈ [ T ] . Let us use the notation ̂ Q t for the kernel ridge predictor with V . That is ̂ Q t ( z ) = k ⊤ Z t ( z )( K Z t + λ 2 I ) -1 Y , where Y ⊤ = [ V ( s τ h +1 )] t τ =1 , and s τ h +1 is the next state drawn randomly from probability distribution P h ( ·| z τ h ) . In addition, to simplify the notation, we use g = r h +[ P h V ] and µ t,g = ̂ Q t . Also, let b t ( z ) = ( k ( z, z ) -k ⊤ Z t ( z )( K Z t + λ 2 I ) -1 k Z t ( z )) 1 2 . Then, we have, with probability at least 1 -δ , for all V ∈ C V k,h ( ϵ ; R,B ) and for all z ∈ Z ,

where β 2 = H +1+ H √ 2 √ Γ k,λ ( t ) + log N V k,h ( ϵ ; R,B ) + 1 + log( 1 δ ) .

Confidence interval (24) is a direct application of Lemma 4 and using a probability union bound over all V ∈ C V k,h ( ϵ ; R,B ) . Note that, ∥ r h + P h V ∥ H k ≤ H +1 (Lemma 1). In addition, V ( s τ h +1 ) -[ P h V ]( z τ h ) ∈ [0 , H ] for all h and τ . A bounded random variable in [0 , H ] is a H/ 2 sub-Gaussian random variable based on Hoeffding inequality (Hoeffding, 1994).

Now, we extend the uniform confidence interval over all V ∈ C V k,h ( ϵ ; R,B ) to a uniform confidence interval over all V ∈ V k,h ( R,B ) . For some V ∈ V k,h ( R,B ) , define f = r h +[ P h V ] and µ t,f = ̂ Q t , similar to g and µ t,g . By definition of C V k,h ( ϵ ; R,B ) , there exists V ∈ C V k,h ( ϵ ; R,B ) , such that ∥ V -V ∥ l ∞ ≤ ϵ . Thus, for all z ∈ Z ,

Therefore, with probability at least 1 -δ ,

Next, we prove that | µ t,f ( z ) -µ t,g ( z ) | ≤ 3 ϵ √ tb t ( z ) λ .

Let us further simplify the notation by introducing α t ( z ) = ( K Z t + λ 2 I ) -1 k Z t ( z ) , F ⊤ t = [ f ( z τ h )] t τ =1 , E ⊤ t = [ ε τ = V ( s τ h +1 ) -[ P h V ]( z τ h )] t τ =1 , G ⊤ t = [ g ( z τ h )] t τ =1 , E ⊤ t = [¯ ε τ = V ( s τ h +1 ) -[ P h V ]( z τ h )] t τ =1 so that µ t,f ( z ) = α ⊤ ( z )( F t + E t ) and µ t,g ( z ) = α ⊤ ( z )( G t + E t ) .

As discussed earlier, the observation noise terms ε t also depend on V . We have, for all t ≥ 1 ,

Using the difference between f and g , as well as the difference between noise terms, we have

where the last inequality follows from ∥ α t ( z ) ∥ ≤ b t ( z ) λ (e.g., see, Vakili et al., 2021a, Proposition 1 ).

The bound on | µ t,f ( z ) -µ t,g ( z ) | combined with (26) proves that for a fixed t ∈ [ T ] , fixed h ∈ [ H ] , for all z ∈ Z and for all V ∈ V k,h ( R,B ) ,

where

The confidence interval holds uniformly for all h ∈ [ H ] and t ∈ [ T ] using a probability union bound, when β 3 is replaced with the following

To complete the proof, we bound N V k,h ( ϵ ; R,B ) in terms of the specific parameters of the problem. Firstly, the ϵ -covering number of V k,h ( R,B ) is bounded by that of Q k,h ( R,B ) (Yang et al., 2020a, proof of Lemma D. 1 ). Recall the definition of Q k,h ( R,B ) in (18). We note that ∥ ̂ Q t h ∥ H k ≤ 2 H √ Γ k,λ ( t ) (Yang et al., 2020a, Lemma C. 1 ). Thus, the theorem follows with β t h ( δ, ϵ ) , where β t h ( δ, ϵ ) is set to some value satisfying

with R t = 2 H √ Γ k,λ ( t ) . That completes the proof of Theorem 1.

C Proof of Lemmas 2 (Maximum Information Gain) and 3 (Covering Number).

We first introduce the projection of the RKHS on a lower dimensional RKHS that is used in the proof of both lemmas. We then present the proofs. Recall the Mercer theorem and the representation of kernel using Mercer eigenvalues and eigenfeatures. Using Mercer representation theorem, any f ∈ B R can be written as

with ∑ ∞ m =1 w 2 m ≤ R 2 . For some D ∈ N , let Π D [ f ] denote the projection of f onto the D -dimensional RKHS corresponding to the first D features with the largest eigenvalues

Let us use the notations w D = [ w 1 , w 2 , · · · , w D ] ⊤ for the D -dimensional column vector of weights, ϕ D ( z ) = [ ϕ 1 ( z ) , ϕ 2 ( z ) , · · · , ϕ D ( z )] ⊤ for the D -dimensional column vector of eigenfeatures, and

Σ D = diag ([ σ 1 , σ 2 , · · · , σ D ]) for the diagonal matrix of eigenvalues with [ σ 1 , σ 2 , · · · , σ D ] as the diagonal entries. We also use the notations

to denote the kernel corresponding to the D -dimensional RKHS, as well as k 0 ( z, z ′ ) = k ( z, z ′ ) -k D ( z, z ′ ) .

C.1 Proof of Lemma 2 on Maximum Information Gain

Recall the definition of Γ k,λ ( t ) . We have

We next bound the two terms on the right hand side.

Term ( i ) : Note that for k D corresponding to the D -dimensional RKHS, we have K D Z t = Φ t Σ D Φ ⊤ t , where Φ t = [ ϕ D ( z )] ⊤ z ∈ Z t is a t × D matrix that stacks the feature vectors ϕ D ( z τ ) , τ = 1 , · · · , t , as it rows. By Weinstein-Aronszajn identity (Pozrikidis, 2014) (a special case of matrix determinant lemma),

The first inequality follows from the inequality of arithmetic and geometric means on eigenvalues of the argument, and the second inequality follows from k D ≤ 1 . For clarity, we used the notations I t and I D for identity matrices of dimension t and D , respectively. Otherwise, we drop the superscript.

Term ( ii ) : Similarly using the inequality of arithmetic and geometric means on eigenvalues, we bound the log det by the log of the trace of the argument. Let us use ϵ D to denote an upper bound on k 0 .

The last inequality uses log(1 + x ) ≤ x which holds for all x ∈ R .

Combining the bounds on Term ( i ) and Term ( ii ) , we have

Now, using the polynomial eigendecay profile given in Definition 2,

The constant C 1 is the uniform bound on m -pη ϕ m , and C p is the parameter in Definition 1.

Choosing D = Ct 1 ˜ p ρ α ˜ p Z (log( t )) -1 ˜ p , with constant C = ( C 2 1 C p (˜ p -1) λ 2 ) 1 ˜ p

that completes the proof.

C.2 Proof of Lemma 3 on Covering Number of State-Action Value Function Class

Recall the definition of the state-action value function class,

and the notation N k,h ( ϵ ; R,B ) for its ϵ -covering number. Let us use the notation N k,R ( ϵ ) for the ϵ -covering number of RKHS ball B k,R = { f : ∥ f ∥ H k ≤ R } , N [0 ,B ] ( ϵ ) for the ϵ -covering number of interval [0 , B ] with respect to Euclidean distance, and N k, b ( ϵ ) for the ϵ -covering number of class of uncertainty functions b k = { b ( z ) = ( k ( z, z ) -k ⊤ Z ( z )( K Z + λ 2 I ) -1 k Z ( z ) ) 1 2 , | Z | ≤ T } .

Consider Q,Q ∈ Q k,h ( R,B ) where Q ( z ) = min { Q 0 ( z ) + βb ( z ) , H -h +1 } and Q ( z ) = min { Q 0 ( z ) + ¯ β ¯ b ( z ) , H -h +1 } . We have

That implies

For the ϵ -covering number of the [0 , B ] interval, we simply have N [0 ,B ] ( ϵ/ 3) ≤ 1+3 B/ϵ . In the next lemmas, we bound the ϵ -covering number of the RKHS ball and the class of uncertainty functions.

Lemma 5 (RKHS Covering Number) Consider a positive definite kernel k : Z × Z → R , with polynomial eigendecay on a hypercube with side length ρ Z . The ϵ -covering number of R -ball in the RKHS satisfies

Lemma 6 (Uncertainty Class Covering Number) Consider a positive definite kernel k : Z×Z → R , with polynomial eigendecay on a hypercube with side length ρ Z . The ϵ -covering number of the class of uncertainty functions satisfies

Combining (40) with Lemmas 5 and 6, we obtain

that completes the proof of Lemma 3. Next, we provide the proof of two lemmas above on the covering numbers of the RKHS ball and the uncertainty function class.

Proof 1 (Proof of Lemma 5) For f in the RKHS, recall the following representation

as well as its projection on the D -dimensional RKHS

We note that

In the expressions above, C 1 is the uniform bound on m -pη ϕ m , and C p is the constant specified in Definition 1. The third inequality follows form Cauchy-Schwarz inequality.

Now, let D 0 be the smallest D such that the right hand side is bounded by ϵ 2 . There exists a constant C 2 > 0 , only depending on constants C 1 , C p , η and ˜ p , such that

For a D -dimensional linear model, where the norm of the weights is bounded by R , the ϵ -covering is at most C 3 D (1 + log( R ϵ ) , for some constant C 3 (e.g., see, Yang et al., 2020a). Using an ϵ/ 2 covering number for the space of Π D [ f ] and using the minimum number of dimensions that ensures | f -Π D [ f ] | ≤ ϵ/ 2 , we conclude that

that completes the proof of the lemma.

Proof 2 (Proof of Lemma 6) Let us define b 2 k = { b 2 : b ∈ b k } and N k, b 2 ( ϵ ) to be its ϵ -covering number. We note that, for b, ¯ b ∈ b ,

Thus, an ϵ -covering number of b is bounded by an ϵ 2 -covering of b 2 :

We next bound N k, b 2 ( ϵ 2 ) .

Using the feature space representation of the kernel, we obtain

for some γ m ∈ [0 , 1] . Based on the GP interpretation of the model, γ m can be understood as the posterior variances of the weights. Let D 0 be the smallest D such that ∑ ∞ m = D +1 σ m ϕ 2 m ( z ) ≤ ϵ 2 / 2 . From Equation (37) , we can see that, for some constant C 4 , only depending on constants C 1 , C p , η and ˜ p ,

For ∑ D 0 m =1 γ m σ m ϕ 2 m ( z ) on a finite D 0 -dimensional spectrum, as shown in Lemma D. 3 of Yang et al. (2020a), an ϵ 2 / 2 covering number scales with D 2 0 . Specifically, an ϵ 2 / 2 covering number of the space of ∑ D 0 m =1 γ m σ m ϕ 2 m ( z ) is bounded by

Combining Equations (50) and (51) , we obtain

that completes the proof of the lemma.

D Proof of Theorem 2 (Regret of π -KRVI).

Following the standard analysis of optimisitc LSVI policies, for any h ∈ [ H ] , t ∈ [ T ] , we define temporal difference error δ t h : Z → R as

Roughly speaking, { δ t h ( z ) } H h =1 quantify how far the { Q t h } H h =1 are from satisfying the Bellman optimality equation.

For any h ∈ [ H ] , t ∈ [ T ] , we also define

Using the notation defined above, we then have the following regret decomposition into three parts.

Lemma 7 (Lemma 5 . 1 in Yang et al. (2020a) on regret decomposition) We have

The third term is negative, by definition of π t h that is the greedy policy with respect to Q t h :

for all s h ∈ S . The second term is bounded using the following lemma.

Lemma 8 (Lemma 5 . 3 in Yang et al. (2020a)) For any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Term ( i ) : It turns out that the dominant term and the challenging term to bound is the first term in Lemma 7. We next provide an upper bound on this term.

For step h , let U T h = ⋃ T t =1 S t h be the union of all cover elements used by π -KRVI over all episodes. The size of U T h is bounded in the following lemma and is useful in the analysis of Term ( i ) .

Lemma 9 (Lemma 2 in Janz et al. (2020)) The size of U T h satisfies

for some constant C 6 .

The size of U T h depends on the dimension of the domain and the parameter α used in the splitting rule in Section 3.1.

Now, consider a cover element Z ′ ∈ U T h . Using Theorem 1, we have, with probability at least 1 -δ , for all h ∈ [ H ] , t ∈ [ T ] , z ∈ Z ′ , for some ϵ t h ∈ (0 , 1) ,

where β t h ( δ, ϵ t h ) is the smallest value satisfying

with N = N T h, Z ′ and ϵ t h = H log( TH δ ) √ N T h, Z′

We also note that

where the first line is based on Lemma 2, and the second line is by the design of partitioning in π -KRVI. Recall that each hypercube is partitioned when ρ -α Z ′ < N t h, Z ′ + 1 , ensuring that N t h, Z ′ remains at most ρ -α Z ′ .

For the covering number, with the choice of ϵ t h = H √ log( TH δ ) √ N t h, Z′ , we have

We thus see that the choice of β t h ( δ, ϵ t h ) = Θ( H √ log( TH δ )) satisfies the requirement for confidence interval width on Z ′ based on Theorem 1. We now use probability union bound over all Z ′ ∈ U T h to obtain

For this value of β T ( δ ) , we have with probability at least 1 -δ , for all h ∈ [ H ] , t ∈ [ T ] , z ∈ Z ,

where in the above expression ϵ t h is the parameter of the covering number corresponding to Z ′ when z ∈ Z ′ .

Therefore, we have, with probability at least 1 -δ

with

We bound the total uncertainty in the kernel ridge regression measured by ∑ T t =1 ( b t h ( z t h )) 2

The first inequality is commonly used in kernelized bandits. For example see (Srinivas et al., 2010a, Lemma 5 . 4 ). The third and fifth lines follow from Equation (58) and Lemma 9, respectively. Also,

we have

We are now ready to bound the

The proof is completed.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies RL with kernel function approximation, specifically where it is assumed that the transition dynamics and reward function live in some RKHS. While there has been some work on RL with kernel function approximation, existing work provides bounds which are suboptimal. This work seeks to tighten these bounds, and ultimately obtains a bound scaling as $O(\\sqrt{\\Gamma(T) T})$ where $\\Gamma(T)$ is the maximum information gain, matching the lower bound in certain settings of the bandit case. Their proposed algorithm is a variant of optimistic LSVI.\n\nSTRENGTHS:\n1. The setting of kernel RL has not been studied as thoroughly as some areas of RL, and optimal bounds do not exist. This work takes a step in obtaining optimal regret, tightening the best-known existing bounds, and obtaining optimal regret in certain special cases (some bandit instances). However, I believe the stated result is incorrect—see below.\n\nWEAKNESSES:\n1. I do not believe the stated result is correct. In the setting of linear bandits/linear MDPs/linear mixture MDPs, the information gain is bounded as $\\Gamma_{k,\\lambda}(T) \\le O(d \\log T)$, which would translate to a regret guarantee scaling as $O(\\sqrt{d T})$. However, there are well-known lower bounds for these settings which scale as $\\Omega(d \\sqrt{T})$ (see e.g. [1] and [2]). Thus, the bound stated in this paper is better than the lower bound by a factor of $\\sqrt{d}$, which is impossible.\n\nThe following are less significant issues, but are also important to address:\n\n2. There is a vast body of literature on RL beyond tabular and linear function approximation which is not referenced or discussed. See references [3-6] below for a start, and the works cited therein. This literature should be cited and discussed.\n3. To make the results concrete, it would be helpful to instantiate Theorem 2 in the setting of tabular and linear MDPs.\n4. The result is only provably optimal in the setting of bandits with Matern kernels. However, this paper considers deterministic rewards, so it doesn’t actually capture the bandit setting and as such does not handle the setting for the stated lower bound. It’s difficult, then, to make the claim that this result is optimal in any setting. I would suggest modifying the setting to allow for noisy rewards, or showing that there is a reduction from the current setting to the bandit setting (by encoding reward randomness in the transitions).\n5. In addition, it would greatly strengthen the paper to show a lower bound for kernelized RL.\n6. It was not clear to me what the $\\eta$ parameter corresponded to or how it was defined. This should be clarified. \n7. A reference or proof should be given for Lemma 1. \n\n[1] Zhou, Dongruo, Quanquan Gu, and Csaba Szepesvari. \"Nearly minimax optimal reinforcement learning for linear mixture markov decision processes.\" Conference on Learning Theory. PMLR, 2021.\n\n[2] Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.\n\n[3] Du, Simon, et al. \"Bilinear classes: A structural framework for provable generalization in rl.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Jin, Chi, Qinghua Liu, and Sobhan Miryoosefi. \"Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms.\" Advances in neural information processing systems 34 (2021): 13406-13418.\n\n[5] Foster, Dylan J., et al. \"The statistical complexity of interactive decision making.\" arXiv preprint arXiv:2112.13487 (2021).\n\n[6] Zhong, Han, et al. \"A posterior sampling framework for interactive decision making.\" arXiv preprint arXiv:2211.01962 (2022).\n\nQUESTIONS:\nPlease provide further clarification on the instantiation of the results to linear bandits/MDPs (see comment above).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reinforcement learning algorithm called $\\pi$-KRVI that achieves order-optimal regret. The algorithm performs local kernelized optimistic least squares value iteration update - specifically, it partitions the state-action space so that each cell contains a small number of observations, and the Q value within a cell is updated to a Gaussian Process based Upper Confidence bound using observations in that cell only. This is motivated by KOVI [10] and $\\pi$-GP-UCB [14]. A sublinear upper bound on the regret is given - this seems to be the first sublinear bound in a general setting, and it is claimed to be order optimal in the number of episode $T$.\n\nSTRENGTHS:\nThe sublinear regret seems to be the first one established in a general setting, and this seems to be order-optimal. The paper is generally clear, but a few things could be improved as mentioned in Weaknesses. The idea seem to be sound, but I didn't read the proofs.\n\nWEAKNESSES:\n* The paper only informally refers to [19] when stating the bound is order optimal. This is not immediately clear due to differences in notations, and some ambiguity in the paper's discussion. A more precise and detailed discussion would be helpful.\n* It is not clear what ${\\cal S}$ and ${\\cal A}$ are, but it seems both are cubes? If yes, is this a necessary assumption?\n* The reward functions and transition distributions are assumed to have a norm $\\le 1$. How will the regret change if the upper bound is larger than 1?\n* Lemma 1: $V$ is not used anywhere. Is it supposed to be $V_{h+1}$? If yes, an arbitrary $V_{h+}$ can potentially have a large norm, and Eq. (11) may not hold? A proof of the lemma seems to be missing.\n* Line 233: why is the target value constructed using a new random state $s_{h+1}'$, rather than the random state $s^{t}_{h+1}$ from the current episode?\n* Line 248: Does the running time refer to the total running time of Algorithm 1? If yes, can you explain why the upper bound is independent of the size of the partitions?\n\nMinor comments\n* Kernel ridge regression is generally described as a method that doesn't provide any uncertainty estimate, while the paper describes it as a method that provides uncertainty estimates. In particular, [38] is cited as the reference, but it doesn't seem to provide such an account of kernel ridge regression. What's described as kernel ridge regression is Gaussian process regression.\n* A brief explanation on the motivation behind maximum information gain would be helpful.\n* Putting the pseudocode of the algorithm in the paper would be helpful. Alternatively, provide a more complete description of the algorithm.\n* Algorithm 1, line 10: $x_{h}^{t}$ should be $s_{h}^{t}$?\n\nQUESTIONS:\nI would appreciate a discussion on the weaknesses before minor comments.\n\nLIMITATIONS:\nA discussion on the limitation of Assumption 1 would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper presents an optimism-based online learning algorithm for RL with large state-action spaces (including continuous spaces). It proposes a (Gaussian) kernel-based function approximation + optimism (building on UCBVI) algorithm. It assumes that the reward and transition density functions are representable in  1-bal of an RKHS (with a Gaussian kernel), which is quite reasonable.  It also introduces a domain-partitioning technique  to make the kernel ridge regression part scalable. The regret bound obtained for the algorithm is shown to be an improvement over SOTA [10]. Specifically, regret scales H^2 and sublinear in T.

STRENGTHS:
Online RL algorithms for continuous state and action spaces is a really challenging problem, and until recently was unresolved. This is the best such result I have seen. It makes a very smart (and now seemingly natural) use of kernel-based function approximation.

WEAKNESSES:
The authors have not presented any numerical results. So it leaves one wondering whether it is all nice theory, and there is some hope of the making the algorithms practical.

Note: Title has a typo for Kernelized".

QUESTIONS:
1. Could you present some numerical studies so we can understand the strength of your algorithm? It could even be large tabular but difficult problems such as DeepSea and Montezuma's revenge.

LIMITATIONS:
Kernel-based method may limit scalability.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically studies the performance of a reinforcement learning algorithm under the assumption that the Q function is a member of RKHS with a known kernel. The authors provide cumulative regret bound on the iterative-least value iteration algorithm and specialise their results to the kernel with polynomial decay of eigenvalues. They improve the regret bounds of prior works by refined analysis of confidence sets in this non-parametric setting.\n\nSTRENGTHS:\nThe paper is clearly written in the context of its rather technical nature. \n\nThe work combines the newest understanding of adaptive confidence sets and their analysis for the case of Matern kernels with Linear MDPs, by providing a kernelized variant of thereof. I am not familiar with the other literature utilising the novel variants of the analysis in the context of RL. \n\nSeems like there is a sub-community interested in this issue given the COLT open problem, however, as somebody not in this community, I have to say it seems certainly a matter of taste rather than importance.\n\nWEAKNESSES:\nThe prior work, [10] indeed does not provide optimal bounds, but arguably the algorithm seems to be more practical than a rather time-varying discretization of the domain in order to facilitate the construction of the order optimal confidence sets. \n\nTo be slightly harsh I wonder if the proper academic solution would not have been informing the authors of [10] of this new technique rather than writing a wholly new paper which utilises the trick with the domain splitting which comes from other prior works e.g. [14]. At the expense of creating more elaborate confidence sets one can indeed improve the bounds, but whether this improves the performance remains unanswered in this paper, as no comparison is provided. The proposed contribution is solely of theoretical nature. \n\nIt is common in online learning to use the doubling trick and mention it in passing in case improved results are desired. I wonder if this is not a similar discretization trick, and whether this deserves an independent publication at NeurISP.\n\nQUESTIONS:\n- Mercer decomposition diagonalize the infinite dimensional operator either under a specific distribution or on a bounded domain (essentially uniform distribution), could you be more specific what you mean by the Mercer decomposition in Section 2.2 \n- Why is there H^2 in the regret bounds? What is the nature of this? One \"H\" would be more intutivie. \n- How are quantities Infogain and covering numbers related?\n- Does one also need info-gain to derive the bound? Clearly there are bound with only the info-gain, can there be only covering number bounds? I wonder if works by van der Geer on high dimensional statistics are related to this. \n- Covering numbers for Sobolev paper are well-studied objects in functional analysis, especially due to seminal paper of field’s medalist Smale. I suggest looking into ref in publications of this author.\n\nLIMITATIONS:\nNone identified.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}