SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sattar Vakili
MediaTek Research
sattar.vakili@mtkresearch.com
Julia Olkhovskaya
TU Delft*
Delft, the Netherlands
julia.olkhovskaya@gmail.com
Reinforcement learning (RL) has shown empirical success in various real world settings with complex models and large state-action spaces. The existing analytical results, however, typically focus on settings with a small number of state-action or simple models such as linearly modeled state-action value functions. To derive RL policies that efficiently handle large state-action spaces with more general value functions, some recent works have considered nonlinear function approximation using kernel ridge regression. We propose  $\pi$ -KRVI, an optimistic modification of least-squares value iteration, when the state-action value function is represented by a reproducing kernel Hilbert space (RKHS). We prove the first order-optimal regret guarantees under a general setting. Our results show a significant polynomial in the number of episodes improvement over the state of the art. In particular, with highly non-smooth kernels (such as Neural Tangent kernel or some Matérn kernels) the existing results lead to trivial (superlinear in the number of episodes) regret bounds. We show a sublinear regret bound that is order optimal in the case of Matérn kernels where a lower bound on regret is known.
Reinforcement learning (RL) in real world often has to deal with large state action spaces and complex unknown models. While RL policies using complex function approximations have been empirically effective in various fields including gaming (Silver et al., 2016; Lee et al., 2018; Vinyals et al., 2019), autonomous driving (Kahn et al., 2017), microchip design (Mirhoseini et al., 2021), robot control (Kalashnikov et al., 2018), and algorithm search (Fawzi et al., 2022), little is known about theoretical performance guarantees in such settings. The analysis of RL algorithms has predominantly focused on simpler cases such as tabular or linear Markov decision processes (MDPs). In a tabular setting, a regret bound of  $\tilde{\mathcal{O}} (\sqrt{H^3|S\times\mathcal{A}|T})$  has been shown for optimistic state-action value learning algorithms (e.g., see, Jin et al., 2018), where  $H$  is the length of episodes,  $T$  is the number of episodes, and  $S$  and  $\mathcal{A}$  are finite state and action spaces. This bound does not scale well when the size of state-action space grows large. Furthermore, when the model (the state-action value function or the transitions) admits a  $d$ -dimensional linear representation in some state-action features, a regret bound of  $\tilde{\mathcal{O}} (\sqrt{H^3d^3T})$  is established (Jin et al., 2020), that scales with the dimension of the linear model rather than the cardinality of the state-action space.
Several recent studies have explored the utilization of complex models with large state-action spaces. A very general model entails representing the state-action value function using a reproducing kernel Hilbert space (RKHS). This approach allows using kernel ridge regression to obtain confidence intervals, which facilitate the design and analysis of RL algorithms. The most significant contribution
to this general RL problem is Yang et al. (2020a), $^{2}$  that provides regret guarantees for an optimistic least-squares value iteration (LSVI) algorithm, referred to as kernel optimistic least-squares value iteration (KOVI). The main assumption is that the state-action value function can be represented using the RKHS of a known kernel  $k$ . The regret bounds reported in Yang et al. (2020a) scale as  $\tilde{\mathcal{O}}\left(H^{2}\sqrt{(\Gamma(T) + \log\mathcal{N}(\epsilon))\Gamma(T)T}\right)$ , with  $\epsilon = \frac{H}{T}$ , where  $\Gamma (T)$  and  $\mathcal{N}(\epsilon)$  are two kernel related complexity terms, respectively, referred to as maximum information gain and  $\epsilon$ -covering number of the class of state-action value functions. The definitions are given in Section 4. Both complexity terms are determined using the spectrum of the kernel. While for smooth kernels, characterized by exponentially decaying Mercer eigenvalues, such as Squared Exponential kernel,  $\Gamma (T)$  and  $\log \mathcal{N}(\frac{H^{\prime}}{T})$  are logarithmic in  $T$ , for more general kernels with greater representation capacity, these terms may grow polynomially in  $T$ , possibly making the regret bound trivial (superlinear).
To have a better understanding of the existing result, let  $\{\sigma_m > 0\}_{m=1}^{\infty}$  denote the Mercer eigenvalues of the kernel  $k$  in a decreasing order. Also, let  $\{\phi_m\}_{m=1}^{\infty}$  denote the corresponding eigenfeatures. Refer to Section 2.2 for details. The kernel  $k$  is said to have a polynomial eigendecay when  $\sigma_m$  decay at least as fast as  $m^{-p}$  for some  $p > 1$ . The polynomial eigendecay profile satisfies for many kernels of practical and theoretical interest such as Matérn family of kernels (Borovitskiy et al., 2020) and the Neural Tangent (NT) kernel (Arora et al., 2019). For a Matérn kernel with smoothness parameter  $\nu$  on a  $d$ -dimensional domain,  $p = \frac{2\nu + d}{d}$  (e.g., see, Janz et al., 2020). For a NT kernel with  $s - 1$  times differentiable activations,  $p = \frac{2s - 1 + d}{d}$  (Vakili et al., 2021b). In Yang et al. (2020a), the regret bound is specialized for the class of kernels with polynomially decaying eigenvalues, by bounding the complexity terms based on the kernel spectrum. However, the reported regret bound is sublinear in  $T$  only when the kernel eigenvalues decay very fast. In particular, let  $\tilde{p} = p(1 - 2\eta)$ , where for  $\eta \geq 0$ ,  $\sigma_m^\eta \phi_m$  is uniformly bounded. Then, Yang et al. (2020a), Corollary 4.4 reports a regret bound of  $\tilde{\mathcal{O}}(T^{\xi^* + \kappa^* + \frac{1}{2}})$ , with
$$
\kappa^ {*} = \max  \left\{\xi^ {*}, \frac {2 d + p + 1}{(d + p) (\tilde {p} - 1)}, \frac {2}{\tilde {p} - 3} \right\}, \quad \xi^ {*} = \frac {d + 1}{2 (p + d)}. \tag {1}
$$
The regret bound  $\tilde{\mathcal{O}}(T^{\xi^{*} + \kappa^{*} + \frac{1}{2}})$  is sublinear only when  $p$  and  $\tilde{p}$  are sufficiently large. That, at least, requires  $2\xi^{*} < \frac{1}{2}$ , implying  $p > d + 2$ , when  $\tilde{p}$  is also sufficiently large. For instance, for Matérn kernels, this requirement can be expressed as  $\nu > \frac{d(d + 1)}{2}$ , when  $\frac{(2\nu + d)(1 - 2\eta)}{d}$  is sufficiently large.
Special case of bandits. A similar issue existed in the simpler problem of kernelized bandits, corresponding to the special case where  $H = 1, |S| = 1$ . Specifically, the  $\tilde{\mathcal{O}}(\Gamma(T)\sqrt{T})$  regret bounds reported for optimistic sampling (Srinivas et al., 2010a, GP-UCB), as well as for Thompson sampling (Chowdhury and Gopalan, 2017, GP-TS) are also trivial (superlinear) when  $\Gamma(T)$  grows faster than  $\sqrt{T}$ . It remains an open problem whether the suboptimal performance guarantees for these two algorithms is a fundamental shortcoming or an artifact of the proof. This observation is formalized as an open problem on the online confidence intervals for RKHS elements in Vakili et al. (2021d). For the kernelized bandits problem, Scarlett et al. (2017) proved lower bounds on regret in the case of Matérn family of kernels. In particular, they proved an  $\Omega(T^{\frac{\nu + d}{2\nu + d}})$  lower bound on regret of any bandit algorithm. Several recent algorithms, different from GP-UCB and GP-TS, have been developed to alleviate the suboptimal and superlinear regret bounds in kernelized bandits and obtain an  $\tilde{\mathcal{O}}(\sqrt{\Gamma(T)}T)$  regret bound (Li and Scarlett, 2022; Salgia et al., 2021), that matches the lower bound in the case of the Matérn family of kernels, up to logarithmic factors. The Sup variations of the UCB algorithms also obtain the optimal regret bound in the contextual kernel bandit setting with finite actions (Valko et al., 2013).
Main contribution. The RL setting presents a greater level of complexity compared to the bandit setting due to the Markovian dynamics. None of the solutions in Li and Scarlett (2022); Salgia et al. (2021); Valko et al. (2013) seem appropriate in the presence of MDP dynamics, thereby leaving the question of order optimal regret bounds largely open. In this work, we leverage the scaling of the kernel spectrum with the size of the domain to improve the regret bounds. We consider kernels with polynomial eigendecay on a hypercubical domain with side length  $\rho$ , where eigenvalues scale with  $\rho^{\alpha}$  for some  $\alpha > 0$ . See Definition 1. This encompasses a large class of common kernels, including the Matérn family, for which,  $\alpha = 2\nu$ . The hypercube domain assumption is a technical formality
2Also, see the extended version on arXiv (Yang et al., 2020b).
that can be relaxed to other regular compact subsets of  $\mathbb{R}^d$ . In Section 3, we propose a domain partitioning kernel ridge regression based least-squares value iteration policy ( $\pi$ -KRVI) that achieves sublinear regret of  $\tilde{\mathcal{O}}(H^2 T^{\frac{d + \alpha / 2}{d + \alpha}})$  for kernels introduced in Definition 1. This is the first sublinear regret bound under such a general stetting. Moreover, with Matérn kernels, our regret bound matches the  $\Omega(T^{\frac{\nu + d}{2\nu + d}})$  lower bound reported in in Scarlett et al. (2017) for the special case of kernelized bandits, up to a logarithmic factor.
Our proposed policy,  $\pi$ -KRVI, is based on least-squares value iteration (similar to KOVI, Yang et al. (2020a)). However, in order to effectively utilize the confidence intervals from kernel ridge regression,  $\pi$ -KRVI creates a partitioning of the state-action domain and builds the confidence intervals only based on the observations within the same partition element. The domain partitioning allows us to leverage the scaling of the kernel eigenvalues with respect to the domain size. The inspiration for this idea is drawn from  $\pi$ -GP-UCB algorithm introduced in Janz et al. (2020) for kernelized bandits. In comparison to Janz et al. (2020),  $\pi$ -KRVI and its analysis present greater complexity due to the Markovian dynamics in the MDP setting. Furthermore, we provide a finer analysis that significantly improves the results compared to Janz et al. (2020). Although Janz et al. (2020) obtained sublinear regret guarantees of  $\tilde{\mathcal{O}}(T^{\frac{2\nu + d(2d + 3)}{4\nu + d(2d + 4)}})$  in the kernelized bandit setting with Matérn kernel, there still remained a polynomial in  $T$  gap between their regret bounds and the lower bound reported in Scarlett et al. (2017). As a consequence of our results, we also close this gap.
There are several novel contributions in our analysis that lead to the improved and order optimal regret bounds. We establish confidence intervals for kernel ridge regression that apply uniformly to all functions in the state-action value function class (Theorem 1). A similar confidence interval was given in Yang et al. (2020a). We however provide flexibility with respect to setting the parameters of the confidence interval, that eventually contributes to the improved regret bounds, with a proper choice of parameters. We also derive bounds on the maximum information gain (Lemma 2) and the function class covering number (Lemma 3), taking into consideration the size of the state-action domain. These bounds are important for the analysis of our domain partitioning policy which effectively controls the number of observations utilized in kernel ridge regression by partitioning the domain into subdomains of diminishing size. These intermediate results may also be of general interest in similar problems.
The  $\pi$ -KRVI policy enjoys an efficient runtime, polynomial in  $T$ , and linear in  $|\mathcal{A}|$ , similar to the runtime of KOVI (Yang et al., 2020a). The dependency of the runtime on  $|\mathcal{A}|$  limits the scope of the policy to finite  $\mathcal{A}$ , while allowing a continuous  $S$  (with  $|S|$  infinite). The assumption of finite  $\mathcal{A}$  can be relaxed, provided there is an efficient optimizer of a certain state-action value function. See the details in Section 3.2.
Other related work. There is an extensive literature on the analysis of RL policies which do not rely on a generative model or an exploratory behavioral policy. The literature has primarily focused on the tabular setting (Jin et al., 2018; Auer et al., 2008; Bartlett and Tewari, 2012). The domain of potential applications for this setting is very limited, as in many real world problems, the state-action space is very large or even infinite. In response to this, recent literature has placed a notable emphasis on employing function approximation in RL, particularly within the context of generalized linear settings. This approach involves representing the value function or transition model through a linear transformation to a well-defined feature mapping. Important contributions include the work of Jin et al. (2020); Yao et al. (2014), as well as subsequent studies by Russo (2019); Zanette et al. (2020a,b); Neu and Pike-Burke (2020); Yang and Wang (2020). Furthermore, there have been several efforts to extend these techniques to a kernelized setting, as explored in Yang et al. (2020a); Yang and Wang (2020); Chowdhury and Gopalan (2019); Yang et al. (2020c); Domingues et al. (2021). These works are also inspired by methods originally designed for linear bandits (Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013), as well as kernelized bandits (Srinivas et al., 2010b; Valko et al., 2013; Chowdhury and Gopalan, 2017). However, all known regret bounds in the RL setting (Yang et al., 2020a; Yang and Wang, 2020; Chowdhury and Gopalan, 2019; Yang et al., 2020c; Domingues et al., 2021) are not order optimal. We compare our regret bounds with the state of the art reported in Yang et al. (2020a). A similar issue existed for classic kernelized bandit algorithms. A detailed discussion can be found in Vakili et al. (2021d). The authors in Yang and Wang (2020) considered finite state-actions under a kernelized MDP model where the transition model can be directly estimated. That is different from the setting considered in our work and Yang et al. (2020a).
In this section, we overview the background on episodic MDPs and kernel ridge regression.
2.1 Episodic Markov Decision Processes
An episodic MDP can be described by the tuple  $M = (\mathcal{S},\mathcal{A},H,P,r)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space, the integer  $H$  is the length of each episode,  $r = \{r_h\}_{h=1}^H$  are the reward functions and  $P = \{P_h\}_{h=1}^H$  are the transition probability distributions. We use the notation  $\mathcal{Z} = \mathcal{S} \times \mathcal{A}$  to denote the state-action space. For each  $h \in [H]$ , the reward  $r_h: \mathcal{Z} \to [0,1]$  is the reward function at step  $h$ , which is supposed to be deterministic for simplicity, and  $P_h(\cdot|s,a)$  is the transition probability distribution on  $\mathcal{S}$  for the next state from state-action pair  $(s,a)$ . The choice of deterministic rewards allows us to concentrate on the core complexities of the problem, and should not be regarded as a limitation. Both the framework and results can be readily extended to a setting with random rewards.
A policy  $\pi = \{\pi_h\}_{h=1}^H$ , at each step  $h$ , determines the (possibly random) action  $\pi_h: \mathcal{S} \to \mathcal{A}$  taken by the agent at state  $s$ . At the beginning of each episode  $t = 1,2,\dots$ , the environment picks an arbitrary state  $s_1^t$ . The agent determines a policy  $\pi^t = \{\pi_h^t\}_{h=1}^H$ . Then, at each step  $h \in [H]$ , the agent observes the state  $s_h^t \in S$ , picks an action  $a_h^t = \pi_h^t(s_h^t)$  and observes the reward  $r_h(s_h^t, a_h^t)$ . The new state  $s_{h+1}^t$  then is drawn from the transition distribution  $P_h(\cdot | s_h^t, a_h^t)$ . The episode ends when the agent receives the final reward  $r_H(s_H^t, a_H^t)$ .
The goal is to find a policy  $\pi$  that maximizes the expected total reward in the episode, starting at step  $h$ , i.e., the value function defined as
$$
V _ {h} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s \right], \quad \forall s \in \mathcal {S}, h \in [ H ], \tag {2}
$$
where the expectation is taken with respect to the randomness in the trajectory  $\{(s_h,a_h)\}_{h = 1}^H$  obtained by the policy  $\pi$ . It can be shown that under mild assumptions (e.g., continuity of  $P_{h}$ , compactness of  $\mathcal{Z}$ , and boundedness of  $r$ ) there exists an optimal policy  $\pi^{\star}$  which attains the maximum possible value of  $V_{h}^{\pi}(s)$  at every step and at every state (e.g., see, Puterman, 2014). We use the notation  $V_{h}^{\star}(s) = \max_{\pi} V_{h}^{\pi}(s)$ ,  $\forall s \in S, h \in [H]$ . By definition  $V_{h}^{\pi^{\star}} = V_{h}^{\star}$ . For a value function  $V: S \to [0, H]$ , we define the following notation
$$
\left[ P _ {h} V \right] (s, a) := \mathbb {E} _ {s ^ {\prime} \sim P _ {h} (\cdot | s, a)} \left[ V \left(s ^ {\prime}\right) \right]. \tag {3}
$$
We also define the state-action value function  $Q_h^\pi : \mathcal{Z} \to [0, H]$  as follows.
$$
Q _ {h} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, a _ {h} = a \right], \tag {4}
$$
where the expectation is taken with respect to the randomness in the trajectory  $\{(s_h,a_h)\}_{h = 1}^H$  obtained by the policy  $\pi$ . The Bellman equation associated with a policy  $\pi$  then is represented as
$$
Q _ {h} ^ {\pi} (s, a) = r _ {h} (s, a) + \left[ P _ {h} V _ {h + 1} ^ {\pi} \right] (s, a), \quad V _ {h} ^ {\pi} (s) = \mathbb {E} _ {\pi} \left[ Q _ {h} ^ {\pi} (s, \pi_ {h} (s)) \right], \quad V _ {H + 1} ^ {\pi} := 0, \tag {5}
$$
where the expectation is taken with respect to the randomness in the policy  $\pi$ . The Bellman optimality equation is also given as  $Q_h^\star(s, a) = r_h(s, a) + [P_h V_{h+1}^\star](s, a)$ ,  $V_h^\star(s) = \max_a Q_h^\star(s, a)$ ,  $V_{H+1}^\star \coloneqq 0$ . The performance of a policy  $\pi^t$  is measured in terms of the loss in the value function, referred to as regret, denoted by  $\mathcal{R}(T)$  in the following definition
$$
\mathcal {R} (T) = \sum_ {t = 1} ^ {T} \left(V _ {1} ^ {\star} \left(s _ {1} ^ {t}\right) - V _ {1} ^ {\pi^ {t}} \left(s _ {1} ^ {t}\right)\right). \tag {6}
$$
Recall that  $\pi^t$  is the policy executed by the agent at episode  $t$ , where  $s_1^t$  is the initial state in that episode determined by the environment.
2We intentionally do note use the standard term transition kernel for  $P_{h}$  , to avoid confusion with the term kernel in kernel-based learning.
2.2 Kernel Ridge Regression
We assume that the state-action value functions belong to a known reproducing kernel Hilbert space (RKHS). See Assumption 1 and Lemma 1 for the formal statement. This is a very general assumption, considering that the RKHS of common kernels can approximate almost all continuous functions on the compact subsets of  $\mathbb{R}^d$  (Srinivas et al., 2010a). Consider a positive definite kernel  $k: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$ . Let  $\mathcal{H}_k$  be the RKHS induced by  $k$ , where  $\mathcal{H}_k$  contains a family of functions defined on  $\mathcal{Z}$ . Let  $\langle \cdot, \cdot \rangle_{\mathcal{H}_k}: \mathcal{H}_k \times \mathcal{H}_k \to \mathbb{R}$  and  $\| \cdot \|_{\mathcal{H}_k}: \mathcal{H}_k \to \mathbb{R}$  denote the inner product and the norm of  $\mathcal{H}_k$ , respectively. The reproducing property implies that for all  $f \in \mathcal{H}_k$ , and  $z \in \mathcal{Z}$ ,  $\langle f, K(\cdot, z) \rangle_{\mathcal{H}_k} = f(z)$ . Without loss of generality, we assume  $k(z, z) \leq 1$  for all  $z$ . Mercer theorem implies, under certain mild conditions,  $k$  can be represented using an infinite dimensional feature map:
$$
k \left(z, z ^ {\prime}\right) = \sum_ {m = 1} ^ {\infty} \sigma_ {m} \phi_ {m} (z) \phi_ {m} \left(z ^ {\prime}\right), \tag {7}
$$
where  $\sigma_{m} > 0$ , and  $\sqrt{\sigma_m}\phi_m\in \mathcal{H}_k$  form an orthonormal basis of  $\mathcal{H}_k$ . In particular, any  $f\in \mathcal{H}_k$  can be represented using this basis and wights  $w_{m}\in \mathbb{R}$  as
$$
f = \sum_ {m = 1} ^ {\infty} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m}, \tag {8}
$$
where  $\| f\|_{\mathcal{H}_k}^2 = \sum_{m = 1}^\infty w_m^2$ . A formal statement and the details are provided in Appendix A. We refer to  $\sigma_{m}$  and  $\phi_{m}$  as (Mercer) eigenvalues and eigenfeatures of  $k$ , respectively.
Kernel-based models provide powerful predictors and uncertainty estimators which can be leveraged to guide the RL algorithm. In particular, consider a fixed unknown function  $f \in \mathcal{H}_k$ . Consider a set  $Z^t = \{z^i\}_{i=1}^t \subset \mathcal{Z}$  of  $t$  inputs. Assume  $t$  noisy observations  $\{Y(z^i) = f(z^i) + \varepsilon_i^t\}_{i=1}^t$  are provided, where  $\varepsilon^i$  are independent zero mean noise terms. Kernel ridge regression provides the following predictor and uncertainty estimate, respectively (see, e.g., Schölkopf et al., 2002),
$$
\mu^ {t, f} (z) = k _ {Z ^ {t}} ^ {\top} (z) \left(K _ {Z ^ {t}} + \lambda^ {2} I ^ {t}\right) ^ {- 1} Y _ {Z ^ {t}},
$$
$$
(b ^ {t} (z)) ^ {2} = k (z, z) - k _ {Z ^ {t}} ^ {\top} (z) \left(K _ {Z ^ {t}} + \lambda^ {2} I\right) ^ {- 1} k _ {Z ^ {t}} (z), \tag {9}
$$
where  $k_{Z^t}(z) = [k(z,z^1),\dots ,k(z,z^t)]^\top$  is a  $t\times 1$  vector of the kernel values between  $z$  and observations,  $K_{Z^{t}} = [k(z^{i},z^{j})]_{i,j = 1}^{t}$  is the  $t\times t$  kernel matrix,  $Y_{Z^t} = [Y(z^1),\ldots ,Y(Z^t)]^\top$  is the  $t\times 1$  observation vector,  $I$  is the identity matrix of dimensions  $t$ , and  $\lambda >0$  is a free regularization parameter. The predictor and uncertainty estimate could be interpreted as posterior mean and variance of a surrogate centered Gaussian process (GP) model with covariance  $k$ , and zero mean Gaussian noise with variance  $\lambda^2$  (e.g., see, Williams and Rasmussen, 2006).
2.3 Technical Assumption
We assume that the reward functions  $\{r_h\}_{h = 1}^H$  and the transition probability distributions  $P_{h}(s^{\prime}|\cdot ,\cdot)$  belong to the 1-ball of the RKHS. We use the notation  $\mathcal{B}_{k,R} = \{f:\| f\|_{\mathcal{H}_k}\leq R\}$  to denote the  $R$ -ball of the RKHS.
Assumption 1 We assume
$$
r _ {h} (\cdot , \cdot), P _ {h} \left(s ^ {\prime} | \cdot , \cdot\right) \in \mathcal {B} _ {k, 1}, \quad \forall h \in [ H ], \forall s ^ {\prime} \in \mathcal {S}. \tag {10}
$$
This is a mild assumption considering the generality of RKHSs, that is also supposed to hold in Yang et al. (2020a). Similar assumptions are made in linear MDPs which are significantly more restrictive (e.g., see, Jin et al., 2020).
An immediate consequence of Assumption 1 is that for any integrable  $V: \mathcal{S} \to [0, H]$ ,  $r_h + [P_h V_{h+1}] \in \mathcal{B}_{k, H+1}$ . This is formalized in the following lemma.
Lemma 1 Consider any integrable  $V: \mathcal{S} \to [0, H]$ . Under Assumption 1, we have
$$
r _ {h} + \left[ P _ {h} V \right] \in \mathcal {B} _ {k, H + 1}. \tag {11}
$$
See (Yeh et al., 2023, Lemma 3) for a proof.
3 Domain Partitioning Least-Squares Value Iteration Policy
A standard policy in episodic MDPs is the least-squares value iteration (LSVI), which computes an estimate  $\widehat{Q}_h^t$  for  $Q_h^\star$  at each step  $h$  of episode  $t$ , by recursively applying Bellman equation as discussed in the previous section. In addition, an exploration bonus term  $b_h^t: \mathcal{Z} \to \mathbb{R}$  is typically added leading to
$$
Q _ {h} ^ {t} = \min  \left\{\widehat {Q} _ {h} ^ {t} + \beta b _ {h} ^ {t}, H - h + 1 \right\}. \tag {12}
$$