SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 4. The derived regret bound can be extended to a larger class of MDPs which consist of transient states in addition to the single irreducible class. Specifically, for any  $\theta_{1},\theta_{2}\in \Theta$ , the Markov process with transition kernel  $P_{\theta_1}^{\pi_{\theta_2}^*}$  obtained from the MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta_1})$  by following policy  $\pi_{\theta_2}^*$  has a single irreducible class  $I_{\theta_1,\theta_2}$  and a set of transient states  $T_{\theta_1,\theta_2}$ . Furthermore, Assumptions 3 and 4 hold for the single irreducible class. The reasoning behind the proof remains true in this case using the following argument: each episode  $k$  starts at  $0^{d}$  which is in the irreducible set for the chosen policy  $\pi_{\theta_k}^*$ , hence, throughout the episode the algorithm remains in the irreducible set that is positive recurrent and never visits any transient states. In other words, episodes starting and ending at  $0^{d}$  with a fixed episode dependent policy implies that reachable set of  $0^{d}$  is all that can be explored, which is positive recurrent by our assumptions. As a result, we can restrict our proof derivations to the subset that is reachable from  $0^{d}$  in each episode and follow the same analysis. The Lyapunov function based bounds apply to the positive recurrent states, and hence, restricting attention to states reachable from  $0^{d}$  within each episode, we can use these bounds for our assessment of regret using norms of the state. Thereafter, the coarse bounds on the norms of the state can be applied as carried out in our proof.
Remark 5. By problem-dependent parameters, we refer to the parameters that characterize the complexity or size of the model class  $\Theta$ . These parameters are not just a function of the size of the state-space and diameter of the MDP (as mentioned in the literature on finite-size problems[5, 23, 49]), as stability needs to be accounted for in the countable state-space setting. The dependence is, thus, more complex and requires the inclusion of stability parameters, such as Lyapunov functions, petite sets, and ergodicity coefficients that are discussed in Assumptions 1-4.
Remark 6. In the subsequent sections, several equalities and inequalities in the proofs are between random variables and hold almost surely (a.s.). Throughout the remainder, we will omit the explicit mention of a.s., but any such statement should be interpreted in this context.
Proof. Let  $\{\alpha_{i}\}_{i\geq 0}$  be the sequence of hitting times of state  $0^{d}$  starting from  $0^{d}$  (set  $\alpha_0 = 0$ ). Define  $\tau_{0^d}^{(i)}$  as the length of the  $i$ -th recurrence time of state  $0^{d}$  for  $i\in \mathbb{N}$ , i.e.,  $\tau_{0^d}^{(i)} = \alpha_i - \alpha_{i-1}$ . For simplicity, we take  $\tau_{0^d} = \tau_{0^d}^{(1)}$ . Each such recurrence time is generated using policy  $\pi_{\theta_i}^*$  that is determined using the algorithm in operation in an MDP governed by parameter  $\pmb{\theta}^*$ . Furthermore,  $\{\tau_{0^d}^{(i)}\}_{i\in \mathbb{N}}$  are independent with length at least 1, but they need not be identically distributed. The time  $T$  can be in the middle of one of these recurrence times, hence the current recurrence interval count is  $N(T) = \inf \{n:\sum_{i=1}^{n}\tau_{0^d}^{(i)}\geq T\}$ . Note that the lower bound of 1 on every  $\tau_{0^d}^{(i)}$  says that  $N(T)\leq T$  a.s. Further, from the skip-free to the right property, the most any component of state can increase in during recurrence time  $\tau_{0^d}^{(i)}$  is  $h\tau_{0^d}^{(i)}$ . Hence, the most any component of the state (and also the  $\| \cdot \|_{\infty}$  norm of the state) can increase is given by  $h\max_{i=1,\dots,T}\tau_{0^d}^{(i)}$  where the random variables
are independent with geometrically decaying tails with a worst case rate of
$$
\sup _ {\theta_ {1}, \theta_ {2} \in \Theta} \tilde {\gamma} _ {\theta_ {1}, \theta_ {2}} ^ {g} = 1 - \left(\sup _ {\theta_ {1}, \theta_ {2} \in \Theta} \tilde {b} _ {\theta_ {1}, \theta_ {2}} ^ {g}\right) ^ {- 1};
$$
see Lemma 11. From Lemma 10, we have
$$
\begin{array}{l} \tilde {b} _ {\theta_ {1}, \theta_ {2}} ^ {g} = \frac {3 b _ {\theta_ {1} , \theta_ {2}} ^ {g} + 1}{1 - \gamma_ {\theta_ {1} , \theta_ {2}} ^ {g}} \Big (\left| C _ {\theta_ {1}, \theta_ {2}} ^ {g} \right| ^ {2} \max \left(1, \max _ {\boldsymbol {u} \in C _ {\theta_ {1}, \theta_ {2}} ^ {g} \setminus \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} ^ {\pi_ {\theta_ {2}} ^ {*}} [ \tau_ {0 ^ {d}} ]\right) \Big) \\ \leq \frac{3b_{*}^{g} + 1}{1 - \gamma_{*}^{g}}\Bigg(|C_{*}^{g}|^{2}\max \left(1,\sup_{\substack{\boldsymbol {u}\in C_{*}^{g}\setminus \{0^{d}\} \\ \theta_{1},\theta_{2}\in \Theta}}\phi_{\theta_{1},\theta_{2}}^{p}(1)\Big(V_{\theta_{1},\theta_{2}}^{p}(\boldsymbol {u}) + b_{\theta_{1},\theta_{2}}^{p}\alpha_{C_{\theta_{1},\theta_{2}}^{p}}\Big)\right)\Bigg) \\ \leq \frac{3b_{*}^{g} + 1}{1 - \gamma_{*}^{g}}\Bigg(|C_{*}^{g}|^{2}\max \Big(1,\sup_{\substack{\boldsymbol {u}\in C_{*}^{g}\setminus \{0^{d}\} \\ \theta_{1},\theta_{2}\in \Theta}}\frac{1}{\beta_{\theta_{1},\theta_{2}}^{p}}\Big(s_{\theta_{1},\theta_{2}}^{p}\| \boldsymbol {u}\|_{\infty}^{r_{\theta_{1},\theta_{2}}^{p}} + \frac{b_{\theta_{1},\theta_{2}}^{p}}{\min_{\boldsymbol{y}\in C_{\theta_{1},\theta_{2}}^{p}}K_{\theta_{1},\theta_{2}}(\boldsymbol {y})}\Big)\Bigg) \\ \leq \frac {3 b _ {*} ^ {g} + 1}{1 - \gamma_ {*} ^ {g}} \left(| C _ {*} ^ {g} | ^ {2} \max  \left(1, \sup  _ {\boldsymbol {u} \in C _ {*} ^ {g} \backslash \{0 ^ {d} \}} \frac {1}{\beta_ {*} ^ {p}} \left(s _ {*} ^ {p} \| \boldsymbol {u} \| _ {\infty} ^ {r _ {*} ^ {p}} + \frac {b _ {*} ^ {p}}{K _ {*}}\right)\right)\right) \tag {21} \\ := \tilde {b} _ {*} ^ {g}, \\ \end{array}
$$
and we define  $\tilde{\gamma}_{*}^{g} := 1 - (\tilde{b}_{*}^{g})^{-1}$ . From the definition of  $b_{\theta_1,\theta_2}^g$  in Assumption 3,  $b_{\theta_1,\theta_2}^g$  is greater than or equal to 2. Thus,  $\tilde{b}_{\theta_1,\theta_2}^g \geq 7$  and we have
$$
\sup _ {\theta_ {1}, \theta_ {2} \in \Theta} c _ {\theta_ {1}, \theta_ {2}} ^ {g} = \sup _ {\theta_ {1}, \theta_ {2} \in \Theta} \frac {b _ {\theta_ {1} , \theta_ {2}} ^ {g} (\tilde {b} _ {\theta_ {1} , \theta_ {2}} ^ {g}) ^ {2}}{\tilde {b} _ {\theta_ {1} , \theta_ {2}} ^ {g} - 1} \leq \frac {b _ {*} ^ {g} (\tilde {b} _ {*} ^ {g}) ^ {2}}{6} := c _ {*} ^ {g},
$$
and as a result of Lemma 11,
$$
\mathbb {P} _ {0 ^ {d}} \left(\tau_ {0 ^ {d}} ^ {(i)} > n\right) \leq c _ {*} ^ {g} \left(\gamma_ {*} ^ {g}\right) ^ {n}, \quad 1 \leq i \leq T. \tag {22}
$$
We upper bound  $\mathbb{E}\left[M_{\pmb{\theta}^*}^T\right]$  using the independence of  $\{\tau_{0^d}^{(i)}\}_{i\in \mathbb{N}}$  and the above equation,
$$
\begin{array}{l} \mathbb {E} \left[ M _ {\boldsymbol {\theta} ^ {*}} ^ {T} \right] \leq h \mathbb {E} \left[ \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \right] = h \sum_ {n = 0} ^ {\infty} \mathbb {P} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} > n\right) \\ = h \sum_ {n = 0} ^ {\infty} \left(1 - \mathbb {P} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \leq n\right)\right) = h \sum_ {n = 0} ^ {\infty} \left(1 - \prod_ {i = 1} ^ {T} \mathbb {P} \left(\tau_ {0 ^ {d}} ^ {(i)} \leq n\right)\right) \\ \leq h n _ {0} + h \sum_ {n = n _ {0}} ^ {\infty} 1 - \left(1 - c _ {*} ^ {g} \left(\gamma_ {*} ^ {g}\right) ^ {n _ {0}} \left(\gamma_ {*} ^ {g}\right) ^ {n - n _ {0}}\right) ^ {T} \\ \leq h (n _ {0} + 1) + h \sum_ {n = n _ {0} + 1} ^ {\infty} 1 - \left(1 - \left(\gamma_ {*} ^ {g}\right) ^ {n - n _ {0}}\right) ^ {T}, \\ \end{array}
$$
where  $n_0$  is the smallest  $n \geq 0$  such that  $c_*^g (\gamma_*^g)^n < 1$ . By Reimann sum approximation, we get
$$
\begin{array}{l} \mathbb {E} \left[ M _ {\boldsymbol {\theta} ^ {*}} ^ {T} \right] \leq h (n _ {0} + 1) + h \sum_ {n = 1} ^ {\infty} 1 - \left(1 - \left(\gamma_ {*} ^ {g}\right) ^ {n}\right) ^ {T} \\ <   h \left(n _ {0} + 1\right) + h \int_ {0} ^ {\infty} 1 - \left(1 - \left(\gamma_ {*} ^ {g}\right) ^ {u}\right) ^ {T} d u \\ = h \left(n _ {0} + 1\right) + \frac {h}{\log \gamma_ {*} ^ {g}} \int_ {0} ^ {1} \frac {1 - u ^ {T}}{1 - u} d u \\ \leq h \left(n _ {0} + 1\right) + \frac {h}{\log \gamma_ {\ast} ^ {g}} (\log T + 1), \\ \end{array}
$$
where the last inequality follows from  $\sum_{n=1}^{T} n^{-1} \leq \log T + 1$  and thus  $\mathbb{E}\left[M_{\boldsymbol{\theta}^*}^T\right]$  is  $O(h \log T)$ . We now extend the result to moments of order greater than one. From (22), for  $1 \leq i \leq T$ ,
$$
\mathbb {P} _ {0 ^ {d}} (\tau_ {0 ^ {d}} ^ {(i)} > n) \leq c _ {*} ^ {g} (\gamma_ {*} ^ {g}) ^ {n} = c _ {*} ^ {g} (\gamma_ {*} ^ {g}) ^ {n _ {0}} (\gamma_ {*} ^ {g}) ^ {n - n _ {0}} <   (\gamma_ {*} ^ {g}) ^ {n - n _ {0}}.
$$
For  $n \geq n_0$ , let  $t = n - n_0 \geq 0$  and  $Y_i = \max(\tau_{0^d}^{(i)} - n_0, 0)$  to get
$$
\mathbb {P} _ {0 ^ {d}} (Y _ {i} > t) = \mathbb {P} _ {0 ^ {d}} (\tau_ {0 ^ {d}} ^ {(i)} - n _ {0} > t) <   (\gamma_ {*} ^ {g}) ^ {t},
$$
which means random variables  $\{Y_i\}_{i=1}^T$  are stochastically dominated by independent and identically distributed geometric random variables with parameter  $1 - \gamma_*^g$ . Furthermore, [56] argues that the  $p$ -th moment of the maximum of  $T$  independent and identically distributed geometric random variables is  $O(\log^p T)$ . Thus, the  $p$ -th moment of  $\max_{1 \leq i \leq T} Y_i$  is  $O(\log^p T)$  and
$$
\begin{array}{l} \max _ {1 \leq i \leq T} Y _ {i} = \max (\tau_ {0 ^ {d}} ^ {(1)} - n _ {0}, \ldots , \tau_ {0 ^ {d}} ^ {(T)} - n _ {0}, 0) = \max (\tau_ {0 ^ {d}} ^ {(1)}, \ldots , \tau_ {0 ^ {d}} ^ {(T)}, n _ {0}) - n _ {0} \\ \geq \max  (\tau_ {0 ^ {d}} ^ {(1)}, \dots , \tau_ {0 ^ {d}} ^ {(T)}) - n _ {0} \geq h ^ {- 1} M _ {\boldsymbol {\theta} ^ {*}} ^ {T} - n _ {0}, \\ \end{array}
$$
$$
\mathbb {E} \left[ \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {p} \right] \leq h ^ {p} \mathbb {E} \left[ \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) ^ {p} \right] \leq h ^ {p} \mathbb {E} \left[ \left(\max  _ {1 \leq i \leq T} Y _ {i} + n _ {0}\right) ^ {p} \right].
$$
Since the right-hand side of the above equation is  $O(h^{p}\log^{p}T)$ , the claim is proved.
B.2 Proof of Lemma 7
Proof. Let  $K_M(\pmb{x}, a)$  be the number of episodes  $k$  such that  $1 \leq k \leq K_T$  and in which the number of visits to the state-action pair  $(\pmb{x}, a)$  is increased more than twice at episode  $k$ , or
$$
K _ {M} (\boldsymbol {x}, a) = | \left\{k \leq K _ {T}: N _ {\tilde {t} _ {k + 1}} (\boldsymbol {x}, a) > 2 N _ {t _ {k}} (\boldsymbol {x}, a) \right\} |.
$$
As for every episode in the above set the number of visits to  $(x, a)$  doubles,
$$
K _ {M} (\boldsymbol {x}, a) \leq \log_ {2} \left(N _ {T + 1} (\boldsymbol {x}, a)\right) + 1,
$$
and we can upper bound  $K_{M}$  as follows
$$
\begin{array}{l} K_{M} = \sum_{\boldsymbol {x}\in \mathcal{X},a\in \mathcal{A}}K_{M}(\boldsymbol {x},a) = \sum_{\substack{\| \boldsymbol {x}\|_{\infty}\leq M^{T}_{\boldsymbol{\theta}^{*}}\\ a\in \mathcal{A}}}K_{M}(\boldsymbol {x},a) \\ \leq \sum_{\substack{\| \boldsymbol {x}\|_{\infty}\leq M^{T}_{\boldsymbol{\theta}^{*}}\\ a\in \mathcal{A}}}\left(1 + \log_{2}N_{T + 1}(\boldsymbol {x},a)\right)\leq \left|\mathcal{A}\right|\left(M^{T}_{\boldsymbol{\theta}^{*}} + 1\right)^{d}\left(1 + \log_{2}T\right). \\ \end{array}
$$
B.3 Proof of Lemma 8
Proof. We define macro episodes with start times  $t_{n_k}$ ,  $k = 1,2,\ldots ,K_M + 1$  where  $t_{n_1} = t_1$ ,  $t_{n_{K_M + 1}} = T + 1$  (which is equivalent to  $n_{K_M + 1} = K_T + 1$ ), and for  $1 < k < K_M + 1$
$$
t _ {n _ {k + 1}} = \min  \left\{t _ {j} > t _ {n _ {k}}: N _ {t _ {j}} (\boldsymbol {x}, a) > 2 N _ {t _ {j - 1}} (\boldsymbol {x}, a) \text {f o r s o m e} (\boldsymbol {x}, a) \right\},
$$
which are episodes wherein the second stopping criterion is triggered. Any episode (except for the last episode) in a macro episode must be triggered by the first stopping criterion; equivalently,  $\tilde{T}_j = \tilde{T}_{j - 1} + 1$  for all  $j = n_k,n_k + 1,\ldots ,n_{k + 1} - 2$ . For  $1\leq k\leq K_M$ , let  $T_{k}^{M} = \sum_{j = n_{k}}^{n_{k + 1} - 1}T_{j}$  be the length of the  $k$ -th macro episode. We have
$$
T _ {k} ^ {M} = \sum_ {j = n _ {k}} ^ {n _ {k + 1} - 1} T _ {j} \geq \sum_ {j = n _ {k}} ^ {n _ {k + 1} - 1} \tilde {T} _ {j} \geq 1 + \sum_ {j = n _ {k}} ^ {n _ {k + 1} - 2} (j - n _ {k} + 2) = 0. 5 (n _ {k + 1} - n _ {k}) (n _ {k + 1} - n _ {k} + 1).
$$
Consequently,  $n_{k + 1} - n_k\leq \sqrt{2T_k^M}$  for all  $1\leq k\leq K_M$  . From this, we obtain
$$
K _ {T} = n _ {K _ {M} + 1} - 1 = \sum_ {k = 1} ^ {K _ {M}} \left(n _ {k + 1} - n _ {k}\right) \leq \sum_ {k = 1} ^ {K _ {M}} \sqrt {2 T _ {k} ^ {M}}.
$$
Using the above equation and the fact that  $\sum_{k=1}^{K_M} T_k^M = T$  we get
$$
K _ {T} \leq \sum_ {k = 1} ^ {K _ {M}} \sqrt {2 T _ {k} ^ {M}} \leq \sqrt {K _ {M} \sum_ {k = 1} ^ {K _ {M}} 2 T _ {k} ^ {M}} = \sqrt {2 K _ {M} T}.
$$
Finally, from Lemma 7 we get
$$
K _ {T} \leq \sqrt {2 K _ {M} T} \leq 2 \sqrt {\left| \mathcal {A} \right| \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + 1\right) ^ {d} T \log_ {2} T}.
$$
B.4 Proof of Lemma 1
Proof. Let  $E_{k} = T_{k} - \tilde{T}_{k} \geq 0$  be the settling time needed to return to state  $0^{d}$  after a stopping criterion is realized in episode  $k$ . We have
$$
\begin{array}{l} R _ {0} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} T _ {k} J \left(\theta_ {k}\right) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right] \\ = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \tilde {T} _ {k} J \left(\theta_ {k}\right) \right] + \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} E _ {k} J \left(\theta_ {k}\right) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right]. \tag {23} \\ \end{array}
$$
We first simplify the first term in the above summation. From the monotone convergence theorem,
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \tilde {T} _ {k} J (\theta_ {k}) \right] = \sum_ {k = 1} ^ {\infty} \mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} \tilde {T} _ {k} J (\theta_ {k}) \right].
$$
Note that the first stopping criterion of Algorithm 1 ensures that  $\tilde{T}_k \leq \tilde{T}_{k-1} + 1$  at all episodes  $k \geq 1$ . Hence
$$
\mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} \tilde {T} _ {k} J (\theta_ {k}) \right] \leq \mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} (\tilde {T} _ {k - 1} + 1) J (\theta_ {k}) \right].
$$
Since  $\mathbb{I}_{\{t_k\leq T\}}(\tilde{T}_{k - 1} + 1)$  is measurable with respect to  $\mathcal{H}_{t_k}$ , by (20) we get
$$
\mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} (\tilde {T} _ {k - 1} + 1) J (\theta_ {k}) \right] = \mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} (\tilde {T} _ {k - 1} + 1) J (\boldsymbol {\theta} ^ {*}) \right].
$$
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \tilde {T} _ {k} J (\theta_ {k}) \right] \leq \sum_ {k = 1} ^ {\infty} \mathbb {E} \left[ \mathbb {I} _ {\{t _ {k} \leq T \}} (\tilde {T} _ {k - 1} + 1) J (\boldsymbol {\theta} ^ {*}) \right] = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} (\tilde {T} _ {k - 1} + 1) J (\boldsymbol {\theta} ^ {*}) \right].
$$
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \tilde {T} _ {k} J \left(\theta_ {k}\right) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right] \leq \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \sum_ {k = 1} ^ {K _ {T}} \left(\tilde {T} _ {k - 1} + 1\right) \right] - \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \sum_ {k = 1} ^ {K _ {T}} T _ {k} \right] \\ = \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \left(K _ {T} + 1 - T _ {K _ {T}} - \sum_ {k = 1} ^ {K _ {T} - 1} E _ {k}\right) \right] \\ \leq \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) K _ {T} \right]. \tag {24} \\ \end{array}
$$
For the second term in (23), from Assumption 5
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} E _ {k} J \left(\theta_ {k}\right) \right] \leq J ^ {*} \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} E _ {k} \right] \leq J ^ {*} \mathbb {E} \left[ K _ {T} \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \right]. \tag {25}
$$
Substitutinh (24) and (25) in (23), we get
$$
\begin{array}{l} R _ {0} \leq \mathbb {E} \left[ K _ {T} J (\boldsymbol {\theta} ^ {*}) \right] + J ^ {*} \mathbb {E} \left[ K _ {T} \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \right] \\ \leq J ^ {*} \mathbb {E} \left[ K _ {T} \right] + J ^ {*} \mathbb {E} \left[ K _ {T} \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \right] \\ = J ^ {*} \mathbb {E} \left[ K _ {T} \Big (\max _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} + 1 \Big) \right]. \\ \end{array}
$$
B.5 Proof of Lemma 2
Proof. We note that the state of the MDP is equal to  $0^d$  at the beginning of all episodes and the relative value function  $v(\pmb{x};\theta)$  is equal to 0 at  $\pmb{x} = 0^d$  for all  $\theta$ . Thus,
$$
\begin{array}{l} R _ {1} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v \left(\boldsymbol {X} (t); \theta_ {k}\right) - v \left(\boldsymbol {X} (t + 1); \theta_ {k}\right) \right] \right] \\ = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \left[ v \left(\boldsymbol {X} \left(t _ {k}\right); \theta_ {k}\right) - v \left(\boldsymbol {X} \left(t _ {k + 1}\right); \theta_ {k}\right) \right] \right] \\ = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T} - 1} \left[ v \left(0 ^ {d}; \theta_ {k}\right) - v \left(0 ^ {d}; \theta_ {k}\right) \right] + v \left(0 ^ {d}; \theta_ {K _ {T}}\right) - v \left(\boldsymbol {X} (T + 1); \theta_ {K _ {T}}\right) \right] \\ = - \mathbb {E} [ v (X (T + 1); \theta_ {K _ {T}}) ]. \\ \end{array}
$$
From the lower bound derived for the relative value function in (14),
$$
- v (\boldsymbol {x}; \theta) \leq J ^ {*} \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ \tau_ {0 ^ {d}} \right] \leq \frac {J ^ {*}}{\beta_ {*} ^ {p}} \left(s _ {*} ^ {p} \| \boldsymbol {x} \| _ {\infty} ^ {r _ {*} ^ {p}} + \frac {b _ {*} ^ {p}}{K _ {*}}\right),
$$
where the second inequality follows from (21) in the proof of Lemma 6. We also note that  $\| X(T + 1)\|_{\infty}\leq M_{\pmb{\theta}^{*}}^{T} + h$ . Thus,
$$
R _ {1} = - \mathbb {E} [ v (\boldsymbol {X} (T + 1); \theta_ {K _ {T}}) ] \leq \mathbb {E} \left[ \frac {J ^ {*}}{\beta_ {*} ^ {p}} \left(s _ {*} ^ {p} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {r _ {*} ^ {p}} + \frac {b _ {*} ^ {p}}{K _ {*}}\right) \right].
$$
From the inequality  $(a + b)^r \leq 2^r (a^r + b^r)$ , we have
$$
R _ {1} \leq \frac {J ^ {*} 2 ^ {r _ {*} ^ {p}} s _ {*} ^ {p}}{\beta_ {*} ^ {p}} \mathbb {E} \left[ \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r _ {*} ^ {p}} \right] + \frac {J ^ {*}}{\beta_ {*} ^ {p}} \left(s _ {*} ^ {p} (2 h) ^ {r _ {*} ^ {p}} + \frac {b _ {*} ^ {p}}{K _ {*}}\right).
$$
B.6 Proof of Lemma 3
Proof. Let  $\mathbf{Z}(t) = \left(\mathbf{X}(t), \pi_{\theta_k}^*(\mathbf{X}(t))\right)$  be the state-action pair at  $t_k \leq t < t_{k+1}$ .  $R_2$  can be upper bounded as
$$
\begin{array}{l} R _ {2} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v \left(\boldsymbol {X} (t + 1); \theta_ {k}\right) - \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} \left(\boldsymbol {y} \mid \boldsymbol {X} (t), \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {X} (t))\right) v (\boldsymbol {y}; \theta_ {k}) \right] \right] \\ \leq \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ \sum_ {\boldsymbol {y} \in \mathcal {X}} | P _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {y} | \boldsymbol {Z} (t)) - P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {Z} (t)) | | v (\boldsymbol {y}; \theta_ {k}) | \right] \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left(\max  _ {\substack {1 \leq k \leq K _ {T} \\ \| \boldsymbol {x} \| _ {\infty} \leq M _ {\boldsymbol {\theta} ^ {*}} ^ {T}}} | v (\boldsymbol {x}; \theta_ {k}) |\right) \| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {Z} (t)) - P _ {\theta_ {k}} (\cdot | \boldsymbol {Z} (t)) \| _ {1} \right]. \tag{26} \\ \end{array}
$$
$\| P_{\pmb{\theta}^{*}}(\cdot |\pmb {Z}(t)) - P_{\theta_{k}}(\cdot |\pmb {Z}(t))\|_{1}\leq \| P_{\pmb{\theta}^{*}}(\cdot |\pmb {Z}(t)) - P_{\hat{\theta}_{k}}(\cdot |\pmb {Z}(t))\|_{1} + \| P_{\theta_{k}}(\cdot |\pmb {Z}(t)) - P_{\hat{\theta}_{k}}(\cdot |\pmb {Z}(t))\|_{1},$  where  $P_{\hat{\theta}_k}(\pmb {y}|\pmb {Z}(t))$  is the empirical transition probability defined as
$$
P _ {\hat {\theta} _ {k}} (\boldsymbol {y} | \boldsymbol {Z} (t)) = \frac {N _ {t _ {k}} (\boldsymbol {Z} (t) , \boldsymbol {y})}{\max \left(1 , N _ {t _ {k}} (\boldsymbol {Z} (t))\right)},
$$
and for any tuple  $(\pmb {x},a,\pmb {y})$  , we define  $N_{1}(\pmb {x},a,\pmb {y}) = 0$  and for  $t > 1$
$N_{t}(\pmb {x},a,\pmb {y}) = |\{t_{k}\leq i <   \tilde{t}_{k + 1}\leq t\text{for some} k\geq 1:(\pmb {X}(i),A(i),\pmb {X}(i + 1)) = (\pmb {x},a,\pmb {y})\} |.$
Thus, from (26) and defining random variable  $v_{M} = \max_{1\leq k\leq K_{T}}|v(\boldsymbol {x};\theta_{k})|,$
$$
\left\| \boldsymbol {x} \right\| _ {\infty} ^ {\left. \cdot \right.} \leq M _ {\boldsymbol {\theta} ^ {*}} ^ {*}
$$
$$
R _ {2} \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ v _ {M} \| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \| _ {1} \right] + \sum_ {t = 1} ^ {T} \mathbb {E} \left[ v _ {M} \| P _ {\theta_ {k}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \| _ {1} \right]. \tag {27}
$$
We define set  $B_{k}$  as the set of parameters  $\theta$  for which the transition kernel  $P_{\theta}(\cdot |\pmb {z})$  is close to the empirical transition kernel  $P_{\hat{\theta}_k}(\cdot |\pmb {z})$  at episode  $k$  for every state-action pair  $\pmb {z} = (\pmb {x},a)\in \mathcal{X}\times \mathcal{A}$ , or
$$
B _ {k} = \left\{\theta : \| P _ {\theta} (\cdot | \boldsymbol {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {z}) \| _ {1} \leq \beta_ {k} (\boldsymbol {z}), \boldsymbol {z} = (\boldsymbol {x}, a) \in \{0, 1, \dots , h T \} ^ {d} \times \mathcal {A} \right\},
$$
where  $\beta_{k}(\pmb {z}) = \sqrt{\frac{14\prod_{i = 1}^{d}(x_{i} + h)}{\max(1,N_{t_{k}}(\pmb {z}))}\log\left(\frac{2|\mathcal{A}|T}{\tilde{\delta}}\right)}$  for  $\pmb {x} = (x_{1},\dots ,x_{d})$  and some  $0 < \tilde{\delta} < 1$ , which will be determined later. We simplify the  $\ell_1$ -difference of the real and empirical transition kernels as follows
$$
\begin{array}{l} \left\| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \right\| _ {1} \\ = \mathbb {I} _ {\left\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \right\}} \| P _ {\theta_ {*}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \| _ {1} + \mathbb {I} _ {\left\{\boldsymbol {\theta} ^ {*} \in B _ {k} \right\}} \| P _ {\theta_ {*}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \| _ {1} \\ \leq 2 \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \beta_ {k} (\boldsymbol {Z} (t)). \\ \end{array}
$$
$$
\left\| P _ {\theta_ {k}} (\cdot | \boldsymbol {Z} (t)) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {Z} (t)) \right\| _ {1} \leq 2 \mathbb {I} _ {\left\{\theta_ {k} \notin B _ {k} \right\}} + \beta_ {k} (\boldsymbol {Z} (t)).
$$
Substituting in (27), we get
$$
R _ {2} \leq \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \left[ \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \right] + \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \beta_ {k} (\boldsymbol {Z} (t)) \right]. \tag {28}
$$
We first find an upper bound for  $v_{M} = \max_{\substack{1 \leq k \leq K_{T} \\ \| \pmb{x} \|_{\infty} \leq M_{\pmb{\theta}^{*}}^{T}}} |v(\pmb{x}; \theta_{k})|$  using the bounds derived in (13) and (14). From (13),