SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} v (\boldsymbol {x}; \theta_ {k}) \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k}} ^ {*}} \left[ K d \left(\| \boldsymbol {x} \| _ {\infty} + h \tau_ {0 ^ {d}}\right) ^ {r} \tau_ {0 ^ {d}} \right] \\ \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k} ^ {*}}} \left[ 2 ^ {r} K d \left(\| \boldsymbol {x} \| _ {\infty} ^ {r} + h ^ {r} \left(\tau_ {0 ^ {d}}\right) ^ {r}\right) \tau_ {0 ^ {d}} \right] \\ = K d \left(2 \| \boldsymbol {x} \| _ {\infty}\right) ^ {r} \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k}} ^ {*}} \left[ \tau_ {0 ^ {d}} \right] + K d (2 h) ^ {r} \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k}} ^ {*}} \left[ \left(\tau_ {0 ^ {d}}\right) ^ {r + 1} \right] \\ \leq K d 2 ^ {r} \left(\| \boldsymbol {x} \| _ {\infty} ^ {r} + h ^ {r}\right) \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k}} ^ {*}} \left[ \left(\tau_ {0 ^ {d}}\right) ^ {r + 1} \right] \\ \leq K d (r + 1) 2 ^ {r} \left(\| \boldsymbol {x} \| _ {\infty} ^ {r} + h ^ {r}\right) \phi_ {\theta_ {k}} ^ {p} (r + 1) \left(V _ {\theta_ {k}} ^ {p} (\boldsymbol {x}) + b _ {\theta_ {k}} ^ {p} \alpha_ {C _ {\theta_ {k}} ^ {p}}\right) \\ \leq K d (r + 1) 2 ^ {r} \left(\| \boldsymbol {x} \| _ {\infty} ^ {r} + h ^ {r}\right) \phi_ {\theta_ {k}} ^ {p} (r + 1) \left(s _ {*} ^ {p} \| \boldsymbol {x} \| _ {\infty} ^ {r _ {*} ^ {p}} + b _ {*} ^ {p} \left(K _ {*}\right) ^ {- 1}\right), \tag {29} \\ \end{array}
$$
where the second line follows from the inequality  $(a + b)^r \leq 2^r (a^r + b^r)$ , the fifth line from Lemma 10, and the last line from Assumption 4 and (21). We further have
$$
\begin{array}{l} \phi_ {\theta_ {1}, \theta_ {2}} ^ {p} (r + 1) = \prod_ {j = 1} ^ {r + 1} \frac {1}{\beta_ {\theta_ {1} , \theta_ {2}} ^ {\eta_ {j}}} \left(2 ^ {j - 1} + (j - 1) \alpha_ {C _ {\theta_ {1}, \theta_ {2}} ^ {p}} b _ {\theta_ {1}, \theta_ {2}} ^ {\eta_ {j}}\right) \\ \leq \prod_ {j = 1} ^ {r + 1} \frac {r + 1}{\min  \left(1 , \beta_ {*} ^ {p}\right)} \left(2 ^ {j - 1} + (j - 1) \left(K _ {*}\right) ^ {- 1} b _ {\theta_ {1}, \theta_ {2}} ^ {\eta_ {j}}\right), \\ \end{array}
$$
where using the definition of  $b_{\theta_1,\theta_2}^{\eta_j}$  in (38),
$$
b _ {\theta_ {1}, \theta_ {2}} ^ {\eta_ {j}} = \left(b _ {\theta_ {1}, \theta_ {2}} ^ {p}\right) ^ {\eta_ {j}} + \eta_ {j} \tilde {\beta} _ {\theta_ {1}, \theta_ {2}} ^ {p} \max \left(1, \left(\tilde {\beta} _ {\theta_ {1}, \theta_ {2}} ^ {p}\right) ^ {(\alpha_ {\theta_ {1}, \theta_ {2}} ^ {p} + \eta_ {j} - 1) / (1 - \alpha_ {\theta_ {1}, \theta_ {2}} ^ {p})}\right) \leq 1 + b _ {*} ^ {p} + \beta_ {*} ^ {p}.
$$
We also define
$$
\phi_ {*} ^ {p} (r + 1) := \prod_ {j = 1} ^ {r + 1} \frac {r + 1}{\min  \left(1 , \beta_ {*} ^ {p}\right)} \left(2 ^ {j - 1} + (j - 1) \left(K _ {*}\right) ^ {- 1} \left(1 + b _ {*} ^ {p} + \beta_ {*} ^ {p}\right)\right).
$$
We next find a lower bound for  $v(\pmb{x}; \theta_k)$  using (14) as follows:
$$
v (\boldsymbol {x}; \theta_ {k}) \geq - J ^ {*} \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {k}} ^ {*}} \left[ \tau_ {0 ^ {d}} \right] \geq - \frac {J ^ {*}}{\beta_ {*} ^ {p}} \left(s _ {*} ^ {p} \| \boldsymbol {x} \| _ {\infty} ^ {r _ {*} ^ {p}} + \frac {b _ {*} ^ {p}}{K _ {*}}\right).
$$
Combining (29) and the above equation, we get a uniform upper bound for  $|v(\pmb{x}; \theta_k)|$  over  $\Theta$ , which we use to upper bound  $v_M = \max_{1 \leq k \leq K_T} |v(\pmb{x}; \theta_k)|$  as below
$$
\left\| \boldsymbol {x} \right\| _ {\infty} \leq M _ {\boldsymbol {\theta}} ^ {*}
$$
$$
\begin{array}{l} v _ {M} \leq \left(J ^ {*} + K d (r + 1) 2 ^ {r}\right) \phi_ {*} ^ {p} (r + 1) \left(\left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r} + h ^ {r}\right) \left(s _ {*} ^ {p} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r _ {*} ^ {p}} + b _ {*} ^ {p} \left(K _ {*}\right) ^ {- 1}\right) \\ = c _ {p _ {1}} \left(\left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r} + h ^ {r}\right) \left(s _ {*} ^ {p} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r _ {*} ^ {p}} + b _ {*} ^ {p} \left(K _ {*}\right) ^ {- 1}\right) \\ \leq c _ {p _ {2}} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r + r _ {*} ^ {p}}, \tag {30} \\ \end{array}
$$
where the constant terms are defined as
$$
c _ {p _ {1}} := \left(J ^ {*} + K d (r + 1) 2 ^ {r}\right) \phi_ {*} ^ {p} (r + 1), \quad c _ {p _ {2}} := \max  \left(1, c _ {p _ {1}} \left(h ^ {r} + 1\right) \left(s _ {*} ^ {p} + b _ {*} ^ {p} \left(K _ {*}\right) ^ {- 1}\right)\right).
$$
A deterministic upper bound on  $v_{M}$  can also be found from the above equation. Noting that from Assumption 2, until time  $T$  only states with each component less than or equal to  $hT$  are visited, we have
$$
v _ {M} \leq c _ {p _ {2}} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r + r _ {*} ^ {p}} \leq c _ {p _ {2}} (T h) ^ {r + r _ {*} ^ {p}} := Q (T),
$$
where  $Q(T)$  is a polynomial defined as above. Using the bounds derived for  $v_{M}$ , we bound  $R_{2}$  starting with the first term on the right-hand side of (28). We have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \left[ \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \right] \leq 2 Q (T) \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \\ \leq 2 T Q (T) \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \\ \leq 2 T Q (T) \sum_ {k = 1} ^ {T} \mathbb {E} \left[ \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \\ \leq 4 T Q (T) \sum_ {k = 1} ^ {T} \mathbb {P} \left\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \right\}, \tag {31} \\ \end{array}
$$
where the last inequality follows from (20) and the fact that set  $B_{k}$  is  $\mathcal{H}_{t_k}$ -measurable. To further simplify the first term in (28), we find an upper bound for  $\mathbb{P}\{\pmb{\theta}^{*} \notin B_{k}\}$  using [64]. For a fixed  $z = (x, a)$  and  $n$  independent samples of the distribution  $P_{\pmb{\theta}^*}(.|z)$ , the  $L^1$ -deviation of the true distribution  $P_{\pmb{\theta}^*}(.|z)$  and empirical distribution at the end of episode  $k$ ,  $P_{\hat{\theta}_k}(.|z)$ , is bounded in [10] as
$$
\mathbb {P} \left\{\| P _ {\pmb {\theta} ^ {*}} (\cdot | \pmb {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \pmb {z}) \| _ {1} \geq \sqrt {\frac {1 4 \prod_ {i = 1} ^ {d} (x _ {i} + h)}{n} \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \right\} \leq \frac {\tilde {\delta}}{2 0 | \mathcal {A} | T ^ {7} \prod_ {i = 1} ^ {d} (x _ {i} + h)}.
$$
$$
\mathbb {P} \left\{\| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {z}) \| _ {1} \geq \beta_ {k} (\boldsymbol {z}) \Big | N _ {t _ {k}} (\boldsymbol {z}) = n \right\} \leq \frac {\tilde {\delta}}{2 0 | \mathcal {A} | T ^ {7} \prod_ {i = 1} ^ {d} (x _ {i} + h)},
$$
$$
\begin{array}{l} \mathbb {P} \left\{\| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {z}) \| _ {1} \geq \beta_ {k} (\boldsymbol {z}) \right\} \\ = \sum_ {n = 1} ^ {T} \mathbb {P} \left\{\| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {z}) \| _ {1} \geq \beta_ {k} (\boldsymbol {z}) \mid N _ {t _ {k}} (\boldsymbol {z}) = n \right\} \mathbb {P} \left\{N _ {t _ {k}} (\boldsymbol {z}) = n \right\} \\ \leq \frac {\tilde {\delta}}{2 0 | \mathcal {A} | T ^ {6} \prod_ {i = 1} ^ {d} (x _ {i} + h)}. \\ \end{array}
$$
The probability that at episode  $k \leq T$ , the true parameter  $\theta^{*}$  does not belong to the confidence set  $B_{k}$  can be bounded using the above and union bound as
$$
\begin{array}{l} \mathbb {P} \left\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \right\} \leq \sum_ {\boldsymbol {z} \in \{0, 1, \dots , h T \} ^ {d} \times \mathcal {A}} \mathbb {P} \left\{\| P _ {\boldsymbol {\theta} ^ {*}} (\cdot | \boldsymbol {z}) - P _ {\hat {\theta} _ {k}} (\cdot | \boldsymbol {z}) \| _ {1} \geq \beta_ {k} (\boldsymbol {z}) \right\} \\ \leq \sum_ {\boldsymbol {z} \in \{0, 1, \dots , h T \} ^ {d} \times \mathcal {A}} \frac {\tilde {\delta}}{2 0 | \mathcal {A} | T ^ {6} \prod_ {i = 1} ^ {d} (x _ {i} + h)} \\ = \sum_ {\boldsymbol {x} \in \{0, 1, \dots , h T \} ^ {d}} \frac {\tilde {\delta}}{2 0 T ^ {6} \prod_ {i = 1} ^ {d} \left(x _ {i} + h\right)} \\ \leq \frac {\tilde {\delta}}{2 0 T ^ {6}} \left(\log (h (T + 1)) + 1\right) ^ {d} \\ \leq \frac {\tilde {\delta}}{2 0 k ^ {6}} \left(\log \left(h (T + 1)\right) + 1\right) ^ {d}. \\ \end{array}
$$
In the summation in the above equation, we have simplified the expression by summing over  $x_{i} \leq hT$  instead of considering the more detailed summation over  $x_{i} \leq M_{\theta^{*}}^{T}$ . However, this simplification does not affect the final evaluation of regret, as this term is not dominant and only contributes to a logarithmic term in the regret bound. Substituting in (31),
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \left[ \mathbb {I} _ {\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \}} + \mathbb {I} _ {\{\theta_ {k} \notin B _ {k} \}} \right] \right] \leq 4 T Q (T) \sum_ {k = 1} ^ {T} \mathbb {P} \left\{\boldsymbol {\theta} ^ {*} \notin B _ {k} \right\} \\ \leq \frac {\tilde {\delta} (\log (h (T + 1)) + 1) ^ {d} T Q (T)}{5} \sum_ {k = 1} ^ {\infty} \frac {1}{k ^ {6}} \\ <   \tilde {\delta} (\log (h (T + 1)) + 1) ^ {d} T Q (T). \tag {32} \\ \end{array}
$$
We now upper bound the second term in (28). From (30),
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \beta_ {k} (\boldsymbol {Z} (t)) \right] \leq 2 c _ {p _ {2}} \mathbb {E} \left[ \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r + r _ {*} ^ {p}} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \beta_ {k} (\boldsymbol {Z} (t)) \right]. \tag {33}
$$
To bound the regret term resulting from the summation of  $\beta_{k}$ $(\mathbf{Z}(t))$  , we note that from the second stopping criterion,  $N_{t}\left(\mathbf{Z}(t)\right)\leq 2N_{t_{k}}\left(\mathbf{Z}(t)\right)$  for all  $t_k\leq t <   t_{k + 1}$  and
$$
\begin{array}{l} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \beta_ {k} (Z (t)) \\ = \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \sqrt {\frac {1 4 \prod_ {i = 1} ^ {d} \left(\boldsymbol {X} _ {i} (t) + h\right)}{\operatorname* {m a x} \left(1 , N _ {t _ {k}} (\boldsymbol {Z} (t))\right)} \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \\ \leq \sqrt {1 4 \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {\tilde {t} _ {k + 1} - 1} \sqrt {\frac {2 \prod_ {i = 1} ^ {d} \left(\boldsymbol {X} _ {i} (t) + h\right)}{\max  \left(1 , N _ {t} (\boldsymbol {Z} (t))\right)}} + \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = \tilde {t} _ {k + 1}} ^ {t _ {k + 1} - 1} \sqrt {\prod_ {i = 1} ^ {d} \left(\boldsymbol {X} _ {i} (t) + h\right)} \right]. \tag {34} \\ \end{array}
$$
The first summation can be simplified as
$$
\begin{array}{l} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {\tilde {t} _ {k + 1} - 1} \sqrt {\frac {2 \prod_ {i = 1} ^ {d} (\boldsymbol {X} _ {i} (t) + h)}{\max (1 , N _ {t} (\boldsymbol {Z} (t)))}} \leq \sqrt {2 (M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h) ^ {d}} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {\tilde {t} _ {k + 1} - 1} \frac {1}{\sqrt {\max (1 , N _ {t} (\boldsymbol {Z} (t)))}} \\ \leq 3 \sqrt {2 \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d}} \sum_ {\boldsymbol {z} \in \{0, 1, \dots , M _ {\boldsymbol {\theta} ^ {*}} ^ {T} \} ^ {d} \times \mathcal {A}} \sqrt {N _ {T + 1} (\boldsymbol {z})} \\ \leq 3 \sqrt {2 | \mathcal {A} |} (M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h) ^ {d} \sqrt {\sum_ {\boldsymbol {z} \in \{0 , 1 , \cdots , M _ {\boldsymbol {\theta} ^ {*}} ^ {T} \} ^ {d} \times \mathcal {A}} N _ {T + 1} (\boldsymbol {z})} ] \\ \leq 3 \sqrt {2 | \mathcal {A} | T} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d}, \\ \end{array}
$$
where the second inequality is due to the following arguments,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {\tilde {t} _ {k + 1} - 1} \frac {1}{\sqrt {\operatorname* {m a x} (1 , N _ {t} (\boldsymbol {Z} (t)))}} = \sum_ {\boldsymbol {z} \in \{0, 1, \dots , M _ {\boldsymbol {\theta} ^ {*}} ^ {T} \} ^ {d} \times \mathcal {A}} \left(\mathbb {I} _ {\{N _ {T + 1} (\boldsymbol {z}) > 0 \}} + \sum_ {i = 1} ^ {N _ {T + 1} (\boldsymbol {z}) - 1} \frac {1}{\sqrt {i}}\right) \\ \leq 3\sum_{\substack{\boldsymbol {z}\in \{0,1,\dots ,M_{\boldsymbol{\theta}^{*}}^{T}\}^{d}\times \mathcal{A}}}\sqrt{N_{T + 1}(\boldsymbol{z})}. \\ \end{array}
$$
For the second term in (34), we get
$$
\begin{array}{l} \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = \tilde {t} _ {k + 1}} ^ {t _ {k + 1} - 1} \sqrt {\prod_ {i = 1} ^ {d} \left(\boldsymbol {X} _ {i} (t) + h\right)} = \sqrt {\left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d}} \sum_ {k = 1} ^ {K _ {T}} E _ {k} \\ \leq K _ {T} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \sqrt {\left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d}} \\ \leq 2 \sqrt {| \mathcal {A} | T \log_ {2} T} \left(\max _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) (M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h) ^ {d}, \\ \end{array}
$$
where  $E_{k} = T_{k} - \tilde{T}_{k}$ , and  $K_{T}$  is bounded from Lemma 8. Thus  $\sum_{k=1}^{K_{T}} \sum_{t=t_{k}}^{t_{k+1}-1} \beta_{k}\left(\mathbf{Z}(t)\right)$  is bounded as
$$
\sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \beta_ {k} \left(\boldsymbol {Z} (t)\right) \leq 2 4 \sqrt {| \mathcal {A} | T \log_ {2} T \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d}.
$$
Substituting the above bound in (33),
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} 2 v _ {M} \beta_ {k} (Z (t)) \right] \\ \leq 4 8 c _ {p _ {2}} \sqrt {| \mathcal {A} | T \log_ {2} T \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \mathbb {E} \left[ \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T}\right) ^ {r + r _ {*} ^ {p}} \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \right] \\ \leq c _ {p _ {3}} \sqrt {| \mathcal {A} | T \log_ {2} T \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \mathbb {E} \left[ (M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h) ^ {d + r + r _ {*} ^ {P}} \left(\max _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \right], \\ \end{array}
$$
where  $c_{p_3} \coloneqq 48c_{p_2}$ . Finally, from the above equation, (32), and (28),
$$
\begin{array}{l} R _ {2} \leq \tilde {\delta} (\log (h (T + 1)) + 1) ^ {d} T Q (T) \\ + c _ {p _ {3}} \sqrt {| \mathcal {A} | T \log_ {2} T \log \left(\frac {2 | \mathcal {A} | T}{\tilde {\delta}}\right)} \mathbb {E} \left[ (M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h) ^ {d + r + r _ {*} ^ {p}} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \right]. \\ \end{array}
$$
By choosing  $\tilde{\delta} = \frac{1}{TQ(T)}$ , we get
$R_{2}$
$$
\begin{array}{l} \leq \left(\log (h(T + 1)) + 1\right)^{d} + c_{p_{3}}\sqrt{\left|\mathcal{A}\right|T\log_{2}T\log(2\left|\mathcal{A}\right|T^{2}Q(T))}\mathbb{E}\left[ \left(M_{\boldsymbol{\theta}^{*}}^{T} + h\right)^{d + r + r_{*}^{p}}\Bigg(\max_{1\leq i\leq T}\tau_{0^{d}}^{(i)}\Bigg)\right], \\ \leq \left(\log (h (T + 1)) + 1\right) ^ {d} + c _ {p _ {3}} \sqrt {| \mathcal {A} | T} \log_ {2} \left(2 | \mathcal {A} | T ^ {2} Q (T)\right) \mathbb {E} \left[ \left(M _ {\boldsymbol {\theta} ^ {*}} ^ {T} + h\right) ^ {d + r + r _ {*} ^ {p}} \left(\max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)}\right) \right], \\ \end{array}
$$
where  $Q(T) = c_{p_2}(Th)^{r + r_*^p}$ .
B.7 Proof of Theorem 1
Proof. Lemmas 1, 2, and 3 along with Cauchy-Schwarz inequality showed that the regret terms  $R_0$  and  $R_2$  are of the order  $\tilde{O} (KrdJ^* h^{d + 2r + r_*^p}\sqrt{|\mathcal{A}|T})$  and the term  $R_1$  is  $\tilde{O}(J^{*}(h)^{r_{*}^{p}})$ . Therefore, from  $R(T,\pi_{TSDE}) = R_0 + R_1 + R_2$ , the regret of Algorithm 1,  $R(T,\pi_{TSDE})$ , is  $\tilde{O} (KrdJ^* h^{d + 2r + r_*^p}\sqrt{|\mathcal{A}|T})$ .
B.8 Requirement of an optimal policy oracle.
To implement our algorithm, we need to find the optimal policy for each model sampled by the algorithm—optimal policy for Theorem 1 and optimal policy within policy class  $\Pi$  for Corollary 1; this has also been used in past work [23, 24, 36]. In the finite state-space setting, [49] provides a schedule of  $\epsilon$  values and selects  $\epsilon$ -optimal policies to obtain  $\tilde{O}(\sqrt{T})$  regret guarantees. The issue with extending the analysis of [49] to the countable state-space setting is that we need to ensure (uniform) ergodicity for the chosen  $\epsilon$ -optimal policies; the lim sup or lim inf of the time-average expected reward (used to define the average cost problem) being finite doesn't imply ergodicity. In other words, we must formulate (and verify) ergodicity assumptions for a potentially large set of close-to-optimal algorithms whose structure is undetermined. Another issue is that, to the best of our knowledge, there isn't a general structural characterization of all  $\epsilon$ -optimal stationary policies for countable state-space MDPs or even a characterization of the policy within this set that is selected by any computational procedure in the literature; current results only discuss existence and characterization of the stationary optimal policy. In the absence of such results, stability assumptions with the same uniformity across models as in our submission will be needed, which are likely too strong to be useful.
If we could verify the stability requirements of Assumptions 3 and 4 for a subset of policies, the optimal oracle is not needed, and instead, by choosing approximately optimal policies within this subset, we can follow the same proof steps as [49] to guarantee regret performance similar to Corollary 1 (without knowledge of model parameters). To theoretically analyze the performance of the algorithm that follows an approximately optimal policy rather than the optimal one, we assume that for a specific sequence of  $\{\epsilon_k\}_{k=1}^{\infty}$ , an  $\epsilon_k$ -optimal policy is given, which is defined below.
Definition 1. Policy  $\pi \in \Pi$  is called an  $\epsilon$ -optimal policy if for every  $\theta \in \Theta$ ,
$$
c (\boldsymbol {x}, \pi (\boldsymbol {x})) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta} (\boldsymbol {y} | \boldsymbol {x}, \pi (\boldsymbol {x})) v (\boldsymbol {y}; \theta) \leq c (\boldsymbol {x}, \pi_ {\theta} ^ {*} (\boldsymbol {x})) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta} (\boldsymbol {y} | \boldsymbol {x}, \pi_ {\theta} ^ {*} (\boldsymbol {x})) v (\boldsymbol {y}; \theta) + \epsilon ,
$$
where  $\pi_{\theta}^{*}$  is the optimal policy in the policy class  $\Pi$  corresponding to parameter  $\theta$  and  $v(:,\theta)$  is the solution to Poisson equation (5).
Given  $\epsilon$ -optimal policies that satisfy Assumptions 3 and 4, in Theorem 2 we extend the regret guarantees of Corollary 1 to the algorithm employing  $\epsilon$ -optimal policy, instead of the best-in-class policy, and show that the same regret upper bounds continue to apply.
Theorem 3. Consider a non-negative sequence  $\{\epsilon_k\}_{k=1}^{\infty}$  such that for every  $k \in \mathbb{N}$ ,  $\epsilon_k$  is bounded above by  $\frac{1}{k+1}$  and an  $\epsilon_k$ -optimal policy satisfying Assumptions 3 and 4 is given. The regret incurred by Algorithm 1 while using the  $\epsilon_k$ -optimal policy during any episode  $k$  is  $\tilde{O}(dh^d \sqrt{|\mathcal{A}|T})$ .
Proof. For the  $\epsilon_{k}$ -optimal policy used in episode  $k$ , shown by  $\pi^{\epsilon_k}$ , we have
$$
\begin{array}{l} c (\boldsymbol {x}, \pi^ {\epsilon_ {k}} (\boldsymbol {x})) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {x}, \pi^ {\epsilon_ {k}} (\boldsymbol {x})) v (\boldsymbol {y}; \theta_ {k}) \leq c (\boldsymbol {x}, \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {x})) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {x}, \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {x})) v (\boldsymbol {y}; \theta_ {k}) + \epsilon_ {k} \\ = J \left(\theta_ {k}\right) + v (\boldsymbol {x}; \theta_ {k}) + \epsilon_ {k}. \\ \end{array}
$$
$$
R (T, \pi_ {T S D E}) = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} c (\boldsymbol {X} (t), \pi^ {\epsilon_ {k}} (\boldsymbol {X} (t))) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right] = R _ {0} + R _ {1} + R _ {2} + \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} T _ {k} \epsilon_ {k} \right]
$$
with  $R_0 = \mathbb{E}\left[\sum_{k = 1}^{K_T}T_kJ(\theta_k)\right] - T\mathbb{E}\left[J(\pmb {\theta}^*)\right]$
$$
R _ {1} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v (\boldsymbol {X} (t); \theta_ {k}) - v (\boldsymbol {X} (t + 1); \theta_ {k}) \right] \right],
$$
$$
R _ {2} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v (\boldsymbol {X} (t + 1); \theta_ {k}) - \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {X} (t), \pi^ {\epsilon_ {k}} (\boldsymbol {X} (t))) v (\boldsymbol {y}; \theta_ {k}) \right] \right].
$$
We assumed that given  $\epsilon$ -optimal policies satisfy Assumptions 3 and 4. As a result, we can utilize the proof of Theorem 1 to deduce that the term  $R_0 + R_1 + R_2$  is of the order  $\tilde{O}(dh^d \sqrt{|\mathcal{A}|T})$ . Moreover, we can simplify the term  $\mathbb{E}\left[\sum_{k=1}^{K_T} T_k \epsilon_k\right]$  as below:
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} T _ {k} \epsilon_ {k} \right] = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \tilde {T} _ {k} \epsilon_ {k} \right] + \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} E _ {k} \epsilon_ {k} \right]. \tag {35}
$$
From the second stopping condition of Algorithm 1, we have  $\tilde{T}_k \leq \tilde{T}_{k-1} + 1 \leq \ldots \leq k+1$  and
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} T _ {k} \epsilon_ {k} \right] \leq \mathbb {E} [ K _ {T} ],
$$
where we have used the assumption that  $\epsilon_k \leq \frac{1}{k + 1}$ . For the second term of (35), from (25)
$$
\mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} E _ {k} \epsilon_ {k} \right] \leq \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \frac {E _ {k}}{k + 1} \right] \leq \mathbb {E} \left[ \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \sum_ {k = 1} ^ {K _ {T}} \frac {1}{k + 1} \right] \leq \mathbb {E} \left[ \max  _ {1 \leq i \leq T} \tau_ {0 ^ {d}} ^ {(i)} \log \left(K _ {T} + 1\right) \right], \tag {36}
$$
where in the last inequality we have used  $\sum_{i=1}^{n} \frac{1}{n} \leq 1 + \log(n)$ . Finally, as a result of Lemma 6 and Lemma 8, the result follows.
C Bounds on hitting times under polynomial and geometric ergodicity
C.1 Polynomial upper bounds for the moments of hitting time of state  $0^{d}$
For any  $\theta_1, \theta_2 \in \Theta$ , consider the Markov process with transition kernel  $P_{\theta_1}^{\pi_{\hat{\theta}_2}}$  obtained from the MDP  $(\mathcal{X}, \mathcal{A}, c, P_{\theta_1})$  by following policy  $\pi_{\theta_2}^*$ . [29, Lemma 3.5] establishes that if the process is polynomially ergodic, equivalently satisfies (4), then for every  $0 < \eta \leq 1$ , there exists constants  $\beta_{\theta_1, \theta_2}^\eta, b_{\theta_1, \theta_2}^\eta > 0$  such that the following holds: