SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When  $h = \widetilde{H} + 1$ , the base assertion holds immediately. Suppose the assertion is true for time  $h + 1$ . At time  $h$ , for any fixed  $(\tau_h, a_h)$ , if  $Q_h(\tau_h, a_h) = H$ , the assertion holds true. Otherwise, we have
$$
\begin{array}{l} Q _ {h} (\tau_ {h}, a _ {h}) - Q _ {h} ^ {*} (\tau_ {h}, a _ {h}) = [ \widehat {\mathcal {P}} _ {h} V _ {h + 1} ] (\tau_ {h}, a _ {h}) - [ \mathcal {P} _ {h} V _ {h + 1} ^ {*} ] (\tau_ {h}, a _ {h}) + b _ {h} ^ {k} (\tau_ {h}, a _ {h}) \\ \geq \underbrace {\left([ \widehat {\mathcal {P}} _ {h} - \mathcal {P} _ {h} ] V _ {h + 1} ^ {*}\right) (\tau_ {h} , a _ {h})} _ {(A)} + b _ {h} ^ {k} (\tau_ {h}, a _ {h}). \\ \end{array}
$$
We show a lower bound on  $(A)$ . If  $h \geq H$ , expanding the transition kernel  $\mathcal{P}_h$  leads to
$$
\begin{array}{l} (A) = \sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} \left(\tau_ {h + 1} \mid \tau_ {h}, a _ {h}\right) - p _ {h} \left(\tau_ {h + 1} \mid \tau_ {h}, a _ {h}\right)\right) \\ \stackrel {(i)} {=} \sum_ {s _ {t _ {h} + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {t _ {h}} \left(s _ {t _ {h} + 1} \mid s _ {t _ {h}}, a _ {t _ {h}}\right) - p _ {t _ {h}} \left(s _ {t _ {h} + 1} \mid s _ {t _ {h}}, a _ {t _ {h}}\right)\right) \\ \stackrel {(i i)} {\geq} - c _ {A, 1} H \sqrt {\frac {H \iota}{N _ {t _ {h}} \left(s _ {t _ {h}} , a _ {t _ {h}}\right)}}, \\ \end{array}
$$
where equality  $(i)$  requires  $\tau_{h + 1}$  to take  $s_{t_h + 1}$  as the new state observation, and inequality  $(ii)$  follows from the Hoeffding's inequality (Lemma D.2) with a constant  $c_{A,1}$ . Note that the  $H\iota$  term in the numerator comes from a union bound over  $\widetilde{S}_{\mathrm{aug}}\times \mathcal{A}$ .
On the other hand, if  $h < H$ , expanding the transition kernel  $\mathcal{P}_h$  yields
$$
\begin{array}{l} (A) = \sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h}) - p _ {h} (\tau_ {h + 1} | \tau_ {h}, a _ {h})\right) \\ = \underbrace {\sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {h} \left(\tau_ {h + 1} \mid \tau_ {h} , a _ {h}\right) - p _ {h} \left(\tau_ {h + 1} \mid \tau_ {h} , a _ {h}\right)\right) \mathbb {1} \left\{\delta_ {t _ {h + 1}} = 0 \right\} \mathbb {1} \left\{t _ {h + 1} = t _ {h} + 1 \right\}} _ {(A _ {1})} \\ + \underbrace {\sum_ {\tau_ {h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\widehat {p} _ {h} (\tau_ {h + 1} | \tau_ {h} , a _ {h}) - p _ {h} (\tau_ {h + 1} | \tau_ {h} , a _ {h})\right) \mathbb {1} \left\{\delta_ {t _ {h + 1}} = \delta_ {t _ {h}} + 1 \right\} \mathbb {1} \left\{t _ {h + 1} = t _ {h} \right\}} _ {(A _ {2})}. \\ \end{array}
$$
Note that  $(A_{1})$  accounts for receiving a new state observation in  $\tau_{h + 1}$ , and  $(A_{2})$  accounts for no new state observation. We tackle these two terms separately. For  $(A_{1})$ , we have
$$
\left(A _ {1}\right)
$$
$$
\begin{array}{l} = \sum_ {s _ {t _ {h + 1}}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left((1 - \widehat {\theta} _ {t _ {h}} (s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}})) \widehat {p} _ {t _ {h}} (s _ {t _ {h + 1}} | s _ {t _ {h}}, a _ {t _ {h}}) - (1 - \theta_ {t _ {h}} (s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}})) p _ {t _ {h}} (s _ {t _ {h + 1}} | s _ {t _ {h}}, a _ {t _ {h}})\right) \\ = \sum_ {s _ {t _ {h + 1}}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) \left(\left(1 - \widehat {\theta} _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right)\right) - \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right)\right)\right) \widehat {p} _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}}, a _ {t _ {h}}\right) \\ + \sum_ {s _ {t h + 1}} V _ {h + 1} ^ {*} (\tau_ {h + 1}) (1 - \theta_ {t h} \left(s _ {t h}, a _ {t h}, \delta_ {t h}\right)) \left(\widehat {p} _ {t h} \left(s _ {t h + 1} \mid s _ {t h}, a _ {t h}\right) - p _ {t h} \left(s _ {t h + 1} \mid s _ {t h}, a _ {t h}\right)\right) \\ \end{array}
$$
$$
\stackrel {(i)} {\geq} - H \left| \widehat {\theta} _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) \right| - c _ {A, 2} H \sqrt {\frac {H \iota}{N _ {t _ {h}} \left(s _ {t _ {h}} , a _ {t _ {h}}\right)}},
$$
where in  $(i)$ , the first term is the estimation error of  $\widehat{\theta}$  using the collected data, the second term follows from Hoeffding's inequality, and  $c_{A,2}$  is an absolute constant. For  $(A_2)$ , we have
$$
\left(A _ {2}\right) \geq - H \left| \widehat {\theta} _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) \right|,
$$
since  $\tau_{h + 1}$  is now uniquely determined. Summing up  $(A_{1})$  and  $(A_{2})$ , we obtain
$$
(A) = (A _ {1}) + (A _ {2}) \geq - 2 H \left| \widehat {\theta} _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) \right| - c _ {A, 2} H \sqrt {\frac {H \iota}{N _ {t _ {h}} \left(s _ {t _ {h}} , a _ {t _ {h}}\right)}}.
$$
It remains to bound the estimation error of  $\widehat{\theta}_{t_h}(s_{t_h},a_{t_h},\delta_{t_h})$ . Using the Hoeffding's inequality again, we obtain
$$
\left| \widehat {\theta} _ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}}, a _ {t _ {h}}, \delta_ {t _ {h}}\right) \right| \leq c _ {\theta} \sqrt {\frac {H \iota}{N _ {t _ {h}} \left(s _ {t _ {h}} , a _ {t _ {h}} , \delta_ {t _ {h}}\right)}}.
$$
Taking  $c_{A} = \max \{c_{A,1}, c_{A,2}, c_{\theta}, 2\}$ , we have
$$
(A) \geq - c _ {A} H \left(\sqrt {\frac {H \iota}{N _ {t _ {h}} (s _ {t _ {h}} , a _ {t _ {h}} , \delta_ {t _ {h}})}} + \sqrt {\frac {H \iota}{N _ {t _ {h}} (s _ {t _ {h}} , a _ {t _ {h}})}}\right).
$$
With the choice of the bonus function, it can be checked that
$$
\widetilde {Q} _ {h, \mathrm {a u g}} ^ {k} \left(\tau_ {h}, a _ {h}\right) - \widetilde {Q} _ {h, \mathrm {a u g}} ^ {*} \left(\tau_ {h}, a _ {h}\right) \geq (A) + b _ {h} ^ {k} \left(\tau_ {h}, a _ {h}\right) \geq 0
$$
with probability  $1 - \gamma$  for any  $(\tau_h, a_h)$ .
Regret analysis In the sequel, we omit subscripts "aug" and tilde  $\widetilde{\cdot}$  for simplicity. Thanks to Lemma B.1, we consider  $(Q_h^k - Q_h^{\pi_k})(\tau_h^k, a_h^k)$  as an upper bound of  $(Q_h^* - Q_h^{\pi_k})(\tau_h^k, a_h^k)$ . We bound  $(Q_h^k - Q_h^{\pi_k})(\tau_h^k, a_h^k)$  by
$$
\begin{array}{l} \left(Q _ {h} ^ {k} - Q _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) \\ \leq \left(\left[ \widehat {\mathcal {P}} _ {h} ^ {k} V _ {h + 1} ^ {k} - \mathcal {P} _ {h} V _ {h + 1} ^ {\pi_ {k}} \right]\right) (\tau_ {h} ^ {k}, a _ {h} ^ {k}) + b _ {h} ^ {k} (\tau_ {h} ^ {k}, a _ {h} ^ {k}) \\ \leq \left(\left[ \widehat {\mathcal {P}} _ {h} ^ {k} - \mathcal {P} _ {h} \right] V _ {h + 1} ^ {*}\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) + \left(\left[ \widehat {\mathcal {P}} _ {h} ^ {k} - \mathcal {P} _ {h} \right] \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right]\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) \\ + \left(\mathcal {P} _ {h} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} \right]\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) + b _ {h} ^ {k} \left(\tau_ {h}, a _ {h} ^ {k}\right) \\ \leq \underbrace {\left([ \widehat {\mathcal {P}} _ {h} ^ {k} - \mathcal {P} _ {h} ] [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ]\right) (\tau_ {h} ^ {k} , a _ {h} ^ {k})} _ {(A)} + \left(\mathcal {P} _ {h} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} ]\right) (\tau_ {h} ^ {k}, a _ {h} ^ {k}) + 2 b _ {h} ^ {k} (\tau_ {h} ^ {k}, a _ {h} ^ {k}). \tag {B.1} \\ \end{array}
$$
Similar to Lemma B.1, for  $h \geq H$ , we expand term (A) into
$$
\begin{array}{l} (A) = \sum_ {\tau_ {h + 1}} \left(\hat {p} _ {h} ^ {k} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k}, a _ {h} ^ {k}\right) - p _ {h} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k}, a _ {h} ^ {k}\right)\right) \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \\ = \sum_ {s _ {t _ {h} + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {t _ {h}} ^ {k} \left(s _ {t _ {h} + 1} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) - p _ {t _ {h}} \left(s _ {t _ {h} + 1} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right). \tag {B.2} \\ \end{array}
$$
On the other hand, for  $h \leq H$ , the decomposition of term (A) is more complicated. We have
$$
\begin{array}{l} (A) = \sum_ {\tau_ {h + 1}} \left(\widehat {p} _ {h} ^ {k} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k}, a _ {h} ^ {k}\right) - p _ {h} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k}, a _ {h} ^ {k}\right)\right) \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \\ = \underbrace {\sum_ {\tau_ {h + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {h} ^ {k} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k} , a _ {h} ^ {k}\right) - p _ {h} \left(\tau_ {h + 1} \mid \tau_ {h} ^ {k} , a _ {h} ^ {k}\right)\right) \mathbb {1} \left\{\delta_ {t _ {h + 1}} = 0 \right\} \mathbb {1} \left\{t _ {h + 1} = t _ {h} ^ {k} + 1 \right\}} _ {(A _ {1})} \\ + \underbrace {\sum_ {\tau_ {h + 1}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) \left(\widehat {p} _ {h} ^ {k} (\tau_ {h + 1} | \tau_ {h} ^ {k} , a _ {h} ^ {k}) - p _ {h} (\tau_ {h + 1} | \tau_ {h} ^ {k} , a _ {h} ^ {k})\right) \mathbb {1} \{\delta_ {t _ {h + 1}} = \delta_ {t _ {h} ^ {k}} + 1 \} \mathbb {1} \{t _ {h + 1} = t _ {h} ^ {k} \}} _ {(A _ {2})}. \\ \end{array}
$$
Term  $(A_{2})$  can be directly bounded by
$$
\begin{array}{l} (A _ {2}) \leq H \left| \widehat {\theta} _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right) \right| \\ \leq c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k})}} \\ \end{array}
$$
with probability  $1 - \gamma$ . To bound  $(A_{1})$ , we have
$$
\begin{array}{l} (A _ {1}) = \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] (\tau_ {h + 1}) \Bigg (\left(1 - \widehat {\theta} _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) \widehat {p} _ {t _ {h}} ^ {k} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) \\ \left. - \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right) \\ = \sum_ {s _ {t _ {h + 1}}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) \left(\left(1 - \widehat {\theta} _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k})\right) - \left(1 - \theta_ {t _ {h}} (s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k})\right)\right) \widehat {p} _ {t _ {h}} ^ {k} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}) \\ + \sum_ {s _ {t _ {h + 1}}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) \left(1 - \theta_ {t _ {h}} (s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k})\right) \left(\widehat {p} _ {t _ {h}} ^ {k} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}) - p _ {t _ {h}} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k})\right) \\ \leq \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {t _ {h}} ^ {k} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) - p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right) \\ + H \left| \widehat {\theta} _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right) - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right) \right| \\ \leq \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {t _ {h}} ^ {k} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) - p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right) \\ + c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}. \\ \end{array}
$$
Putting  $(A_{1})$  and  $(A_{2})$  together, we obtain
$$
\begin{array}{l} (A) \leq \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) \left(\widehat {p} _ {t _ {h}} ^ {k} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) - p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right) \\ + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t h} ^ {k} \left(s _ {t h} ^ {k} , a _ {t h} ^ {k} , \delta_ {t h} ^ {k}\right)}}. \tag {B.3} \\ \end{array}
$$
In both (B.2) and (B.3) for different ranges of  $h$ , we apply the Bernstein inequality (Lemma D.1) to derive
$$
\begin{array}{l} \sum_ {s _ {t _ {h + 1}}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) \left(\hat {p} _ {t _ {h}} ^ {k} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}) - p _ {t _ {h}} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k})\right) \\ \leq c \cdot \sum_ {s _ {t _ {h + 1}}} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} ] (\tau_ {h + 1}) \left[ \sqrt {\frac {p _ {t _ {h}} (s _ {t _ {h + 1}} | s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k}) \iota}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k})}} + \frac {\iota}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k})} \right] \\ \stackrel {(i)} {\leq} c \cdot \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] (\tau_ {h + 1}) \left[ \frac {p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k}\right)}{2 c H} + \frac {(2 c H + 1) \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k}\right)} \right] \\ \leq c \cdot \left(\frac {S H (2 c H + 1) \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k}\right)} + \frac {1}{2 c H} \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right)\right), \tag {B.4} \\ \end{array}
$$
where inequality  $(i)$  follows from  $\sqrt{ab} \leq a + b$ . Substituting (B.4) into (B.2), for  $h \geq H$ , we deduce
$$
\begin{array}{l} (A) \leq \frac {1}{2 H} \sum_ {s _ {t h + 1}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) p _ {t _ {h}} \left(s _ {t h + 1} \mid s _ {t h} ^ {k}, a _ {t h} ^ {k}\right) + \frac {c S H (2 c H + 1) u}{N _ {t h} ^ {k} \left(s _ {t h} ^ {k} , a _ {t h} ^ {k}\right)} \\ \stackrel {(i)} {\leq} \frac {1}{2 H} \left(\mathcal {P} _ {h} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} ]\right) (\tau_ {h} ^ {k}, a _ {h} ^ {k}) + c ^ {\prime} \frac {m S H ^ {2} \iota}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k})}, \\ \end{array}
$$
where  $c'$  is a sufficiently large constant. By the same reasoning, substituting (B.4) into (B.3), for  $h < H$ , we have
$$
(A) \leq \frac {1}{2 H} \left(1 - \theta_ {t _ {h}} \left(s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}, \delta_ {t _ {h}} ^ {k}\right)\right) \sum_ {s _ {t _ {h + 1}}} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {*} \right] \left(\tau_ {h + 1}\right) p _ {t _ {h}} \left(s _ {t _ {h + 1}} \mid s _ {t _ {h}} ^ {k}, a _ {t _ {h}} ^ {k}\right) + \frac {c S H (2 c H + 1) u _ {0}}{N _ {t h} ^ {k} \left(s _ {t h} ^ {k} , a _ {t h} ^ {k}\right)}
$$
$$
\begin{array}{l} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}} \\ \stackrel {(i)} {\leq} \frac {1}{2 H} \left(\mathcal {P} _ {h} [ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} ]\right) (\tau_ {h} ^ {k}, a _ {h} ^ {k}) + c ^ {\prime} \frac {S H ^ {2} \imath}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k})} + 2 c _ {\theta} H \sqrt {\frac {H \imath}{N _ {t _ {h}} ^ {k} (s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k})}}. \\ \end{array}
$$
We denote  $\zeta_h^k = c' \frac{SH^2\iota}{N_{t_h}^k(s_{t_h}^k,a_{t_h}^k)}$ . Now we have a unified upper bound on  $(A)$  for any  $h \in [1,\widetilde{H}]$  as
$$
(A) \leq \frac {1}{2 H} \left(\mathcal {P} _ {h} \left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} \right]\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) + \zeta_ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}. \tag {B.5}
$$
Substituting (B.5) back into (B.1), we have
$$
\begin{array}{l} \left(V _ {h} ^ {k} - V _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}\right) = \left(Q _ {h} ^ {k} - Q _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) \\ \leq \left(1 + \frac {1}{2 H}\right) \left(\mathcal {P} _ {h} \left[ V _ {h} ^ {k} - V _ {h} ^ {\pi_ {k}} \right]\right) \left(\tau_ {h} ^ {k}, a _ {h} ^ {k}\right) + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}. \\ \end{array}
$$
We further denote  $\xi_h^k = \left(\mathcal{P}_h\left[V_h^k -V_h^{\pi_k}\right]\right)\left(\tau_h^k,a_h^k\right) - \left[V_{h + 1}^k -V_{h + 1}^{\pi_k}\right]\left(\tau_{h + 1}^k\right)$  and rewrite  $\left(V_h^k -V_h^{\pi_k}\right)\left(\tau_h^k\right)$  as
$$
\left(V _ {h} ^ {k} - V _ {h} ^ {\pi_ {k}}\right) \left(\tau_ {h} ^ {k}\right) \leq \left(1 + \frac {1}{2 H}\right) \left(\left[ V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi_ {k}} \right] \left(\tau_ {h + 1} ^ {k}\right) + \xi_ {h} ^ {k}\right) + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}.
$$
Recall  $\widetilde{H} = 2H$ . Using a recursive summation argument, we deduce
$$
\begin{array}{l} \left(V _ {1} ^ {k} - V _ {1} ^ {\pi_ {k}}\right) \left(\tau_ {1} ^ {k}\right) \leq \sum_ {h = 1} ^ {\widetilde {H}} \left(1 + \frac {1}{2 H}\right) ^ {h} \left(\xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}\right) \\ \leq e \sum_ {h = 1} ^ {2 H} \left(\xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}\right). \\ \end{array}
$$
As a consequence, the total regret is bounded by
$$
\operatorname {R e g r e t} (K) \leq e \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \left(\xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + 2 b _ {h} ^ {k} + 2 c _ {\theta} H \sqrt {\frac {H \iota}{N _ {t h} ^ {k} \left(s _ {t h} ^ {k} , a _ {t h} ^ {k} , \delta_ {t h} ^ {k}\right)}}\right). \tag {B.6}
$$
We need to sum over  $\zeta_h^k, \xi_h^k, b_h^k$ . Consider  $\zeta_h^k$  first. We have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \zeta_ {h} ^ {k} = c ^ {\prime} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \frac {S H ^ {2} \iota}{N _ {t h} ^ {k} \left(s _ {t h} ^ {k} , a _ {t h} ^ {k}\right)} \\ \stackrel {(i)} {\leq} c ^ {\prime} H \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {S H ^ {2} \iota}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)} \\ \stackrel {(i i)} {\leq} c _ {\zeta} H ^ {4} S ^ {2} A \iota^ {2}, \tag {B.7} \\ \end{array}
$$
where inequality  $(i)$  invokes the fact that  $t_h$  only takes value in  $\{1, \ldots, H\}$  and each  $N_{t_h}^k (s_{t_h}^k, a_{t_h}^k)$  is repeated at most  $H$  times, and inequality  $(ii)$  follows from the pigeon-hole argument in Azar et al. [2017].
Next we bound the summation over  $\xi_h^k$ . This is a martingale difference sequence. We apply Azuma-Hoeffding's inequality (Lemma D.3) with  $n = 2H$  and  $c_{i} = 4H$  to obtain
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \xi_ {h} ^ {k} \leq c _ {\xi} \sqrt {K H ^ {4} \iota}. \tag {B.8}
$$
The additional  $H$  dependence above comes from a union bound over  $\widetilde{S}_{\mathrm{aug}} \times \mathcal{A}$ . Lastly, we tackle the summation over bonus functions  $b_h^k$ . We have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} b _ {h} ^ {k} = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} c _ {A} H \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} , a _ {t _ {h}}\right)}} \\ \leq c _ {A} H \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} H \sqrt {\frac {H \iota}{N _ {t h} ^ {k} \left(s _ {t _ {h}} , a _ {t _ {h}}\right)}} \\ \leq c _ {b} H ^ {7 / 2} \sqrt {S A K \iota}. \tag {B.9} \\ \end{array}
$$
Putting (B.7), (B.8) and (B.9) together, we deduce
$$
\operatorname {R e g r e t} (K) \leq c \left(H ^ {7 / 2} \sqrt {S A K \iota} + H ^ {4} S ^ {2} A \iota^ {2} + \sqrt {H ^ {4} K \iota}\right) + 2 e c _ {\theta} H \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \sqrt {\frac {H \iota}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}}
$$
for some constant  $c$ . To this end, the only remaining task is to find  $\sum_{k=1}^{K} \sum_{h=1}^{2H} \sqrt{\frac{1}{N_{t_h}^k (s_{t_h}^k, a_{t_h}^k, \delta_{t_h}^k)}},$  which undergoes a similar argument as the bonus summation. We have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \sqrt {\frac {1}{N _ {t _ {h}} ^ {k} \left(s _ {t _ {h}} ^ {k} , a _ {t _ {h}} ^ {k} , \delta_ {t _ {h}} ^ {k}\right)}} \leq H \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sqrt {\frac {1}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , \delta_ {h} ^ {k}\right)}} \\ = H \sum_ {(h, s, a, \delta)} \sum_ {i = 1} ^ {N _ {h} ^ {K} (s, a, \delta)} \sqrt {\frac {1}{i}} \\ \stackrel {(i)} {\leq} 2 H \sum_ {\delta} \sum_ {(h, s, a)} \sqrt {N _ {h} ^ {K} (s , a , \delta)} \\ \stackrel {(i i)} {\leq} 2 H \sum_ {\delta} \sqrt {S A K H} \\ \stackrel {(i i i)} {\leq} 2 H ^ {2} \sqrt {S A K H}, \tag {B.10} \\ \end{array}
$$
where inequality  $(i)$  invokes  $\sum_{i=1}^{n} 1 / \sqrt{i} \leq 2\sqrt{n}$ , inequality  $(ii)$  follows from Cauchy-Schwarz, and inequality  $(iii)$  uses the fact that  $\delta$  is bounded by  $H$ . Plugging (B.10) into the regret bound, we obtain the desired result
$$
\operatorname {R e g r e t} (K) \leq c \left(H ^ {4} \sqrt {S A K \iota} + H ^ {4} S ^ {2} A \iota^ {2} + \sqrt {H ^ {4} K \iota}\right)
$$
with probability  $1 - \gamma$ . Absorbing  $\sqrt{H^4K\iota}$  into  $H^4\sqrt{SAK\iota}$  yields the bound in Theorem 4.1.
B.2 Regret under bounded delay
To obtain the regret bound under bounded delay, we only need to modify several steps in the proof of Theorem 4.1. Specifically, in Lemma B.1, we replace  $H$  by  $D + 1$  in the square root, so that the bonus function becomes
$$
b _ {h} ^ {k} (\tau_ {h}, a _ {h}) = c _ {A} H \left(\sqrt {\frac {(D + 1) \iota}{N _ {t _ {h}} (s _ {t _ {h}} , a _ {t _ {h}} , \delta_ {t _ {h}})}} + \sqrt {\frac {(D + 1) \iota}{N _ {t _ {h}} (s _ {t _ {h}} , a _ {t _ {h}})}}\right).
$$
The optimism still holds, since the expanded state space contains at most  $D$  historical actions.
The second modification is to observe that since the delay is bounded by  $D$ , each counting number  $N_h^k$  only repeats at most  $D + 1$  times. In this way, (B.7) becomes
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {2 H} \zeta_ {h} ^ {k} \leq c _ {\zeta} (D + 1) H ^ {3} S ^ {2} A t ^ {2}.
$$