SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \sum_ {s, a} \frac {1}{\gamma_ {t} (s , a)} q _ {t} (s, a) ^ {2} b _ {t} (s) ^ {2} \\ \leq \frac {1}{2 5 6 L ^ {2} | S |} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \sum_ {s, a} q _ {t} (s, a) ^ {2} b _ {t} (s) ^ {2} \\ \leq \frac {4 L}{2 5 6 L ^ {2} | S |} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} q _ {t} (s) b _ {t} (s) \\ = \mathcal {O} \left(C ^ {\mathrm {P}} \log (| S | T)\right), \tag {55} \\ \end{array}
$$
where the first step uses the fact that  $\gamma_t(s, a) \geq 256L^2|S|$ , the second step bounds  $b_t(s)^2 \leq 4Lb_t(s)$  and  $\sum_{a} q_t(s, a)^2 \leq q_t(s)$ , and the last step applies Lemma 3.1.
Then, we bound the remaining term:
$$
\mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) + 2 \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \sum_ {s, a} \frac {\nu_ {t} (s , a)}{\gamma_ {t} (s , a)} \right]. \tag {56}
$$
Using Definition 5.2, we can bound Eq. (56) as:
$$
\begin{array}{l} \mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) + 2 \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \sum_ {s, a} \frac {\nu_ {t} (s , a)}{\gamma_ {t} (s , a)} \right] \\ = \mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} \left(\gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) + \frac {4}{D} \left(\gamma_ {t _ {i + 1} - 1} (s, a) - \gamma_ {t _ {i}} (s, a)\right) \right] \right. \\ \leq \mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} \left(\log (\iota) + \frac {4}{D}\right) \gamma_ {t _ {i + 1} - 1} (s, a) \right] \\ = \mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} 5 \log (\iota) \gamma_ {t _ {i + 1} - 1} (s, a) \right] \\ \leq \mathbb {E} _ {t _ {i}} \left[ \sum_ {s, a} \left(5 \log (\iota) \sqrt {\frac {1}{\log (\iota)} \sum_ {j = t _ {i}} ^ {t _ {i + 1} - 2} \nu_ {j} (s , a)} + 2 5 6 L ^ {2} | S |\right) \right] \\ \leq \mathcal {O} \left(\sqrt {\log (\iota)} \mathbb {E} _ {t _ {i}} \left[ \sqrt {\sum_ {s , a} \sum_ {j = t _ {i}} ^ {t _ {i + 1} - 1} \nu_ {j} (s , a)} \right] + L ^ {2} | S | ^ {2} | A |\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi (s)} \sqrt {L ^ {2} \log (\iota) \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right) \\ + \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ \delta L ^ {2} | S | ^ {2} | A | \log (\iota) \left(t _ {i + 1} - t _ {i}\right) + \sqrt {L | S | ^ {3} | A | ^ {2} \log (\iota) \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathrm {P}}} \right]\right), \tag {57} \\ \end{array}
$$
where the first step applies the definition of  $\gamma_t(s, a)$  which gives:  $\frac{\nu_t(s, a)}{\gamma_t(s, a)} = \frac{2}{D} (\gamma_{t+1}(s, a) - \gamma_t(s, a))$ ; the second step simplifies the telescopic sum and uses  $\gamma_{t_i}(s, a) \geq 0$ ; the third step uses  $D = \frac{1}{\log(\ell)}$ ; the fourth step uses Proposition C.7.3; the sixth step applies Proposition C.7.4.
C.7.3 Bounding ESTREG
Using the equality  $\mathrm{ESTREG}(\pi^{\star}) = \mathbb{E}\big[\sum_{i = 1}^{N}\mathrm{ESTREG}_i(\pi^{\star})\big]$  where  $N$  is the number of epochs, we are now ready to complete the proof of Lemma C.3.4.
Lemma C.7.8. Over the course of  $T$  episodes, Algorithm 4 runs at most  $\mathcal{O}\left(|S||A|\log (T)\right)$  epochs.
Proof. By definition, the algorithm resets each time a counter of the number of visits to a specific state-action pair doubles. Each state-action pair is visited at most once for each of the  $T$  rounds, so it can trigger a new epoch at most  $\log T$  times. Summing on all state-action pairs finishes the proof.
Proof of Lemma C.3.4. Using Lemma C.7.5 and Lemma C.7.7, we have
ESTREG(π\*）
$$
\begin{array}{l} = \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \operatorname {E S T R E G} _ {i} \left(\pi^ {\star}\right) \right] \\ = \mathcal {O} \left(\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \left(\frac {L + L C ^ {\mathsf {P}} | S |}{T} + \delta L ^ {2} | S | ^ {2} | A | \log (\iota) (t _ {i + 1} - t _ {i}) + \sqrt {L | S | ^ {3} | A | ^ {2} \log (\iota) \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathsf {P}})}\right) \right]\right) \\ + \mathcal {O} \left(\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \left(C ^ {\mathsf {P}} \log (| S | T) + L \sqrt {\log (\iota)} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \mathbb {E} _ {t _ {i}} \left[ \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right) \right]\right) \\ = \mathcal {O} \left(\left(\frac {L + L C ^ {\mathsf {P}} | S |}{T}\right) | S | | A | \log (T) + \delta L ^ {2} | S | ^ {2} | A | \log (\iota) T + | A | | S | ^ {2} \log (\iota) \sqrt {L | A | C ^ {\mathsf {P}}}\right) \\ + \mathcal {O} \left(C ^ {\mathsf {P}} | S | | A | \log (\iota) ^ {2} + \sqrt {L ^ {2} \log (\iota)} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {*} (s)} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right) \\ = \mathcal {O} \left(\left(\frac {L + L C ^ {\mathsf {P}} | S |}{T}\right) | S | | A | \log (T) + \delta L ^ {2} | S | ^ {2} | A | \log (\iota) T + | A | | S | ^ {2} \log (\iota) \left(| A | + L C ^ {\mathsf {P}}\right)\right) \\ + \mathcal {O} \left(C ^ {\mathbb {P}} | S | | A | \log (\iota) ^ {2} + \sqrt {L ^ {2} \log (\iota)} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {*} (s)} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right) \\ = \mathcal {O} \left(\left(\frac {L | S | | A | \log (T)}{T}\right) + \delta L ^ {2} | S | ^ {2} | A | \log (\iota) T + C ^ {\mathsf {P}} L | S | ^ {2} | A | \log (\iota) + | A | ^ {2} | S | ^ {2} \log (\iota)\right) \\ + \mathcal {O} \left(C ^ {\mathsf {P}} | S | | A | \log (\iota) ^ {2} + \sqrt {L ^ {2} | S | | A | \log^ {2} (\iota)} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \mathbb {E} \left[ \sqrt {\sum_ {t = 1} ^ {T} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right), \\ \end{array}
$$
where the first step follows from the definition of  $\mathrm{ESTREG}(\pi^{\star})$ ; the third step uses the fact that  $N = \mathcal{O}\left(|S||A|\log (T)\right)$  and the Cauchy-Schwarz inequality; the fourth step uses  $\sqrt{xy} \leq x + y$  for any  $x, y \geq 0$  with  $x = LC^{\mathsf{P}}$  and  $y = |A|$ ; the last step follows from Cauchy-Schwarz inequality again.
C.8 Properties of Optimistic Transition
We summarize the properties guaranteed by the optimistic transition defined in Definition C.1.1.
Lemma C.8.1. For any epoch  $i$ , any transition  $P' \in \mathcal{P}_i$ , any policy  $\pi$ , and any initial state  $u \in S$ , it holds that
$$
q ^ {\widetilde {P} _ {i}, \pi} (s, a | u) \leq q ^ {P ^ {\prime}, \pi} (s, a | u), \quad \forall (s, a) \in S \times A.
$$
Proof. We prove this result via a forward induction from layer  $k(u)$  to layer  $L - 1$ .
Base Case: for the initial state  $u$ ,  $q^{\tilde{P}_i,\pi}(u,a|u) = q^{P',\pi}(u,a|u) = \pi (a|u)$  for any action  $a\in A$ . For the other state  $s\in S_{k(u)}$ , we have  $q^{\widetilde{P}_i,\pi}(s,a|u) = q^{P',\pi}(s,a|u) = 0$ .
Induction step: Suppose  $q^{\widetilde{P}_i,\pi}(s,a|u) \leq q^{P',\pi}(s,a|u)$  holds for all the state-action pair  $(s,a)$  with  $k(s) < h$ . Then, for any  $(s,a) \in S_h \times A$ , we have
$$
\begin{array}{l} q ^ {\widetilde {P} _ {i}, \pi} (s, a | u) = \pi (a | s) \cdot \sum_ {s ^ {\prime} \in S _ {h - 1}} \sum_ {a ^ {\prime} \in A} q ^ {\widetilde {P} _ {i}, \pi} (s ^ {\prime}, a ^ {\prime} | u) \widetilde {P} _ {i} (s | s ^ {\prime}, a ^ {\prime}) \\ \leq \pi (a | s) \cdot \sum_ {s ^ {\prime} \in S _ {h - 1}} \sum_ {a ^ {\prime} \in A} q ^ {P ^ {\prime}, \pi} \left(s ^ {\prime}, a ^ {\prime} | u\right) P ^ {\prime} \left(s | s ^ {\prime}, a ^ {\prime}\right) \\ = q ^ {P ^ {\prime}, \pi} (s, a | u), \\ \end{array}
$$
where the second step follows from the induction hypothesis and the definition of optimistic transition in Definition C.1.1.
Corollary C.8.2. Conditioning on the event  $\mathcal{E}_{\mathrm{CON}}$ , it holds for any epoch  $i$  and any policy  $\pi$  that
$$
q ^ {\widetilde {P} _ {i}, \pi} (s, a) \leq q ^ {P, \pi} (s, a), \quad \forall (s, a) \in S \times A.
$$
Lemma C.8.3. (Optimism of Optimistic Transition) Suppose the high-probability event  $\mathcal{E}_{\mathrm{CON}}$  holds. Then for any policy  $\pi$ , any  $(s, a) \in S \times A$ , and any valid loss function  $\ell: S \times A \to \mathbb{R}_{\geq 0}$ , it holds that
$$
Q ^ {\widetilde {P} _ {i}, \pi} (s, a; \ell) \leq Q ^ {P, \pi} (s, a; \ell), a n d V ^ {\widetilde {P} _ {i}, \pi} (s; \ell) \leq V ^ {P, \pi} (s; \ell), \forall (s, a) \in S \times A.
$$
Proof. According to Corollary C.8.2, we have for all epoch  $i$  that
$$
q ^ {\widetilde {P} _ {i}, \pi} (s, a | u) \leq q ^ {P, \pi} (s, a | u), \quad \forall u \in S \quad \forall (s, a) \in S \times A. \tag {58}
$$
$$
\begin{array}{l} V ^ {\widetilde {P} _ {i}, \pi} (s; \ell) = \sum_ {u \in S} \sum_ {v \in A} q ^ {\widetilde {P} _ {i}, \pi} (u, v | s) \ell (u, v) \\ \leq \sum_ {u \in S} \sum_ {v \in A} q ^ {P, \pi} (u, v | s) \ell (u, v) \\ = V ^ {P, \pi} (s; \ell), \\ \end{array}
$$
where the second step follows from Eq. (58). The statement for the  $Q$ -function can be proven in the same way.
Next, we argue that our optimistic transition provides a tighter performance estimation compared to the approach of Jin et al. [2021]. Specifically, Jin et al. [2021] proposes to subtract the following exploration bonuses  $\mathrm{BONUS}_i: S \times A \to \mathbb{R}$  from the loss functions
$$
\mathrm {B O N U S} _ {i} (s, a) = L \cdot \min  \left\{1, \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} B _ {i} (s, a, s ^ {\prime}) \right\},
$$
where  $B_{i}(s,a,s^{\prime})$  is the confidence bound defined in Eq. (4). This makes sure  $Q^{\bar{P}_i,\pi}\left(s,a;\ell -\mathrm{BONUS}_i\right)$  is no larger than the true  $Q$ -function  $Q^{P,\pi}(s,a;\ell)$  as well, but is a looser lower bound as shown below.
Lemma C.8.4. (Tighter Performance Estimation) For any policy  $\pi$ , any  $(s, a) \in S \times A$ , and any bounded loss function  $\ell: S \times A \to [0,1]$ , it holds that
$$
Q ^ {\bar {P} _ {i}, \pi} (s, a; \ell - \mathrm {B O N U S} _ {i}) \leq Q ^ {\widetilde {P} _ {i}, \pi} (s, a; \ell), \forall (s, a) \in S \times A.
$$
Proof. We prove this result via a backward induction from layer  $L$  to layer 0.
Base Case: for the terminal state  $s_L$ , we have  $Q^{\bar{P}_i,\pi}(s,a;\ell - \mathrm{BONUS}_i) = Q^{\bar{P}_i,\pi}(s,a;\ell) = 0$ .
Induction step: suppose the induction hypothesis holds for all the state-action pairs  $(s, a) \in S \times A$  with  $k(s) > h$ . For any state-action pair  $(s, a) \in S_h \times A$ , we first have
$$
\begin{array}{l} Q ^ {\bar {P} _ {i}, \pi} (s, a; \ell - \mathrm {B O N U S} _ {i}) = \ell (s, a) - \mathrm {B O N U S} _ {i} (s, a) + \sum_ {u \in S _ {h + 1}} \bar {P} _ {i} (u | s, a) V ^ {\bar {P} _ {i}, \pi} (u; \ell - \mathrm {B O N U S} _ {i}) \\ \leq \ell (s, a) - \mathrm {B O N U S} _ {i} (s, a) + \sum_ {u \in S _ {h + 1}} \bar {P} _ {i} (u | s, a) V ^ {\tilde {P} _ {i}, \pi} (u; \ell). \\ \end{array}
$$
Clearly, when  $\sum_{u\in S_{k(s) + 1}}B_i(s,a,u)\geq 1$ , we have  $Q^{\bar{P}_i,\pi}(s,a;\ell -\mathsf{BONUS}_i)\leq 0$  by the definition of  $\mathsf{BONUS}_i$ , which directly implies that  $Q^{\bar{P}_i,\pi}(s,a;\ell -\mathsf{BONUS}_i)\leq Q^{\tilde{P}_i,\pi}(s,a;\ell)$ . So we continue the bound under the condition  $\sum_{u\in S_{k(s) + 1}}B_i(s,a,u) < 1$ :
$$
\begin{array}{l} Q ^ {\bar {P} _ {i}, \pi} (s, a; \ell - \mathrm {B O N U S} _ {i}) \leq \ell (s, a) - \mathrm {B O N U S} _ {i} (s, a) + \sum_ {u \in S _ {h + 1}} \bar {P} _ {i} (u | s, a) V ^ {\tilde {P} _ {i}, \pi} (u; \ell) \\ \leq \ell (s, a) + \sum_ {u \in S _ {h + 1}} \left(\bar {P} _ {i} (u | s, a) - B _ {i} (s, a, u)\right) V ^ {\bar {P} _ {i}, \pi} (u; \ell) \\ \leq \ell (s, a) + \sum_ {u \in S _ {h + 1}} \widetilde {P} _ {i} (u | s, a) V ^ {\bar {P} _ {i}, \pi} (u; \ell) \\ = Q ^ {\bar {P} _ {i}, \pi} (s, a; \ell), \\ \end{array}
$$
where the second step follows from the fact that  $V^{\tilde{P}_i,\pi}(u;\ell) \leq L$ ; the third step follows from the definition of optimistic transition  $\widetilde{P}_i$ .
Combining these two cases proves that  $Q^{\bar{P}_i,\pi}(s,a;\ell -\mathbf{BONUS}_i)\leq Q^{\bar{P}_i,\pi}(s,a;\ell)$  for any  $(s,a)\in S_h\times A$ , finishing the induction.
D Supplementary Lemmas
D.1 Expectation
Lemma D.1.1. ([Jin et al., 2021, Lemma D.3.6]) Suppose that a random variable  $X$  satisfies the following conditions:
-  $X < R$  where  $R$  is a constant.
-  $X < Y$  conditioning on event  $A$ , where  $Y \geq 0$  is a random variable.
Then, it holds that  $\mathbb{E}[X] \leq \mathbb{E}[Y] + \operatorname*{Pr}[A^c] \cdot R$  where  $A^c$  is the complementary event of  $A$ .
D.2 Confidence Bound with Known Corruption
In this subsection, we show that the empirical transition is centered around the transition  $P$ . Let  $[T] := \{1, \dots, T\}$ . Recall that  $E_i := \{t \in [T] : \text{episode } t \text{ belongs to epoch } i\}$ ,  $\iota = \frac{|S||A|T}{\delta}$ , and we define the following quantities:
$$
T _ {i} (s, a) = \left\{t \in \cup_ {j = 1} ^ {i - 1} E _ {j}: \exists k \text {s u c h t h a t} \left(s _ {t, k}, a _ {t, k}\right) = (s, a) \right\} \quad \forall (s, a) \in S _ {k} \times A, \forall i \geq 2, \forall k <   L;
$$
$$
C _ {t} ^ {\mathsf {P}} (s, a, s ^ {\prime}) = | P (s ^ {\prime} | s, a) - P _ {t} (s ^ {\prime} | s, a) |, \quad \forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i \in [ T ], \forall k <   L;
$$
$$
C _ {i} ^ {\mathsf {P}} (s, a, s ^ {\prime}) = \sum_ {t \in T _ {i} (s, a)} C _ {t} ^ {\mathsf {P}} (s, a, s ^ {\prime}), \quad \forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i \in [ T ], \forall k <   L.
$$
Note that based on definition of  $T_{i}(s,a)$ , we have  $m_{i}(s,a) = |T_{i}(s,a)|$ . Then, we present the following lemma which shows the concentration bound between  $P(s'|s,a)$  and  $\bar{P}_i(s'|s,a)$ .
Lemma D.2.1. (Detailed restatement of Lemma 2.2) Event  $\mathcal{E}_{\mathrm{CON}}$  occurs with probability at least  $1 - \delta$  where,
$$
\mathcal {E} _ {\text {C O N}} := \left\{\forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i \in [ T ], \forall k <   L: \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq B _ {i} (s, a, s ^ {\prime}) \right\}, \tag {59}
$$
and  $B_{i}(s,a,s^{\prime})$  is defined in Eq. (4) as:
$$
B _ {i} (s, a, s ^ {\prime}) = \min  \left\{1, 1 6 \sqrt {\frac {\bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log (\iota)}{m _ {i} (s , a)}} + 6 4 \frac {\left(C ^ {P} + \log (\iota)\right)}{m _ {i} (s , a)} \right\}. \tag {60}
$$
Proving this lemma requires several auxiliary results stated below. For any episode  $t \in [T]$  and any layer  $k < L$ , we use  $s_{t,k}^{\mathrm{IMG}}$  to denote an imaginary random state sampled from  $P(\cdot | s_{t,k}, a_{t,k})$ . For any  $(s, a, s') \in W_k$ , let
$$
\bar {P} _ {i} ^ {\mathrm {I M G}} (s ^ {\prime} | s, a) = \frac {1}{m _ {i} (s , a)} \sum_ {t \in T _ {i} (s, a)} \mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \}.
$$
We now proceed with a couple lemmas.
Lemma D.2.2 (Lemma 2, [Jin et al., 2020]). Event  $\mathcal{E}^1$  occurs with probability at least  $1 - 3\delta /4$  where
$$
\mathcal {E} ^ {1} := \left\{\forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i, k: \left| P (s ^ {\prime} | s, a) - \bar {P} _ {i} ^ {\mathrm {I M G}} (s ^ {\prime} | s, a) \right| \leq \bar {\omega} _ {i} (s, a, s ^ {\prime}) \right\},
$$
and  $\bar{\omega}_i(s,a,s')$  for any  $(s,a,s')\in W_k$  and  $0\leq k\leq L - 1$  is defined as
$$
\bar {\omega} _ {i} (s, a, s ^ {\prime}) = \min  \left\{1, 2 \sqrt {\frac {\bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log \iota}{m _ {i} (s , a)}} + \frac {1 4 \log \iota}{3 m _ {i} (s , a)} \right\}. \tag {61}
$$
We note that as long as  $|A|T \geq 16/3$ , event  $\mathcal{E}^1$  can occur with probability at least  $1 - 3\delta / 4$  (unlike  $1 - 4\delta$  in Lemma 2 of Jin et al. [2020]).
Lemma D.2.3. Event  $\mathcal{E}^2$  occurs with probability at least  $1 - \delta /4$  where
$$
\mathcal {E} ^ {2} := \left\{\forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i, k: \left| \bar {P} _ {i} ^ {\mathrm {I M G}} \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq \omega_ {i} (s, a, s ^ {\prime}) \right\},
$$
and  $\omega_{i}(s,a,s^{\prime})$  for any  $(s,a,s^{\prime})\in W_{k}$  and  $0\leq k\leq L - 1$  is defined as
$$
\omega_ {i} (s, a, s ^ {\prime}) = \min  \left\{1, \frac {4 C _ {i} ^ {P} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \sqrt {\frac {2 4 P (s ^ {\prime} \mid s , a) \log \iota}{m _ {i} (s , a)}} + \frac {6 \log \iota}{m _ {i} (s , a)} \right\}. \tag {62}
$$
Proof. For any fixed  $(s, a, s')$ , if  $m_i(s, a) = 0$ , the claimed bound in  $\mathcal{E}^2$  holds trivially, so we consider the case  $m_i(s, a) \neq 0$  below. By definition we have:
$$
\bar {P} _ {i} ^ {\mathrm {I M G}} (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) = \frac {1}{m _ {i} (s , a)} \sum_ {t \in T _ {i} (s, a)} \left(\mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \} - \mathbb {I} \{s _ {t, k (s) + 1} = s ^ {\prime} \}\right).
$$
Then, we construct the martingale difference sequence  $\{X_{t}(s,a,s^{\prime})\}_{t = 1}^{\infty}$  w.r.t. filtration  $\{\mathcal{F}_{t,k(s)}\}_{t = 1}^{\infty}$  (see [Lykouris et al., 2019, Definition 4.9] for the formal definition of these filtrations) where
$$
X _ {t} (s, a, s ^ {\prime}) = \mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \} - \mathbb {I} \{s _ {t, k (s) + 1} = s ^ {\prime} \} - \left(P (s ^ {\prime} | s, a) - P _ {t} (s ^ {\prime} | s, a)\right).
$$
With the definition of  $X_{t}(s,a,s^{\prime})$ , one can show
$$
\begin{array}{l} \sum_ {t \in T _ {i} (s, a)} \mathbb {E} \left[ X _ {t} (s, a, s ^ {\prime}) ^ {2} | \mathcal {F} _ {t, k (s)} \right] \\ \leq \sum_ {t \in T _ {i} (s, a)} \mathbb {E} \left[ \left(\mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \} - \mathbb {I} \{s _ {t, k (s) + 1} = s ^ {\prime} \} - \left(P \left(s ^ {\prime} | s, a\right) - P _ {t} \left(s ^ {\prime} | s, a\right)\right)\right) ^ {2} \mid \mathcal {F} _ {t, k (s)} \right] \\ \leq \sum_ {t \in T _ {i} (s, a)} \mathbb {E} \left[ 2 \left(\mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \} - \mathbb {I} \{s _ {t, k (s) + 1} = s ^ {\prime} \}\right) ^ {2} + 2 \left(P \left(s ^ {\prime} | s, a\right) - P _ {t} \left(s ^ {\prime} | s, a\right)\right) ^ {2} \mid \mathcal {F} _ {t, k (s)} \right] \\ \leq \sum_ {t \in T _ {i} (s, a)} \mathbb {E} \left[ 2 \mathbb {I} \{s _ {t, k (s) + 1} ^ {\mathrm {I M G}} = s ^ {\prime} \} + 2 \mathbb {I} \{s _ {t, k (s) + 1} = s ^ {\prime} \} + 2 C _ {t} ^ {\mathsf {P}} (s, a, s ^ {\prime}) \mid \mathcal {F} _ {t, k (s)} \right] \\ = 2 \sum_ {t \in T _ {i} (s, a)} \left(P \left(s ^ {\prime} | s, a\right) + P _ {t} \left(s ^ {\prime} | s, a\right) + C _ {t} ^ {\mathsf {P}} (s, a, s ^ {\prime})\right), \tag {63} \\ \end{array}
$$
where the second step uses  $(x - y)^2 \leq 2(x^2 + y^2)$  for any  $x, y \in \mathbb{R}$ ; the third step uses  $(x - y)^2 \leq x^2 + y^2$  for  $x, y \in \mathbb{R}_{\geq 0}$  and the fact that  $C_t^{\mathsf{P}}(s, a, s') \in [0,1]$ , thereby  $C_t^{\mathsf{P}}(s, a, s')^2 \leq C_t^{\mathsf{P}}(s, a, s')$ ; the last step holds based on the definitions of  $T_i(s, a)$ ,  $s_{t,k(s) + 1}^{\mathrm{IMG}}$ , and  $s_{t,k(s) + 1}$  as well as the fact that  $C_t^{\mathsf{P}}(s, a, s')$  is  $\mathcal{F}_{t,k(s)}$ -measurable.
By using the result in Eq. (63), we bound the average second moment  $\sigma^2$  as
$$
\sigma^ {2} = \frac {\sum_ {t \in T _ {i} (s , a)} \mathbb {E} \left[ X _ {t} ^ {2} \mid \mathcal {F} _ {t , k (s)} \right]}{m _ {i} (s , a)} \leq \frac {2 \sum_ {t \in T _ {i} (s , a)} \left(P \left(s ^ {\prime} \mid s , a\right) + P _ {t} \left(s ^ {\prime} \mid s , a\right) + C _ {t} ^ {\mathsf {P}} \left(s , a , s ^ {\prime}\right)\right)}{m _ {i} (s , a)}. \tag {64}
$$
By applying Lemma D.2.4 with  $b = 2$  and the upper bound of  $\sigma^2$  shown in Eq. (64), as well as using the fact that  $m_i(s, a) = |T_i(s, a)|$ , for any  $(s, a, s')$ , we have the following with probability at least  $1 - \delta / (4T|S|^2|A|)$ ,
$$
\begin{array}{l} | \bar {P} _ {i} ^ {\mathrm {I M G}} (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) | \\ \leq \left| P \left(s ^ {\prime} \mid s, a\right) - \frac {\sum_ {t \in T _ {i} (s , a)} P _ {t} \left(s ^ {\prime} \mid s , a\right)}{m _ {i} (s , a)} \right| + \frac {4 \log \left(\frac {8 m _ {i} (s , a) T | S | ^ {2} | A |}{\delta}\right)}{3 m _ {i} (s , a)} \\ + \sqrt {\frac {4 \log \left(\frac {1 6 m _ {i} (s , a) ^ {2} T | S | ^ {2} | A |}{\delta}\right) \sum_ {t \in T _ {i} (s , a)} \left(P \left(s ^ {\prime} | s , a\right) + P _ {t} \left(s ^ {\prime} | s , a\right) + C _ {t} ^ {\mathsf {P}} \left(s , a , s ^ {\prime}\right)\right)}{m _ {i} ^ {2} (s , a)}} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left| P (s ^ {\prime} | s, a) - \frac {\sum_ {t \in T _ {i} (s , a)} P _ {t} (s ^ {\prime} | s , a)}{m _ {i} (s , a)} \right| + \frac {8 \log \iota}{3 m _ {i} (s , a)} \\ + \sqrt {\frac {1 2 \log \iota \sum_ {t \in T _ {i} (s , a)} \left(P \left(s ^ {\prime} \mid s , a\right) + P _ {t} \left(s ^ {\prime} \mid s , a\right) + C _ {t} ^ {\mathbb {P}} \left(s , a , s ^ {\prime}\right)\right)}{m _ {i} ^ {2} (s , a)}} \\ \leq \left| \frac {\sum_ {t \in T _ {i} (s , a)} P \left(s ^ {\prime} \mid s , a\right)}{m _ {i} (s , a)} - \frac {\sum_ {t \in T _ {i} (s , a)} P _ {t} \left(s ^ {\prime} \mid s , a\right)}{m _ {i} (s , a)} \right| + \frac {8 \log \iota}{3 m _ {i} (s , a)} \\ + \sqrt {\frac {1 2 \log \iota \sum_ {t \in T _ {i} (s , a)} \left(P \left(s ^ {\prime} | s , a\right) + P \left(s ^ {\prime} | s , a\right) + C _ {t} ^ {\mathsf {P}} \left(s , a , s ^ {\prime}\right) + C _ {t} ^ {\mathsf {P}} \left(s , a , s ^ {\prime}\right)\right)}{m _ {i} ^ {2} (s , a)}} \\ \leq \frac {C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \sqrt {\frac {2 4 P (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \frac {8 \log \iota}{3 m _ {i} (s , a)} + \frac {\sqrt {2 4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime}) \log \iota}}{m _ {i} (s , a)} \\ \leq \frac {C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \sqrt {\frac {2 4 P (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \frac {8 \log \iota}{3 m _ {i} (s , a)} + \frac {\sqrt {6} \left(C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime}) + \log \iota\right)}{m _ {i} (s , a)} \\ \leq \omega_ {i} (s, a, s ^ {\prime}), \\ \end{array}
$$
where the first inequality applies Lemma D.2.4; the second inequality bounds all logarithmic terms by  $\log \iota$  with an appropriate constant factor (using  $m_{i}(s,a)\leq T$  and  $|A|\geq 2$ ); the third step follows the fact that  $P_{t}(s^{\prime}|s,a)\leq P(s^{\prime}|s,a) + C_{t}^{\mathsf{P}}(s,a,s^{\prime})$ ; the fourth step uses the definition  $C_i^\mathsf{P}(s,a,s') = \sum_{t\in T_i(s,a)}C_t^\mathsf{P}(s,a,s')$ ; the fifth step applies the inequality  $\sqrt{4xy}\leq x + y,\forall x,y\geq 0$  for  $x = C_i^\mathsf{P}(s,a,s')$  and  $y = \log \iota$ . Applying a union bound over all  $(s,a,s')$  and epochs  $i\leq T$ , we complete the proof.
Lemma D.2.4 (Anytime Version of Azuma-Bernstein). Let  $\{X_i\}_{i=1}^{\infty}$  be  $b$ -bounded martingale difference sequence with respect to  $\mathcal{F}_i$ . Let  $\sigma^2 = \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}[X_i^2 | F_{i-1}]$ . Then, with probability at least  $1 - \delta$ , for any  $N \in \mathbb{N}^+$ , it holds that:
$$
\left| \frac {1}{N} \sum_ {i = 1} ^ {N} X _ {i} \right| \leq \sqrt {\frac {2 \sigma^ {2} \log (4 N ^ {2} / \delta)}{N}} + \frac {2 b \log (2 N / \delta)}{3 N}.
$$
Proof. This follows the same argument as Lemma G.2 in [Lykouris et al., 2019].  $\square$
Lemma D.2.5. Event  $\mathcal{E}$  occurs with probability at least  $1 - \delta$ , where