SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {E} := \left\{\forall (s, a, s ^ {\prime}) \in W _ {k}, \forall i, k: \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq \omega_ {i} (s, a, s ^ {\prime}) + \bar {\omega} _ {i} (s, a, s ^ {\prime}) \right\}.
$$
Proof. Conditioning on events  $\mathcal{E}^1$  and  $\mathcal{E}^2$ , we have
$$
\begin{array}{l} \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} ^ {\mathrm {I M G}} \left(s ^ {\prime} \mid s, a\right) \right| + \left| \bar {P} _ {i} ^ {\mathrm {I M G}} \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \\ \leq \bar {\omega} _ {i} (s, a, s ^ {\prime}) + \omega_ {i} (s, a, s ^ {\prime}). \tag {65} \\ \end{array}
$$
Using a union bound for  $\mathcal{E}^1$  and  $\mathcal{E}^2$ , we complete the proof.
Armed with above results, we are now ready to prove Lemma D.2.1.
Proof of Lemma D.2.1. Conditioning on event  $\mathcal{E}$ , for any fixed  $(s, a, s')$  with  $m_i(s, a) \neq 0$  (otherwise the desired bound holds trivially), we have
$$
\begin{array}{l} \omega_ {i} (s, a, s ^ {\prime}) \\ \leq \sqrt {\frac {2 4 P (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \frac {6 \log \iota}{m _ {i} (s , a)} + \frac {4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} \\ \leq \sqrt {\frac {2 4 \left(\bar {P} _ {i} (s ^ {\prime} | s , a) + \bar {\omega} _ {i} (s , a , s ^ {\prime}) + \frac {4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \omega_ {i} (s , a , s ^ {\prime})\right) \log \iota}{m _ {i} (s , a)}} + \frac {6 \log \iota}{m _ {i} (s , a)} + \frac {4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)}} \\ \end{array}
$$
$$
\begin{array}{l} \leq \sqrt {\frac {2 4 \bar {P} _ {i} (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \sqrt {\frac {2 4 \bar {\omega} _ {i} (s , a , s ^ {\prime}) \log \iota}{m _ {i} (s , a)}} \\ + \sqrt {\frac {9 6 \frac {C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} \log \iota}{m _ {i} (s , a)}} + \sqrt {\frac {2 4 \omega_ {i} (s , a , s ^ {\prime}) \log \iota}{m _ {i} (s , a)}} + \frac {6 \log \iota}{m _ {i} (s , a)} + \frac {4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} \\ \leq \sqrt {\frac {2 4 \bar {P} _ {i} (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \bar {\omega} _ {i} (s, a, s ^ {\prime}) + \frac {\sqrt {9 6 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime}) \log \iota}}{m _ {i} (s , a)} + \frac {\omega_ {i} (s , a , s ^ {\prime})}{2} + \frac {2 4 \log \iota + 4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} \\ \leq \sqrt {\frac {2 4 \bar {P} _ {i} (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \bar {\omega} _ {i} (s, a, s ^ {\prime}) + \frac {2 8 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \frac {\omega_ {i} (s , a , s ^ {\prime})}{2} + \frac {2 5 \log \iota}{m _ {i} (s , a)}, \\ \end{array}
$$
where the second step holds under  $\mathcal{E}$ ; the third step uses  $\sqrt{\sum_{i=1}^{n} x_i} \leq \sum_{i=1}^{n} \sqrt{x_i}$  for all  $x_i \in \mathbb{R}_{\geq 0}$ ; the fourth step and the fifth step use  $2\sqrt{xy} \leq x + y$  for  $x, y \geq 0$ . Rearranging the above, we obtain
$$
\omega_ {i} (s, a, s ^ {\prime}) \leq \sqrt {\frac {9 6 \bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log \iota}{m _ {i} (s , a)}} + 2 \bar {\omega} _ {i} (s, a, s ^ {\prime}) + \frac {5 6 C _ {i} ^ {\mathrm {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \frac {5 0 \log \iota}{m _ {i} (s , a)} \tag {66}
$$
Thus, conditioning on event  $\mathcal{E}$ , one can show for all  $(s,a,s')\in W_k$  and  $k < L - 1$
$$
\begin{array}{l} \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq \omega_ {i} (s, a, s ^ {\prime}) + \bar {\omega} _ {i} (s, a, s ^ {\prime}) \\ \leq \sqrt {\frac {9 6 \bar {P} _ {i} (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + 3 \bar {\omega} _ {i} (s, a, s ^ {\prime}) + \frac {5 6 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + \frac {5 0 \log \iota}{m _ {i} (s , a)} \\ \leq 1 6 \sqrt {\frac {\bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log \iota}{m _ {i} (s , a)}} + \frac {5 6 C _ {i} ^ {\mathsf {P}} \left(s , a , s ^ {\prime}\right)}{m _ {i} (s , a)} + \frac {6 4 \log \iota}{m _ {i} (s , a)} \\ \leq 1 6 \sqrt {\frac {\bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log \iota}{m _ {i} (s , a)}} + 6 4 \frac {C _ {i} ^ {\mathrm {P}} \left(s , a , s ^ {\prime}\right) + \log \iota}{m _ {i} (s , a)}, \tag {67} \\ \end{array}
$$
where the second step uses Eq. (66), the third step applies the definition of  $\bar{\omega}_i(s,a,s')$ . Finally, using the fact  $C_i^{\mathsf{P}}(s,a,s') \leq C^{\mathsf{P}}$ , we complete the proof.
Then, we present an immediate corollary of Lemma D.2.1.
Corollary D.2.6. Consider any epoch  $i$  and any transition  $P' \in \mathcal{P}_i$ . The following holds (recall  $\widehat{m}_i$  defined at the beginning of the appendix),
$$
\| P ^ {\prime} (\cdot | s, a) - \bar {P} _ {i} (\cdot | s, a) \| _ {1} \leq 2 \cdot \min  \left\{1, \frac {3 2 C ^ {P}}{\widehat {m} _ {i} (s , a)} + 8 \sqrt {\frac {| S _ {k (s) + 1} | \log \iota}{\widehat {m} _ {i} (s , a)}} + \frac {3 2 | S _ {k (s) + 1} | \log \iota}{\widehat {m} _ {i} (s , a)} \right\}.
$$
Proof. As  $P' \in \mathcal{P}_i$ , we start from Eq. (67):
$$
\begin{array}{l} \| P ^ {\prime} (\cdot | s, a) - \bar {P} _ {i} (\cdot | s, a) \| _ {1} \leq \sum_ {s ^ {\prime} \in k (s) + 1} \left(\frac {6 4 C _ {i} ^ {\mathsf {P}} (s , a , s ^ {\prime})}{m _ {i} (s , a)} + 1 6 \sqrt {\frac {\bar {P} _ {i} (s ^ {\prime} | s , a) \log \iota}{m _ {i} (s , a)}} + \frac {6 4 \log \iota}{m _ {i} (s , a)}\right) \\ \leq \frac {6 4 C ^ {\mathsf {P}}}{m _ {i} (s , a)} + 1 6 \sqrt {\frac {| S _ {k (s) + 1} | \log \iota}{m _ {i} (s , a)}} + \frac {6 4 | S _ {k (s) + 1} | \log \iota}{m _ {i} (s , a)}, \tag {68} \\ \end{array}
$$
where the last step uses the Cauchy-Schwarz inequality and the fact that  $\sum_{s' \in k(s) + 1} C_i^{\mathsf{P}}(s, a, s') \leq C^{\mathsf{P}}$ . Since  $\| P(\cdot | s, a) - \bar{P}_i(\cdot | s, a)\|_1 \leq 2$ , we combine this trivial bound and the bound of  $\| P(\cdot | s, a) - \bar{P}_i(\cdot | s, a)\|_1$  in Eq. (68) to arrive at
$$
\| P ^ {\prime} (\cdot | s, a) - \bar {P} _ {i} (\cdot | s, a) \| _ {1} \leq 2 \cdot \min  \left\{1, \frac {3 2 C ^ {\mathsf {P}}}{m _ {i} (s , a)} + 8 \sqrt {\frac {| S _ {k (s) + 1} | \log \iota}{m _ {i} (s , a)}} + \frac {3 2 | S _ {k (s) + 1} | \log \iota}{m _ {i} (s , a)} \right\}
$$
$$
= 2 \cdot \min  \left\{1, \frac {3 2 C ^ {\mathsf {P}}}{\widehat {m} _ {i} (s , a)} + 8 \sqrt {\frac {| S _ {k (s) + 1} | \log \iota}{\widehat {m} _ {i} (s , a)}} + \frac {3 2 | S _ {k (s) + 1} | \log \iota}{\widehat {m} _ {i} (s , a)} \right\},
$$
We conclude this subsection with two other useful lemmas.
Lemma D.2.7. Conditioning on event  $\mathcal{E}_{\mathrm{CON}}$ , it holds for all tuple  $(s, a, s')$  and epoch  $i$  that
$$
\left| P (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) \right| \leq \mathcal {O} \left(\min  \left\{1, \sqrt {\frac {P (s ^ {\prime} | s , a) \log (\iota)}{\widehat {m} _ {i} (s , a)}} + \frac {C ^ {\mathcal {P}} + \log (\iota)}{\widehat {m} _ {i} (s , a)} \right\}\right).
$$
Proof. Fix the epoch  $i$  and tuple  $(s, a, s')$ . According to the definitions of  $\mathcal{E}_{\mathrm{CON}}$  in Eq. (59) and  $\widehat{m}$ , we have
$$
\left| P (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) \right| \leq \min  \left\{1, 1 6 \sqrt {\frac {\bar {P} _ {i} (s ^ {\prime} | s , a) \log (\iota)}{\hat {m} _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\hat {m} _ {i} (s , a)} \right\}.
$$
Therefore, by direct calculation, we have
$$
\begin{array}{l} \left| P (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) \right| \\ \leq 1 6 \sqrt {\frac {\left(\left| P \left(s ^ {\prime} \mid s , a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \right| + P \left(s ^ {\prime} \mid s , a\right)\right) \log (\iota)}{\hat {m} _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\hat {m} _ {i} (s , a)} \\ \leq 8 \left(\sqrt {\frac {P (s ^ {\prime} | s , a) \log (\iota)}{\widehat {m} _ {i} (s , a)}} + \sqrt {\frac {\left| P (s ^ {\prime} | s , a) - \bar {P} _ {i} (s ^ {\prime} | s , a) \right| \log (\iota)}{\widehat {m} _ {i} (s , a)}}\right) + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i} (s , a)} \\ = 8 \sqrt {\frac {P (s ^ {\prime} | s , a) \log (\iota)}{\widehat {m} _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i} (s , a)} + \sqrt {\left| P (s ^ {\prime} | s , a) - \bar {P} _ {i} (s ^ {\prime} | s , a) \right| \cdot \frac {6 4 \log (\iota)}{\widehat {m} _ {i} (s , a)}} \\ \leq 8 \sqrt {\frac {P (s ^ {\prime} | s , a) \log (\iota)}{\widehat {m} _ {i} (s , a)}} + 9 6 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i} (s , a)} + \frac {1}{2} \left| P (s ^ {\prime} | s, a) - \bar {P} _ {i} (s ^ {\prime} | s, a) \right|, \\ \end{array}
$$
where the second step and last step follow from the fact that  $\sqrt{xy} \leq \frac{1}{2}(x + y)$  for any  $x, y \geq 0$ . Finally, rearranging the above inequality finishes the proof.
Lemma D.2.8. (Lower Bound of Upper Occupancy Measure) For any episode  $t$  and state  $s \neq s_L$ , it always holds that  $u_t(s) \geq 1 / |s|T$ .
Proof. Fix the episode  $t$  and state  $s$ . We prove the lemma by constructing a specific transition  $\widehat{P} \in \mathcal{P}_{i(t)}$ , such that  $q^{\widehat{P},\pi}(s) \geq 1 / |S|T$  for any policy  $\pi$ , which suffices due to the definition of  $u_{t}(s)$ .
Specifically,  $\widehat{P}$  is defined as, for any tuple  $(s,a,s')\in W_k$  and  $k = 0,\dots ,L - 1$ :
$$
\widehat {P} (s ^ {\prime} | s, a) = \bar {P} _ {i (t)} (s ^ {\prime} | s, a) \cdot \left(1 - \frac {1}{T}\right) + \frac {1}{| S _ {k + 1} | T}.
$$
By direct calculation, one can verify that  $\widehat{P}$  is a valid transition function. Then, we show that  $\widehat{P} \in \mathcal{P}_{i(t)}$  by verifying the condition for any transition tuple  $(s,a,s') \in W_k$  and  $k = 0,\dots,L-1$ :
$$
\left| \widehat {P} (s ^ {\prime} | s, a) - \bar {P} _ {i (t)} (s ^ {\prime} | s, a) \right| = \frac {1}{T} \cdot \left| \bar {P} _ {i (t)} (s ^ {\prime} | s, a) - \frac {1}{| S _ {k + 1} |} \right| \leq \frac {1}{T} \leq B _ {i (t)} (s, a, s ^ {\prime}),
$$
where the last step follows from the definition of confidence intervals in Eq. (4).
Finally, we show that  $q^{\widehat{P},\pi}(s)\geq 1 / |S|T$  as:
$$
q ^ {\widehat {P}, \pi} (s) = \sum_ {u \in S _ {k (s)} - 1} \sum_ {v \in A} q ^ {\widehat {P}, \pi} (u, v) \widehat {P} (s | u, v) \geq \sum_ {u \in S _ {k (s)} - 1} \sum_ {v \in A} q ^ {\widehat {P}, \pi} (u, v) \cdot \frac {1}{T | S |} = \frac {1}{T | S |},
$$
D.3 Difference Lemma
Lemma D.3.1 (Theorem 5.2.1 of [Kakade, 2003]). (Performance Difference Lemma) For any policies  $\pi_1, \pi_2$  and any loss function  $\ell: S \times A \to \mathbb{R}$ ,
$$
\begin{array}{l} V ^ {P, \pi_ {1}} (s _ {0}; \ell) - V ^ {P, \pi_ {2}} (s _ {0}; \ell) \\ = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {2}} (s, a) \left(V ^ {P, \pi_ {1}} (s; \ell) - Q ^ {P, \pi_ {1}} (s, a; \ell)\right) \\ = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {2}} (s) \left(\pi_ {1} (a | s) - \pi_ {2} (a | s)\right) Q ^ {P, \pi_ {1}} (s, a; \ell). \\ \end{array}
$$
In fact, the same also holds for our optimistic transition where the layer structure is violated. For completeness, we include a proof below.
Lemma D.3.2. For any policies  $\pi_1, \pi_2$ , any loss function  $\ell: S \times A \to \mathbb{R}$ , and any transition  $\widetilde{P}$  where  $\sum_{s' \in S_{k(s)} + 1} \widetilde{P}(s'|s, a) \leq 1$  for all state-action pairs  $(s, a)$  (the remaining probability is assigned to  $s_L$ ), we have
$$
V ^ {\widetilde {P}, \pi_ {1}} (s _ {0}; \ell) - V ^ {\widetilde {P}, \pi_ {2}} (s _ {0}; \ell) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi_ {2}} (s, a) \left(V ^ {\widetilde {P}, \pi_ {1}} (s; \ell) - Q ^ {\widetilde {P}, \pi_ {1}} (s, a; \ell)\right).
$$
Proof. By direct calculation, we have for any state  $s \neq s_L$ :
$$
\begin{array}{l} V ^ {\widetilde {P}, \pi_ {1}} (s _ {0}; \ell) - V ^ {\widetilde {P}, \pi_ {2}} (s _ {0}; \ell) = \left(\sum_ {a \in A} \pi_ {2} (a | s _ {0})\right) V ^ {\widetilde {P}, \pi_ {1}} (s _ {0}; \ell) - \sum_ {a \in A} \pi_ {2} (a | s _ {0}) Q ^ {\widetilde {P}, \pi_ {1}} (s _ {0}, a; \ell) \\ + \sum_ {a \in A} \pi_ {2} (a | s _ {0}) \left(Q ^ {\tilde {P}, \pi_ {1}} (s _ {0}, a; \ell) - Q ^ {\tilde {P}, \pi_ {2}} (s _ {0}, a; \ell)\right) \\ = \sum_ {a \in A} q ^ {\widetilde {P}, \pi_ {2}} (s _ {0}, a) \left(V ^ {\widetilde {P}, \pi_ {1}} (s _ {0}; \ell) - Q ^ {\widetilde {P}, \pi_ {1}} (s _ {0}, a; \ell)\right) \\ + \sum_ {a \in A} \sum_ {s ^ {\prime} \in S _ {1}} \pi_ {2} (a | s _ {0}) \widetilde {P} (s ^ {\prime} | s _ {0}, a) \left(V ^ {\widetilde {P}, \pi_ {1}} (s ^ {\prime}; \ell) - V ^ {\widetilde {P}, \pi_ {2}} (s ^ {\prime}; \ell)\right) \\ = \sum_ {a \in A} q ^ {\widetilde {P}, \pi_ {2}} (s _ {0}, a) \left(V ^ {\widetilde {P}, \pi_ {1}} (s _ {0}; \ell) - Q ^ {\widetilde {P}, \pi_ {1}} (s _ {0}, a; \ell)\right) \\ + \sum_ {a \in A} \sum_ {s ^ {\prime} \in S _ {1}} q ^ {\widetilde {P}, \pi_ {2}} (s ^ {\prime}) \left(V ^ {\widetilde {P}, \pi_ {1}} (s ^ {\prime}; \ell) - V ^ {\widetilde {P}, \pi_ {2}} (s ^ {\prime}; \ell)\right) \\ = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi_ {2}} (s, a) \left(V ^ {\widetilde {P}, \pi_ {1}} (s; \ell) - Q ^ {\widetilde {P}, \pi_ {1}} (s, a; \ell)\right), \\ \end{array}
$$
where the last step follows from recursively repeating the first three steps.
Lemma D.3.3. (Occupancy Measure Difference, [Jin et al., 2021, Lemma D.3.1]) For any transition functions  $P_{1}, P_{2}$  and any policy  $\pi$ ,
$$
\begin{array}{l} q ^ {P _ {1}, \pi} (s) - q ^ {P _ {2}, \pi} (s) = \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P _ {1}, \pi} (u, v) \left(P _ {1} (w | u, v) - P _ {2} (w | u, v)\right) q ^ {P _ {2}, \pi} (s | w) \\ = \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P _ {2}, \pi} (u, v) \left(P _ {1} (w | u, v) - P _ {2} (w | u, v)\right) q ^ {P _ {1}, \pi} (s | w), \\ \end{array}
$$
where  $q^{P',\pi}(s|w)$  is the probability of visiting  $s$  starting  $w$  under policy  $\pi$  and transition  $P'$ .
Lemma D.3.4. ([Dann et al., 2023, Lemma 17]) For any policies  $\pi_1, \pi_2$  and transition function  $P$ ,
$$
\sum_ {s \neq s _ {L}} \sum_ {a \in A} \left| q ^ {P, \pi_ {1}} (s, a) - q ^ {P, \pi_ {2}} (s, a) \right| \leq L \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {1}} (s) | \pi_ {1} (a | s) - \pi_ {2} (a | s) |.
$$
Corollary D.3.5. For any policies  $\pi_1$ , mapping  $\pi_2: S \to A$  (that is, a deterministic policy), and transition function  $P$ , we have
$$
\sum_ {s \neq s _ {L}} \sum_ {a \in A} \left| q ^ {P, \pi_ {1}} (s, a) - q ^ {P, \pi_ {2}} (s, a) \right| \leq 2 L \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi_ {2} (s)} q ^ {P, \pi_ {1}} (s, a).
$$
Proof. According to Lemma D.3.4, we have
Note that, for every state  $s$ , it holds that
$$
\sum_ {a \in A} | \pi_ {1} (a | s) - \pi_ {2} (a | s) | = \sum_ {a \neq \pi_ {2} (s)} \pi_ {1} (a | s) + | \pi_ {1} (\pi_ {2} (s) | s) - 1 | = 2 \sum_ {a \neq \pi_ {2} (s)} \pi_ {1} (a | s),
$$
where the first step follows from the fact that  $\pi_2(a|s) = 1$  when  $a = \pi_2(s)$ , and  $\pi_2(b|s) = 0$  for any other action  $b \neq \pi_2(s)$ .
$$
L \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {1}} (s) | \pi_ {1} (a | s) - \pi_ {2} (a | s) | \leq 2 L \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi_ {2} (s)} q ^ {P, \pi_ {1}} (s, a),
$$
According to Lemma D.3.3, we can estimate the occupancy measure difference caused by the corrupted transition function  $P_{t}$  at episode  $t$ .
Corollary D.3.6. For any episode  $t$  and any policy  $\pi$ , we have
$$
\left| q ^ {P, \pi} (s) - q ^ {P _ {t}, \pi} (s) \right| \leq C _ {t} ^ {\mathcal {P}}, \quad \forall s \neq s _ {L}, \quad a n d \quad \sum_ {s \neq s _ {L}} \left| q ^ {P, \pi} (s) - q ^ {P _ {t}, \pi} (s) \right| \leq L C _ {t} ^ {\mathcal {P}}.
$$
Proof. By direct calculation, we have for any episode  $t$  and any  $s \neq s_L$
$$
\begin{array}{l} \left| q ^ {P, \pi} (s) - q ^ {P _ {t}, \pi} (s) \right| \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi} (u, v) | P (w | u, v) - P _ {t} (w | u, v) | q ^ {P _ {t}, \pi} (s | w) \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {u \in S _ {k}} \sum_ {v \in A} q ^ {P, \pi} (u, v) \| P (\cdot | u, v) - P _ {t} (\cdot | u, v) \| _ {1} \\ \leq C _ {t} ^ {\mathsf {P}}, \\ \end{array}
$$
where the first step follows from Lemma D.3.3, the second step bounds  $q^{P_t,\pi}(s|w) \leq 1$ , and the last two steps follow from the definition of  $C_t^{\mathsf{P}}$ .
Moreover, taking the summation over all states  $s \neq s_L$ , we have
$$
\begin{array}{l} \sum_ {s \neq s _ {L}} \left| q ^ {P, \pi} (s) - q ^ {P _ {t}, \pi} (s) \right| \\ \leq \sum_ {s \neq s _ {L}} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi} (u, v) | P (w | u, v) - P _ {t} (w | u, v) | q ^ {P _ {t}, \pi} (s | w) \\ = \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi} (u, v) | P (w | u, v) - P _ {t} (w | u, v) | \sum_ {h = k + 1} ^ {L - 1} \sum_ {s \in S _ {h}} q ^ {P _ {t}, \pi} (s | w) \\ \leq L \sum_ {k = 0} ^ {k (s) - 1} \sum_ {u \in S _ {k}} \sum_ {v \in A} q ^ {P, \pi} (u, v) \| P (\cdot | u, v) - P _ {t} (\cdot | u, v) \| _ {1} \\ \leq L C _ {t} ^ {\mathsf {P}}, \\ \end{array}
$$
where the third step follows from the fact that  $\sum_{s\in S_h}q^{P_t,\pi}(s|w) = 1$  for any  $h\geq k(w)$ .
Corollary D.3.7. For any policy sequence  $\{\pi_t\}_{t=1}^T$  and loss functions  $\{\ell_t\}_{t=1}^T$  such that  $\ell_t \in S \times A \to [0,1]$  for any  $t \in \{1, \dots, T\}$ , it holds that
$$
\sum_ {t = 1} ^ {T} \left| \left\langle q ^ {P, \pi_ {t}} - q ^ {P _ {t}, \pi_ {t}}, \ell_ {t} \right\rangle \right| \leq L C ^ {\mathcal {P}}.
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left| \left\langle q ^ {P, \pi_ {t}} - q ^ {P _ {t}, \pi_ {t}}, \ell_ {t} \right\rangle \right| \leq \sum_ {t = 1} ^ {T} \left\| q ^ {P, \pi_ {t}} - q ^ {P _ {t}, \pi_ {t}} \right\| _ {1} \\ = \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left| q ^ {P, \pi_ {t}} (s, a) - q ^ {P _ {t}, \pi_ {t}} (s, a) \right| \\ = \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left| q ^ {P, \pi_ {t}} (s) - q ^ {P _ {t}, \pi_ {t}} (s) \right| \pi_ {t} (a | s) \\ = \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \left| q ^ {P, \pi_ {t}} (s) - q ^ {P _ {t}, \pi_ {t}} (s) \right| \\ \leq \sum_ {t = 1} ^ {T} L C _ {t} ^ {\mathsf {P}} = L C ^ {\mathsf {P}}, \\ \end{array}
$$
where the first step applies the Hölder's inequality; the third step follows from the fact that  $q^{P', \pi_t}(s, a) = q^{P', \pi_t}(s) \pi_t(a|s)$  for any state-action pair  $(s, a)$  and any transition function  $P'$ ; the fifth step applies Corollary D.3.6; the last step follows from the definition of  $C^{\mathsf{P}}$ .
Following the same idea in the proof of [Dann et al., 2023, Lemma 16], we also consider a tighter bound of the difference between occupancy measures in the following lemma.
Lemma D.3.8. Suppose the event  $\mathcal{E}_{\mathrm{CON}}$  holds. For any state  $s \neq s_L$ , episode  $t$  and transition function  $P_t' \in \mathcal{P}_{i(t)}$ , we have
$$
\begin{array}{l} \left| q ^ {P _ {t} ^ {\prime}, \pi_ {t}} (s) - q ^ {P, \pi_ {t}} (s) \right| \leq \mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\hat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w)\right) \tag {69} \\ + \mathcal {O} \left(| S | ^ {2} \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {P} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right)\right). \\ \end{array}
$$
Proof. According to Lemma D.3.3, we have
$$
\begin{array}{l} \left| q ^ {P _ {t} ^ {\prime}, \pi_ {t}} (s) - q ^ {P, \pi_ {t}} (s) \right| \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) | P (w | u, v) - P _ {t} ^ {\prime} (w | u, v) | q ^ {P _ {t} ^ {\prime}, \pi_ {t}} (s | w) \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) | P (w | u, v) - P _ {t} ^ {\prime} (w | u, v) | q ^ {P, \pi_ {t}} (s | w) \\ + \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) | P (w | u, v) - P _ {t} ^ {\prime} (w | u, v) | \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {k}} q ^ {P, \pi_ {t}} (x, y | w) | P (z | x, y) - P _ {t} ^ {\prime} (z | x, y) | \\ \leq \mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w)\right) \\ \end{array}
$$
$$
\begin{array}{l} + \underbrace {\mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u , v , w) \in W _ {k}} q ^ {P , \pi_ {t}} (u , v) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}\right) q ^ {P , \pi_ {t}} (s | w)\right)} _ {\text {T E R M (A)}} \\ + \underbrace {\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u , v , w) \in W _ {k}} q ^ {P , \pi_ {t}} (u , v) | P (w | u , v) - P _ {t} ^ {\prime} (w | u , v) | \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x , y , z) \in W _ {h}} q ^ {P , \pi_ {t}} (x , y | w) | P (z | x , y) - P _ {t} ^ {\prime} (z | x , y) |} _ {\text {T E R M (B)}}, \\ \end{array}
$$
where the second step first subtracts and adds  $q^{P_t', \pi_t}(s|w)$  and then applies Lemma D.3.3 again for  $|q^{P_t', \pi_t}(s|w) - q^{P, \pi_t}(s|w)|$ , and the third step follows from Lemma D.2.7.
Clearly, we can bound term (a) as:
$$
\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}\right) q ^ {P, \pi_ {t}} (s | w) \leq | S | \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right).
$$
On the other hand, for term (b), we decompose it into three terms in Eq. (70), Eq. (71), and Eq. (72):
$$
\begin{array}{l} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) | P (w | u, v) - P _ {t} ^ {\prime} (w | u, v) | \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {k}} q ^ {P, \pi_ {t}} (x, y | w) | P (z | x, y) - P _ {t} ^ {\prime} (z | x, y) | \\ \leq \mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) \sqrt {\frac {P (z | x , y) \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}}\right) (70) \\ + \mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}\right)\right) (71) \\ + \mathcal {O} \left(\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\hat {m} _ {i (t)} (u , v)}\right) \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {k}} q ^ {P, \pi_ {t}} (x, y | w)\right). (72) \\ \end{array}
$$
According to the AM-GM inequality, we have the term in Eq. (70) bounded as