SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) \sqrt {\frac {P (z | x , y) \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}} \\ = \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (x, y | w) \sqrt {\frac {P (z | x , y) \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}} \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} \frac {q ^ {P , \pi_ {t}} (u , v) q ^ {P , \pi_ {t}} (x , y | w) P (z | x , y) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \\ + \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} \frac {q ^ {P , \pi_ {t}} (u , v) P (w | u , v) q ^ {P , \pi_ {t}} (x , y | w) \log (\iota)}{\widehat {m} _ {i (t)} (x , y)} \\ = \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \frac {q ^ {P , \pi_ {t}} (u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \left(\sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) P (z | x, y)\right) \\ + \sum_ {h = 0} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} \frac {\log (\iota)}{\widehat {m} _ {i (t)} (x , y)} \left(\sum_ {k = 0} ^ {h - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) P (w | u, v) q ^ {P, \pi_ {t}} (x, y | w)\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq L \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \frac {q ^ {P , \pi_ {t}} (u , v) \log (\iota)}{\hat {m} _ {i (t)} (u , v)} + L \sum_ {h = 0} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} \frac {q ^ {P , \pi_ {t}} (x , y) \log (\iota)}{\hat {m} _ {i (t)} (x , y)} \\ \leq \mathcal {O} \left(L | S | \sum_ {k = 0} ^ {L - 1} \sum_ {s \in S _ {k}} \sum_ {a \in A} \frac {q ^ {P , \pi_ {t}} (s , a) \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right), \\ \end{array}
$$
where the first step follows from re-arranging the summation; the second step applies the fact that  $\sqrt{xy} \leq x + y$  for any  $x, y \geq 0$ ; the third step rearranges the summation order, and the fourth step follows from the facts that  $\sum_{x \in S_h} \sum_{y \in A} q^{P,\pi_t}(x, y|w) P(z|x, y) = q^{P,\pi_t}(z|w)$  and  $\sum_{(u,v,w) \in W_k} q^{P,\pi_t}(u,v) P(w|u,v) q^{P,\pi_t}(x,y|w) = q^{P,\pi_t}(x,y)$  for any  $k$  and  $(x,y,z)$ .
Similarly, we have the term in Eq. (71) bounded as
$$
\begin{array}{l} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}\right) \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} \frac {q ^ {P , \pi_ {t}} (u , v) q ^ {P , \pi_ {t}} (x , y | w) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \\ + \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) P (w | u, v) q ^ {P, \pi_ {t}} (x, y | w) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}\right) \\ \leq \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \frac {q ^ {P , \pi_ {t}} (u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \left(\sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w)\right) \\ + \sum_ {h = 0} ^ {L - 1} \sum_ {(x, y, z) \in W _ {h}} \sum_ {k = 0} ^ {k (s) - 1} \left(\sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) P (w | u, v) q ^ {P, \pi_ {t}} (x, y | w)\right) \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (x , y)} \\ \leq | S | \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} \frac {q ^ {P , \pi_ {t}} (u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} + \sum_ {h = 0} ^ {L - 1} \sum_ {(x, y, z) \in W _ {h}} \sum_ {k = 0} ^ {k (s) - 1} q ^ {P, \pi_ {t}} (x, y) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (x , y)}\right) \\ \leq | S | ^ {2} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \frac {q ^ {P , \pi_ {t}} (s , a) \log (\iota)}{\widehat {m} _ {i (t)} (s , a)} + L | S | \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right), \\ \end{array}
$$
where the first step follows from the fact that  $\widehat{m}_{i(t)}(s,a)\geq C + \log (\iota)$  according to its definition; the third step follows from the facts that  $\sum_{x\in S_h}\sum_{y\in A}q^{P,\pi_t}(x,y|w)\leq 1$  and  $\sum_{(u,v,w)\in W_k}q^{P,\pi_t}(u,v)P(w|u,v)q^{P,\pi_t}(x,y|w) = q^{P,\pi_t}(x,y)$ .
For the term in Eq. (72), we have
$$
\begin{array}{l} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {k}} q ^ {P, \pi_ {t}} (x, y | w) \\ = \sum_ {k = 0} ^ {k (s) - 1} \sum_ {u \in S _ {k}} \sum_ {v \in A} \sum_ {w \in S _ {k + 1}} q ^ {P, \pi_ {t}} (u, v) \frac {C ^ {\mathbf {P}} + \log (\iota)}{\hat {m} _ {i (t)} (u , v)} \left(\sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {z \in S _ {k + 1}} 1\right) \\ \leq |S|^{2}\sum_{u\neq s_{L}}\sum_{v\in A}q^{P,\pi_{t}}(s,a)\left(\frac{C^{\mathsf{P}} + \log(\iota)}{\widehat{m}_{i(t)}(u,v)}\right), \\ \end{array}
$$
according to the fact that  $\sum_{x\in S_h}\sum_{y\in A}q^{P,\pi_t}(x,y|w)\leq 1$
Putting all the bounds for the terms in Eq. (70), Eq. (71), and Eq. (72) together yields the bound of TERM (B) that
$$
\sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) | P (w | u, v) - P _ {t} ^ {\prime} (w | u, v) | \sum_ {h = k + 1} ^ {k (s) - 1} \sum_ {(x, y, z) \in W _ {h}} q ^ {P, \pi_ {t}} (x, y | w) | P (z | x, y) - P _ {t} ^ {\prime} (z | x, y) |
$$
$$
= \mathcal {O} \left(| S | ^ {2} \sum_ {u \neq s _ {L}} \sum_ {v \in A} q ^ {P, \pi_ {t}} (u, v) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}\right)\right).
$$
Combining this bound with that of TERM (A) finishes the proof.
D.4 Loss Shifting Technique with Optimistic Transition
Lemma D.4.1. Fix an optimistic transition function  $\widetilde{P}$  (defined in Section 5). For any policy  $\pi$  and any loss function  $\ell : S \times A \to \mathbb{R}$ , we define function  $g: S \times A \to \mathbb{R}$  similar to that of Jin et al. [2021] as:
$$
g ^ {\widetilde {P}, \pi} (s, a; \ell) \triangleq \left(Q ^ {\widetilde {P}, \pi} (s, a; \ell) - V ^ {\widetilde {P}, \pi} (s; \ell) - \ell (s, a)\right),
$$
where the state-action and state value function  $Q^{\widetilde{P},\pi}$  and  $V^{\widetilde{P},\pi}$  are defined with respect to the optimistic transition  $\widetilde{P}$  and  $\pi$  as following:
$$
Q ^ {\widetilde {P}, \pi} (s, a; \ell) = \ell (s, a) + \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} \widetilde {P} (s ^ {\prime} | s, a) V ^ {\widetilde {P}, \pi} (s ^ {\prime}; \ell),
$$
$$
V ^ {\widetilde {P}, \pi} (s; \ell) = \left\{ \begin{array}{l l} 0, & s = s _ {L}, \\ \sum_ {a \in A} \pi (a | s) Q ^ {\widetilde {P}, \pi} (s, a; \ell), & s \neq s _ {L}. \end{array} \right.
$$
Then, it holds for any policy  $\pi^{\prime}$  that,
$$
\left\langle q ^ {\widetilde {P}, \pi^ {\prime}}, g ^ {\widetilde {P}, \pi} \right\rangle = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi^ {\prime}} (s, a) \cdot g ^ {\widetilde {P}, \pi} (s, a; \ell) = - V ^ {\widetilde {P}, \pi} (s _ {0}; \ell),
$$
where  $V^{\widetilde{P},\pi}(s_0;\ell)$  is only related to  $\widetilde{P}$ ,  $\pi$  and  $\ell$ , and is independent with  $\pi'$ .
Proof. By the extended performance difference lemma of the optimistic transition in Lemma D.3.1, we have the following equality holds for any policy  $\pi'$ :
$$
V ^ {\widetilde {P}, \pi^ {\prime}} (s _ {0}; \ell) - V ^ {\widetilde {P}, \pi} (s _ {0}; \ell) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi^ {\prime}} (s, a) \left(Q ^ {\widetilde {P}, \pi} (s, a; \ell) - V ^ {\widetilde {P}, \pi} (s; \ell)\right).
$$
$$
V ^ {\widetilde {P}, \pi^ {\prime}} (s _ {0}; \ell) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi^ {\prime}} (s, a) \ell (s, a).
$$
Therefore, subtracting  $V^{\widetilde{P},\pi '}(s_0;\ell)$  yields that
$$
- V ^ {\widetilde {P}, \pi} (s _ {0}; \ell) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {\widetilde {P}, \pi^ {\prime}} (s, a) \left(Q ^ {\widetilde {P}, \pi} (s, a; \ell) - V ^ {\widetilde {P}, \pi} (s; \ell) - \ell (s, a)\right).
$$
The proof is finished after using the definition of  $g^{\tilde{P},\pi}$ .
Therefore, we have the following result for the FTRL framework which is similar to [Jin et al., 2021, Corollary A.1.2].
Corollary D.4.2. The FTRL update in Algorithm 4 can be equivalently written as:
$$
\widehat {q} _ {t} = \operatorname * {a r g m i n} _ {q \in \Omega (\widetilde {P} _ {i})} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} (\widehat {\ell_ {\tau}} - b _ {\tau}) \right\rangle + \phi_ {t} (q) = \operatorname * {a r g m i n} _ {x \in \Omega (\widetilde {P} _ {i})} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} (g _ {\tau} - b _ {\tau}) \right\rangle + \phi_ {t} (q),
$$
where  $g_{\tau}(s,a) = Q^{\widetilde{P}_{i},\pi_{\tau}}(s,a;\widehat{\ell}_{\tau}) - V^{\widetilde{P}_{i},\pi_{\tau}}(s;\widehat{\ell}_{\tau})$  for any state-action pair  $(s,a)$ .
D.5 Estimation Error
Lemma D.5.1. ([Jin et al., 2020, Lemma 10]) With probability at least  $1 - \delta$ , we have for all  $k = 0, \ldots, L - 1$ ,
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\max \left\{m _ {i (t)} (s , a) , 1 \right\}} = \mathcal {O} \left(| S _ {k} | | A | \log (T) + \log \left(\frac {L}{\delta}\right)\right),
$$
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\sqrt {\max  \left\{m _ {i (t)} (s , a) , 1 \right\}}} = \mathcal {O} \left(\sqrt {| S _ {k} | | A | T} + | S _ {k} | | A | \log (T) + \log \left(\frac {L}{\delta}\right)\right).
$$
Proof. Simply replacing the stationary transition  $P$  with the sequence of transitions  $\{P_t\}_{t=1}^T$  in the proof of [Jin et al., 2020, Lemma 10] suffices.
Proposition D.5.2. Let  $\mathcal{E}_{\mathrm{EST}}$  be the event such that we have for all  $k = 0,\dots ,L - 1$  simultaneously
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\max \left\{m _ {i (t)} (s , a) , 1 \right\}} = \mathcal {O} \left(| S _ {k} | | A | \log (T) + \log (\iota)\right),
$$
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\sqrt {\max  \left\{m _ {i (t)} (s , a) , 1 \right\}}} = \mathcal {O} \left(\sqrt {| S _ {k} | | A | T} + | S _ {k} | | A | \log (T) + \log (\iota)\right).
$$
We have  $\operatorname{Pr}[\mathcal{E}_{\mathrm{EST}}] \geq 1 - \delta$ .
Proof. The proof directly follows from the definition of  $\iota$ , which ensures that  $\iota \geq L / \delta$ .
Based on the event  $\mathcal{E}_{\mathrm{EST}}$ , we introduce the following lemma which is critical in analyzing the estimation error.
Lemma D.5.3. Suppose the event  $\mathcal{E}_{\mathrm{EST}}$  defined in Proposition D.5.2 holds. Then, we have for all  $k = 0,\dots ,L - 1$
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P , \pi_ {t}} (s , a)}{\widehat {m} _ {i (t)} (s , a)} = \mathcal {O} \left(| S _ {k} | | A | \log (T) + \log (\iota)\right),
$$
$$
\sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathcal {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right) = \mathcal {O} \left(\left(C ^ {\mathcal {P}} + \log (\iota)\right) | S _ {k} | | A | \log (\iota)\right),
$$
Proof. According to Corollary D.3.6, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P , \pi_ {t}} (s , a)}{\widehat {m} _ {i (t)} (s , a)} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\widehat {m} _ {i (t)} (s , a)} + \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {\left| q ^ {P , \pi_ {t}} (s , a) - q ^ {P _ {t} , \pi_ {t}} (s , a) \right|}{\widehat {m} _ {i (t)} (s , a)} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\max  \left\{1 , m _ {i (t)} (s , a) \right\}} + \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {C _ {t} ^ {\mathsf {P}}}{C ^ {\mathsf {P}} + \log (\iota)} \\ \leq \mathcal {O} \left(\left| S _ {k} \right| | A | \log (T) + \log (\iota)\right), \\ \end{array}
$$
where the second step follows from the definitions of  $C^{\mathsf{P}}$  and  $\widehat{m}_{i(t)}(s,a)$ , and the last step applies Lemma D.5.1.
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \left(\left| q ^ {P _ {t}, \pi_ {t}} (s, a) - q ^ {P, \pi_ {t}} (s, a) \right| + q ^ {P _ {t}, \pi_ {t}} (s, a)\right) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \left(C _ {t} ^ {\mathsf {P}} + q ^ {P _ {t}, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right)\right) \\ \leq | S _ {k} | \left| A \right| \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} + \left(C ^ {\mathsf {P}} + \log (\iota)\right) \sum_ {t = 1} ^ {T} \sum_ {(s, a) \in S _ {k} \times A} \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{\widehat {m} _ {i (t)} (s , a)} \\ = \mathcal {O} \left(| S _ {k} | | A | C ^ {\mathsf {P}} + \left(C ^ {\mathsf {P}} + \log (\iota)\right) (| S _ {k} | | A | \log (T) + \log (\iota))\right) \\ = \mathcal {O} \left(\left(C ^ {\mathsf {P}} + \log (\iota)\right) | S _ {k} | | A | \log (\iota)\right), \\ \end{array}
$$
where the first step adds and subtracts  $q^{P_t,\pi_t}$ ; the second step follows from Corollary D.3.6; the fourth step uses Proposition D.5.2 due to the fact that  $\widehat{m}_i(s,a) \geq \max \{m_i(s,a),1\}$ .
Lemma D.5.4. (Extension of [Dann et al., 2023, Lemma 16] for adversarial transition) Suppose the high-probability events  $\mathcal{E}_{\mathrm{EST}}$  (defined in Proposition D.5.2) and  $\mathcal{E}_{\mathrm{CON}}$  (defined in Lemma D.2.1) hold together. Let  $P_{t}^{s}$  be a transition function in  $\mathcal{P}_{i(t)}$  which depends on  $s$ , and let  $g_{t}(s)\in [0,G]$  for some  $G > 0$ . Then,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \left| q ^ {P _ {t} ^ {s}, \pi_ {t}} (s) - q ^ {P, \pi_ {t}} (s) \right| g _ {t} (s) = \mathcal {O} \left(\sqrt {L | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P , \pi_ {t}} (s) g _ {t} (s) ^ {2}}\right) \\ + \mathcal {O} \left(G \left(C ^ {\mathcal {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota)\right). \\ \end{array}
$$
Proof. By Lemma D.3.8, under the event  $\mathcal{E}_{\mathrm{CON}}$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \left| q ^ {P _ {t} ^ {s}, \pi_ {t}} (s) - q ^ {P, \pi_ {t}} (s) \right| g _ {t} (s) \\ \leq \mathcal {O} \left(\sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} g _ {t} (s) \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\hat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w)\right) \tag {73} \\ + \mathcal {O} \left(G | S | ^ {3} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right)\right). \\ \end{array}
$$
By Lemma D.5.3, the last two terms can be bounded under the event  $\mathcal{E}_{\mathrm{EST}}$  as
$$
\begin{array}{l} \mathcal {O} \left(G | S | ^ {3} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi_ {t}} (s, a) \left(\frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right)\right) \\ \leq \mathcal {O} \left(G | S | ^ {3} \sum_ {k = 0} ^ {L - 1} \left(C ^ {\mathsf {P}} + \log (\iota)\right) | S _ {k} | | A | \log (\iota)\right) \\ = \mathcal {O} \left(G \left(C ^ {\mathsf {P}} + \log (\iota)\right) L | S | ^ {4} | A | \log (\iota)\right). \\ \end{array}
$$
For the first term in Eq. (73), we first rewrite it as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {h = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w) g _ {t} (s) \\ \leq \sum_ {h = 0} ^ {L - 1} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w) g _ {t} (s). \\ \end{array}
$$
For any  $\theta > 0$  and layer  $h$ , conditioning on the event  $\mathcal{E}_{\mathrm{EST}}$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w) g _ {t} (s) \\ = \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} \sqrt {\frac {q ^ {P , \pi_ {t}} (u , v) q ^ {P , \pi_ {t}} (s | w) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \cdot q ^ {P , \pi_ {t}} (u , v) P (w | u , v) q ^ {P , \pi_ {t}} (s | w) g _ {t} (s) ^ {2}}} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} \left(\theta \cdot \frac {q ^ {P , \pi_ {t}} (u , v) q ^ {P , \pi_ {t}} (s | w) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} + \frac {1}{\theta} \cdot q ^ {P, \pi_ {t}} (u, v) P (w | u, v) q ^ {P, \pi_ {t}} (s | w) g _ {t} (s) ^ {2}\right) \\ = \theta \cdot \sum_ {t = 1} ^ {T} \sum_ {u \in S _ {h}} \sum_ {v \in A} \frac {q ^ {P , \pi_ {t}} (u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)} \cdot \left(\sum_ {w \in S _ {h + 1}} \sum_ {s \neq s _ {L}} q ^ {P, \pi_ {t}} (s | w)\right) \\ + \frac {1}{\theta} \cdot \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \left(\sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) P (w | u, v) q ^ {P, \pi_ {t}} (s | w)\right) g _ {t} (s) ^ {2} \\ \leq \theta \cdot L | S _ {h + 1} | \log (\iota) \sum_ {t = 1} ^ {T} \sum_ {u \in S _ {h}} \sum_ {v \in A} \frac {q ^ {P , \pi_ {t}} (u , v)}{\widehat {m} _ {i (t)} (u , v)} + \frac {1}{\theta} \cdot \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P, \pi_ {t}} (s) g _ {t} (s) ^ {2} \\ \leq \mathcal {O} \left(\theta \cdot L | S _ {h + 1} | \log (\iota) (| S _ {h} | | A | \log (T) + \log (\iota)) + \frac {1}{\theta} \cdot \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P, \pi_ {t}} (s) g _ {t} (s) ^ {2}\right) \\ = \mathcal {O} \left(\theta L \cdot | S _ {h} | | S _ {h + 1} | | A | \log^ {2} (\iota) + \frac {1}{\theta} \cdot \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P, \pi_ {t}} (s) g _ {t} (s) ^ {2}\right), \\ \end{array}
$$
where the second step uses the fact that  $\sqrt{xy} \leq x + y$  for all  $x, y \geq 0$ , the fourth step follows from Lemma D.5.3.
Then, for any layer  $h = 0,\dots L - 1$ , picking the optimal  $\theta$  gives
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w) g _ {t} (s) \\ = \mathcal {O} \left(\sqrt {L \left| S _ {h} \right| \left| S _ {h + 1} \right| \left| A \right| \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P , \pi_ {t}} (s) g _ {t} (s) ^ {2}}\right) \\ \leq \mathcal {O} \left(\left(| S _ {h} | + | S _ {h + 1} |\right) \sqrt {L | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P , \pi_ {t}} (s) g _ {t} (s) ^ {2}}\right). \\ \end{array}
$$
Finally, taking the summation over all the layers yields
$$
\sum_ {h = 0} ^ {L - 1} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {(u, v, w) \in W _ {h}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} q ^ {P, \pi_ {t}} (s | w) g _ {t} (s)
$$
$$
\leq \mathcal {O} \left(\sqrt {L | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P , \pi_ {t}} (s) g _ {t} (s) ^ {2}}\right),
$$