SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the second step uses  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$  for any  $x, y \in \mathbb{R}_{\geq 0}$  and the last step uses  $2\sqrt{xy} \leq x + y$  for any  $x, y \geq 0$ .
C.3 Regret Decomposition of  $\operatorname{Reg}_T(\pi^\star)$  and Proof of Lemma C.2.3
In the following sections, we will first decompose  $\mathrm{Reg}_T(\pi^\star)$  for any mapping  $\pi^\star : S \to A$  into several parts, and then bound each part separately from Section C.4 to Section C.7, in order to prove Lemma C.2.3.
For any mapping  $\pi^{\star}:S\to A$  , we start from the following decomposition of  $\mathrm{Reg}_T(\pi^\star)$  as
$$
\begin{array}{l} \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t} , \pi_ {t}} , \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}} , \widehat {\ell} _ {t} - b _ {t} \right\rangle \right]} _ {\text {E R R O R} _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}} , \widehat {\ell} _ {t} - b _ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t} , \pi^ {\star}} , \widehat {\ell} _ {t} - b _ {t} \right\rangle \right]} _ {\text {E S T R E G}} \tag {32} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi^ {\star}} , \widehat {\ell} _ {t} - b _ {t} \right\rangle - \left\langle q ^ {P _ {t} , \pi^ {\star}} , \ell_ {t} \right\rangle \right]} _ {\text {E R R O R} _ {2}}, \\ \end{array}
$$
where  $\widehat{P}_t = \widetilde{P}_{i(t)}$  denotes the optimistic transition for episode  $t$  for simplicity (which is consistent with earlier definition such that  $\widehat{q}_t = q^{\widehat{P}_t,\pi_t}$ ). Here, ESTREG is the estimated regret controlled by FTRL, while  $\mathrm{ERROR}_1$  and  $\mathrm{ERROR}_2$  are estimation errors incurred on the selected policies  $\{\pi_t\}_{t=1}^T$  and that on the comparator policy  $\pi^\star$  respectively.
In order to achieve the gap-dependent bound under Condition (9), we consider these two estimation error terms  $\mathrm{ERROR}_1$  and  $\mathrm{ERROR}_2$  together as:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle - \left\langle q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \tag {33} \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \mathbb {E} _ {t} \left[ \widehat {\ell} _ {t} \right] - b _ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \mathbb {E} _ {t} \left[ \widehat {\ell} _ {t} \right] - b _ {t} \right\rangle - \left\langle q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right], \\ \end{array}
$$
where  $\mathbb{E}_t[\cdot ]$  denotes the conditional expectation given the history prior to episode  $t$ .
To better analyze the conditional expectation of the loss estimators, we define  $\alpha_{t},\beta_{t}:S\times A\to \mathbb{R}$  as:
$$
\alpha_ {t} (s, a) \triangleq \frac {q ^ {P , \pi_ {t}} (s , a) \ell_ {t} (s , a)}{u _ {t} (s , a)}, \quad \beta_ {t} (s, a) \triangleq \frac {\left(q ^ {P _ {t} , \pi_ {t}} (s , a) - q ^ {P , \pi_ {t}} (s , a)\right) \ell_ {t} (s , a)}{u _ {t} (s , a)},
$$
which ensures that  $\mathbb{E}_t\left[\widehat{\ell}_t(s,a)\right] = \alpha_t(s,a) + \beta_t(s,a)$  for any state-action pair  $(s,a)$ .
With the help of  $\alpha_{t},\beta_{t}$  , we have
$$
\left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \mathbb {E} _ {t} \left[ \widehat {\ell} _ {t} \right] - b _ {t} \right\rangle = \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \beta_ {t} - b _ {t} \right\rangle ,
$$
$$
\left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \mathbb {E} _ {t} \left[ \widehat {\ell} _ {t} \right] - b _ {t} \right\rangle = \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \beta_ {t} - b _ {t} \right\rangle ,
$$
which helps us further rewrite Eq. (33) as
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi^ {\star}} - q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle - \left\langle q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \tag {34} \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, b _ {t} - \beta_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \beta_ {t} - b _ {t} \right\rangle \right]. \\ \end{array}
$$
Based on this decomposition of  $\mathrm{ERROR}_1 + \mathrm{ERROR}_2$ , we then bound each parts respectively in the following lemmas.
Lemma C.3.1. For any  $\delta \in (0,1)$  and any policy sequence  $\{\pi_t\}_{t=1}^T$ , Algorithm 4 ensures that
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi^ {\star}} - q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] = \mathcal {O} \left(C ^ {P} L\right).
$$
Lemma C.3.2. For any  $\delta \in (0,1)$  and any mapping  $\pi^{\star}:S\to A$  Algorithm 4 ensures that
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, b _ {t} - \beta_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \beta_ {t} - b _ {t} \right\rangle \right] = \mathcal {O} \left(C ^ {P} L | S | ^ {2} | A | \log^ {2} (T)\right).
$$
Lemma C.3.3. For any  $\delta \in (0,1)$  and any mapping  $\pi^{\star}:S\to A$  Algorithm 4 ensures that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle - \left\langle q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ = \mathcal {O} \left(\mathbb {S} _ {1} \left(L ^ {3} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \left(C ^ {\mathcal {P}} + 1\right) L ^ {2} | S | ^ {4} | A | \log^ {2} (\iota) + \delta L | S | ^ {2} | A | T ^ {2}\right). \\ \end{array}
$$
Lemma C.3.4. With the learning rates  $\{\gamma_t\}_{t=1}^T$  defined in Definition 5.2, Algorithm 4 ensures that for any  $\delta \in (0,1)$  and any mapping  $\pi^\star$  that,
$$
\begin{array}{l} \operatorname {E S T R E G} \left(\pi^ {\star}\right) \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right] \\ = \mathcal {O} \left(\mathbb {S} _ {2} \left(L ^ {2} | S | | A | \log^ {2} (\iota)\right) + \left(C ^ {\mathcal {P}} + 1\right) L | S | ^ {2} | A | ^ {2} \log^ {2} (\iota) + \delta T L ^ {2} | S | ^ {2} | A | \log (\iota)\right). \\ \end{array}
$$
Proof of Lemma C.2.3. According to the previous discussion, we have the regret against any mapping  $\pi^{\star}: S \to A$  decomposed as:
$$
\begin{array}{l} \operatorname {R e g} _ {T} \left(\pi^ {\star}\right) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi^ {\star}} - q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, b _ {t} - \beta_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi^ {*}}, \beta_ {t} - b _ {t} \right\rangle \right] \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle - \left\langle q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right], \\ \end{array}
$$
where the first term is mainly caused by the difference between  $\{P_t\}_{t=1}^T$  and  $P$ , which is unavoidable and can be bounded by  $\mathcal{O}(C^p L)$  as shown in Lemma C.3.1; the second term is the extra cost of using the amortized losses  $b_t$  to handle the biases of loss estimators, which is controlled by  $\widetilde{\mathcal{O}}(C^p)$  as shown in Lemma C.3.2; the third term measures the estimation error related to the optimistic transitions  $\{\widehat{P}_t\}_{t=1}^T$ , which can be bounded by some self-bounding quantities in Lemma C.3.2; the final term is the estimated regret calculated with respect to the optimistic transitions  $\{\widehat{P}_t\}_{t=1}^T$ , which is controlled by FTRL as shown in Lemma C.3.4. Putting all these bounds together finishes the proof.
C.4 Proof of Lemma C.3.1
The result is immediate by directly applying Corollary D.3.7.
C.5 Proof of Lemma C.3.2
For this proof, we bound  $\mathbb{E}\left[\sum_{t=1}^{T}\left\langle q^{\widehat{P}_t,\pi_t}, b_t - \beta_t \right\rangle\right]$  and  $\mathbb{E}\left[\sum_{t=1}^{T}\left\langle q^{\widehat{P}_t,\pi^*}, \beta_t - b_t \right\rangle\right]$  separately.
Bounding  $\mathbb{E}\left[\sum_{t = 1}^{T}\left\langle q^{\widehat{P}_t,\pi_t},b_t - \beta_t\right\rangle \right]$  We have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, b _ {t} - \beta_ {t} \right\rangle \\ = \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \left(b _ {t} (s) - \frac {\left(q ^ {P _ {t} , \pi_ {t}} (s , a) - q ^ {P , \pi_ {t}} (s , a)\right) \ell_ {t} (s , a)}{u _ {t} (s , a)}\right) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \left(b _ {t} (s) + \frac {\left| q ^ {P _ {t} , \pi_ {t}} (s , a) - q ^ {P , \pi_ {t}} (s , a) \right|}{u _ {t} (s , a)}\right) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) b _ {t} (s) + \sum_ {t = 1} ^ {T} \sum_ {s, a} \left| q ^ {P _ {t}, \pi_ {t}} (s, a) - q ^ {P, \pi_ {t}} (s, a) \right| \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) b _ {t} (s) + L C ^ {\mathsf {p}}, \\ \end{array}
$$
where the second step bounds  $\ell_t(s,a)\leq 1$ ; the third step follows the fact that  $q^{\widehat{P}_t,\pi_t}(s,a)\leq u_t(s,a)$  for all  $(s,a)$ . Let  $E_{i}$  be a set of episodes that belong to epoch  $i$  and let  $N$  be the total number of epochs through  $T$  episodes. Then, we turn to bound
$$
\sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) b _ {t} (s) = \sum_ {i = 1} ^ {N} \sum_ {t \in E _ {i}} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) b _ {t} (s) \leq \mathcal {O} \left(L | S | ^ {2} | A | \log^ {2} (T) C ^ {\mathsf {P}}\right),
$$
where the last step repeats the same argument of Lemma 3.1 for every epoch and the number of epochs is at most  $O(|S||A|\log T)$  according to [Jin et al., 2021, Lemma D.3.12].
Bounding  $\mathbb{E}\left[\sum_{t=1}^{T}\left\langle q^{\widehat{P}_t,\pi^\star}, \beta_t - b_t \right\rangle\right]$ . For this term, we show that for any given epoch  $i$ ,  $\sum_{t \in E_i}\left\langle q^{\widehat{P}_t,\pi^\star}, \beta_t - b_t \right\rangle \leq 0$  which yields  $\sum_{t=1}^{T}\left\langle q^{\widehat{P}_t,\pi^\star}, \beta_t - b_t \right\rangle \leq 0$ . To this end, we first consider a fixed epoch  $i$  and upper-bound
$$
\left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \beta_ {t} \right\rangle = \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi^ {\star}} (s, a) \left(\frac {\left(q ^ {P _ {t} , \pi_ {t}} (s) - q ^ {P , \pi_ {t}} (s)\right) \ell_ {t} (s , a)}{u _ {t} (s)}\right) \leq \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi^ {\star}} (s, a) \left(\frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)}\right),
$$
where the second step uses Corollary D.3.6 to bound  $\left(q^{P_t,\pi_t}(s) - q^{P,\pi_t}(s)\right) \leq C_t^{\mathsf{P}}$ .
For any epoch  $i$  and episode  $t \in E_i$ , we have  $\widehat{P}_t = \widetilde{P}_{i(t)}$ , which gives that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \beta_ {t} - b _ {t} \right\rangle = \sum_ {i = 1} ^ {N} \sum_ {t \in E _ {i}} \left\langle q ^ {\widetilde {P} _ {i}, \pi^ {\star}}, \beta_ {t} - b _ {t} \right\rangle \\ \leq \sum_ {i = 1} ^ {N} \sum_ {s, a} q ^ {\widetilde {P} _ {i}, \pi^ {\star}} (s, a) \sum_ {t \in E _ {i}} \left(\frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)} - b _ {t} (s)\right) \\ \leq 0, \\ \end{array}
$$
where the last step applies Lemma 3.1 for every epoch  $i$ . Thus,  $\mathbb{E}\left[\sum_{t=1}^{T} \left\langle q^{\widehat{P}_t, \pi^\star}, \beta_t - b_t \right\rangle\right] \leq 0$  holds.
C.6 Proof of Lemma C.3.3
We introduce the following lemma to evaluate the estimated performance via the true occupancy measure, which helps us analyze the estimation error.
Lemma C.6.1. For any transition function pair  $(P, \widehat{P})$  ( $P$  and  $\widehat{P}$  can be optimistic transition), policy  $\pi$ , and loss function  $\ell$ , it holds that  $\langle q^{P,\pi}, \ell \rangle = \langle q^{\widehat{P},\pi}, \mathcal{Z}_{\ell}^{P, \widehat{P}, \pi} \rangle$ , where  $\mathcal{Z}_{\ell}^{P, \widehat{P}, \pi}$  is defined as
$$
\begin{array}{l} \mathcal {Z} _ {\ell} ^ {P, \widehat {P}, \pi} (s, a) = Q ^ {P, \pi} (s, a; \ell) - \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} \widehat {P} (s ^ {\prime} | s, a) V ^ {P, \pi} (s ^ {\prime}; \ell) \\ = \ell (s, a) + \sum_ {s ^ {\prime} \in S _ {k (s)} + 1} \left(P (s ^ {\prime} | s, a) - \widehat {P} (s ^ {\prime} | s, a)\right) V ^ {P, \pi} (s ^ {\prime}; \ell) \\ \end{array}
$$
for all state-action pairs  $(s,a)$ .
Proof. By direct calculation, we have:
$$
\begin{array}{l} V ^ {P, \pi} (s; \ell) = \sum_ {a} \pi (a | s) Q ^ {P, \pi} (s, a; \ell) \\ = \sum_ {a} \pi (a | s) \left(Q ^ {P, \pi} (s, a; \ell) - \sum_ {s ^ {\prime} \in S _ {k (s)} + 1} \widehat {P} (s ^ {\prime} | s, a) V ^ {P, \pi} (s; \ell)\right) \\ + \sum_ {a} \pi (a | s) \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} \widehat {P} \left(s ^ {\prime} \mid s, a\right) V ^ {P, \pi} \left(s ^ {\prime}; \ell\right) \\ = \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} q ^ {\widehat {P}, \pi} \left(s ^ {\prime} | s\right) V ^ {P, \pi} \left(s ^ {\prime}; \ell\right) \\ + \sum_ {a} \pi (a | s) \left(Q ^ {P, \pi} (s, a; \ell) - \sum_ {s ^ {\prime} \in S _ {k (s)} + 1} \widehat {P} \left(s ^ {\prime} \mid s, a\right) V ^ {P, \pi} (s; \ell)\right) \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {k = k (s)} ^ {L - 1} \sum_ {s ^ {\prime} \in S _ {k}} \sum_ {a \in A} q ^ {\hat {P}, \pi} \left(s ^ {\prime}, a | s\right) \left(Q ^ {P, \pi} \left(s ^ {\prime}, a; \ell\right) - \sum_ {s ^ {\prime \prime} \in S _ {k \left(s ^ {\prime}\right) + 1}} \hat {P} \left(s ^ {\prime \prime} \mid s ^ {\prime}, a\right) V ^ {P, \pi} \left(s ^ {\prime \prime}; \ell\right)\right) \\ = \sum_ {k = k (s)} ^ {L - 1} \sum_ {s ^ {\prime} \in S _ {k}} \sum_ {a \in A} q ^ {\widehat {P}, \pi} \left(s ^ {\prime}, a | s\right) \mathcal {Z} _ {\ell} ^ {P, \widehat {P}, \pi} \left(s ^ {\prime}, a\right) \\ \end{array}
$$
where the second to last step follows from recursively repeating the first three steps. The proof is completed by noticing  $\langle q^{P,\pi},\ell \rangle = V^{P,\pi}(s_0,\ell)$ .
According to Lemma C.6.1, we rewrite  $\left\langle q^{\widehat{P}_t,\pi_t},\alpha_t\right\rangle$  and  $\left\langle q^{\widehat{P}_t,\pi^\star},\alpha_t\right\rangle$  with  $q^{P,\pi_t}$  and  $q^{P,\pi^{\star}}$  for any policy  $\pi^{\star}$  as
$$
\left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle = \left\langle q ^ {P, \pi_ {t}}, \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi_ {t}} \right\rangle , \quad \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle = \left\langle q ^ {P, \pi^ {\star}}, \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi^ {\star}} \right\rangle .
$$
Therefore, we can further decompose as
$$
\begin{array}{l} \mathbb {E} \left[ \left\langle q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle - \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \alpha_ {t} \right\rangle + \left\langle q ^ {\widehat {P} _ {t}, \pi^ {\star}}, \alpha_ {t} \right\rangle - \left\langle q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}}, \ell_ {t} - \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi_ {t}} \right\rangle - \left\langle q ^ {P, \pi^ {\star}}, \ell_ {t} - \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi^ {\star}} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}} - q ^ {P, \pi^ {\star}}, \ell_ {t} - \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi^ {\star}} \right\rangle + \left\langle q ^ {P, \pi_ {t}}, \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi^ {\star}} - \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t}, P, \pi_ {t}} \right\rangle \right] \\ = \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(q ^ {P , \pi_ {t}} (s , a) - q ^ {P , \pi^ {\star}} (s , a)\right) \left(\ell_ {t} (s , a) - \alpha_ {t} (s , a)\right)} _ {\text {T E R M} 1 (\pi^ {\star})} \right] \\ + \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(q ^ {P , \pi_ {t}} (s , a) - q ^ {P , \pi^ {\star}} (s , a)\right) \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} \left(P (s ^ {\prime} | s , a) - \widehat {P} _ {t} (s ^ {\prime} | s , a)\right) V ^ {\widehat {P} _ {t} , \pi^ {\star}} (s ^ {\prime} ; \alpha_ {t})} _ {\text {T E R M} 2 (\pi^ {\star})} \right] \\ + \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \left\langle q ^ {P , \pi_ {t}} , \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t} , P , \pi^ {\star}} - \mathcal {Z} _ {\alpha_ {t}} ^ {\widehat {P} _ {t} , P , \pi_ {t}} \right\rangle} _ {\text {T E R M} 3 (\pi^ {\star})} \right]. \\ \end{array}
$$
We will bound these terms with some self-bounding quantities in Lemma C.6.2, Lemma C.6.3 and Lemma C.6.4 respectively. In these proofs, we will follow the idea of Lemma D.1.1 to first bound these terms conditioning on the events  $\mathcal{E}_{\mathrm{EST}}$  and  $\mathcal{E}_{\mathrm{CON}}$  defined in Proposition D.5.2 and Eq. (4), while ensuring that these terms are always bounded by  $\mathcal{O}\left(|S|^2 |A|T^2\right)$  in the worst case.
C.6.1 Bounding Term 1
Lemma C.6.2. For any  $\delta \in (0,1)$  and any mapping  $\pi^{\star}:S\to A$  Algorithm 4 ensures that  $\mathbb{E}$  [TERM  $1(\pi^{\star})]$  is bounded by
$$
\mathcal {O} \left(\mathbb {S} _ {1} \left(L ^ {2} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \delta | S | ^ {2} | A | T ^ {2} + \left(C ^ {\mathcal {P}} + 1\right) L ^ {2} | S | ^ {4} | A | \log^ {2} (\iota)\right).
$$
Proof. Clearly, we have  $\alpha_{t}(s,a)\leq |S|T$  for every state-action pair  $(s,a)$ , due to the fact that  $u_{t}(s)\geq 1 / |s|T$  (Lemma D.2.8). Therefore, we have  $\mathrm{TERM}1(\pi^{\star})\leq |S|^{2}|A|T^{2}$  holds always. In the remaining, our main goal is to bound  $\mathrm{TERM}1(\pi^{\star})$  conditioning on  $\mathcal{E}_{\mathrm{CON}}\wedge \mathcal{E}_{\mathrm{EST}}$ .
For any state-action pair  $(s, a) \in S \times A$  and episode  $t$ , we have
$$
\ell_ {t} (s, a) - \alpha_ {t} (s, a) = \frac {\left(u _ {t} (s , a) - q ^ {P , \pi_ {t}} (s , a)\right) \ell_ {t} (s , a)}{u _ {t} (s , a)} = \frac {\left(u _ {t} (s) - q ^ {P , \pi_ {t}} (s)\right) \ell_ {t} (s , a)}{u _ {t} (s)} \geq 0,
$$
conditioning on the event  $\mathcal{E}_{\mathrm{CON}}$
Therefore, under event  $\mathcal{E}_{\mathrm{CON}}\wedge \mathcal{E}_{\mathrm{EST}}$  , TERM  $1(\pi^{\star})$  is bounded by
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(q ^ {P, \pi_ {t}} (s, a) - q ^ {P, \pi^ {\star}} (s, a)\right) (\ell_ {t} (s, a) - \alpha_ {t} (s, a)) \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left[ q ^ {P, \pi_ {t}} (s, a) - q ^ {P, \pi^ {\star}} (s, a) \right] _ {+} \frac {\left(u _ {t} (s) - q ^ {P , \pi_ {t}} (s)\right) \ell_ {t} (s , a)}{u _ {t} (s)} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \frac {\left[ q ^ {P , \pi_ {t}} (s , a) - q ^ {P , \pi^ {*}} (s , a) \right] _ {+}}{u _ {t} (s)} \cdot \left| u _ {t} (s) - q ^ {P, \pi_ {t}} (s) \right| \\ = \mathcal {O} \left(\sqrt {L | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} q ^ {P , \pi_ {t}} (s) \cdot \left(\sum_ {a} \frac {\left[ q ^ {P , \pi_ {t}} (s , a) - q ^ {P , \pi^ {\star}} (s , a) \right] _ {+}}{u _ {t} (s)}\right) ^ {2}}\right) \\ + \mathcal {O} \left(\left(C ^ {\mathsf {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota)\right) \\ \leq \mathcal {O} \left(\sqrt {L | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \in A} [ q ^ {P , \pi_ {t}} (s , a) - q ^ {P , \pi^ {\star}} (s , a) ] _ {+}}\right) \\ + \mathcal {O} \left(\left(C ^ {\mathsf {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota)\right) \\ \leq \mathcal {O} \left(\sqrt {L ^ {2} | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P , \pi_ {t}} (s , a)} + \left(C ^ {\mathsf {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota)\right) \\ \leq \mathcal {O} \left(\sqrt {L ^ {2} | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t , \pi_ {t}}} (s , a)} + \left(C ^ {\mathsf {P}} + 1\right) L ^ {2} | S | ^ {4} | A | \log^ {2} (\iota)\right), \\ \end{array}
$$
where the first step follows from the non-negativity of  $\ell_t(s,a) - \alpha_t(s,a)$ ; the third step applies Lemma D.5.4 with  $G = 1$  as  $\sum_{a}\left[q^{P,\pi_t}(s,a) - q^{P,\pi^\star}(s,a)\right]_+\leq q^{P,\pi_t}(s)\leq u_t(s)$ ; the fifth step follows from the fact that  $\sum_{s\neq s_L}\sum_{a\in A}\left[q^{P,\pi_t}(s,a) - q^{P,\pi^\star}(s,a)\right]_+\leq 2L\sum_{s\neq s_L}\sum_{a\neq \pi^\star (s)}q^{P,\pi_t}(s,a)$  according to Corollary D.3.5; the last step applies Corollary D.3.6.
Applying Lemma D.1.1 with event  $\mathcal{E}_{\mathrm{CON}}\wedge \mathcal{E}_{\mathrm{EST}}$  yields that
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {T e r m} 1 \left(\pi^ {\star}\right) \right] = \mathcal {O} \left(\mathbb {E} \left[ \sqrt {L ^ {2} | S | ^ {2} | A | \log^ {2} (\iota) \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t} , \pi_ {t}} (s , a)} \right]\right) \\ + \mathcal {O} \left(\left(C ^ {\mathsf {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota) + \delta | S | ^ {2} | A | T ^ {2}\right) \\ = \mathcal {O} \left(\mathbb {S} _ {1} \left(L ^ {2} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \delta | S | ^ {2} | A | T ^ {2} + \left(C ^ {\mathsf {P}} + \log (\iota)\right) L ^ {2} | S | ^ {4} | A | \log (\iota)\right). \\ \end{array}
$$
C.6.2 Bounding Term 2
Lemma C.6.3. For any  $\delta \in (0,1)$  and any mapping  $\pi^{\star}:S\to A$  Algorithm 4 ensures that
$$
\mathbb {E} \left[ \operatorname {T e r m} 2 (\pi^ {\star}) \right] = \mathcal {O} \left(\mathbb {S} _ {1} \left(L | S | ^ {2} | A | \log^ {2} (\iota)\right) + \delta L | S | ^ {2} | A | T + L | S | ^ {2} | A | \left(C ^ {P} + \log (\iota)\right) \log (T)\right).
$$
Proof. Suppose that  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  occurs. We have