SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {s \in S} \left| u _ {t} (s) - q ^ {P _ {t}, \pi_ {t}} (s) \right| \\ \leq \mathcal {O} \left(\sum_ {t = 1} ^ {T} L C _ {t} ^ {\mathsf {P}} + L \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\hat {m} _ {i (t)} (u , v)}}\right) \\ + \mathcal {O} \left(| S | ^ {3} \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {P, \pi_ {t}} (s, a) \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right) \\ \leq \mathcal {O} \left(L C ^ {\mathsf {P}} + | S | ^ {3} \sum_ {k = 0} ^ {L - 1} \left(C ^ {\mathsf {P}} + \log (\iota)\right) | S _ {k} | | A | \log (\iota)\right) \\ + \mathcal {O} \left(L \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v) \in S _ {k} \times A} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {| S _ {k + 1} | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}}\right) \\ \leq \mathcal {O} \left(| S | ^ {4} | A | \left(C ^ {\mathsf {P}} + \log (\iota)\right) \log (\iota) + L \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v) \in S _ {k} \times A} q ^ {P _ {t}, \pi_ {t}} (u, v) \sqrt {\frac {| S _ {k + 1} | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}}\right) \\ + \mathcal {O} \left(L \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v) \in S _ {k} \times A} \left| q ^ {P, \pi_ {t}} (u, v) - q ^ {P _ {t}, \pi_ {t}} (u, v) \right| \sqrt {\frac {| S | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathcal {O} \left(| S | ^ {4} | A | \left(C ^ {\mathsf {P}} + \log (\iota)\right) \log (\iota) + L \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v) \in S _ {k} \times A} q ^ {P _ {t}, \pi_ {t}} (u, v) \sqrt {\frac {| S _ {k + 1} | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}}\right) \\ + \mathcal {O} \left(L \sqrt {| S |} \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v) \in S _ {k} \times A} \left| q ^ {P, \pi_ {t}} (u, v) - q ^ {P _ {t}, \pi_ {t}} (u, v) \right|\right) \\ \leq \mathcal {O} \left(| S | ^ {4} | A | \left(C ^ {\mathbb {P}} + \log (\iota)\right) \log (\iota) + L \sum_ {k = 0} ^ {L - 1} \sqrt {| S _ {k + 1} | \log (\iota)} \left(\sqrt {| S _ {k} | | A | T} + | S _ {k} | | A | \log (\iota)\right)\right) \\ = \mathcal {O} \left(| S | ^ {4} | A | \left(C ^ {\mathsf {P}} + \log (\iota)\right) \log (\iota) + L | S | \sqrt {| A | T \log (\iota)}\right), \\ \end{array}
$$
where the second step uses the Cauchy-Schwartz inequality:  $\sum_{w\in S_{k + 1}}\sqrt{P(w|u,v)}\leq \sqrt{|S_{k + 1}|}$  and also applies Lemma D.5.3; the fourth step bounds  $\hat{m}_{i(t)}(u,v)\geq \log (\iota)$  for any  $(u,v)$  and  $|S_{k + 1}|\leq |S|$ ; the fifth applies Proposition D.5.2, and Corollary D.3.7 to bound  $\sum_{t = 1}^{T}\sum_{k = 0}^{L - 1}\sum_{(u,v)\in S_k\times A}\left|q^{P,\pi_t}(u,v) - q^{P_t,\pi_t}(u,v)\right|$  by  $\mathcal{O}\left(LC^{\mathsf{P}}\right)$ ; the last step uses the fact that  $\sqrt{xy}\leq x + y$  for any  $x,y\geq 0$ , and thus we have  $\sum_{k = 0}^{L - 1}\sqrt{|S_k||S_{k + 1}|}\leq 2\sum_{k = 0}^{L - 1}|S_k| = 2|S|$ .
Finally, using a similar argument in the proof of Lemma A.2.1 to bound the case that  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  does not hold, we complete the proof.
A.4 Bounding REG
Lemma A.4.1 (Bound of REG). If learning rate  $\eta$  satisfies  $\eta \in (0, \frac{1}{8L}]$ , then, Algorithm 1 ensures
$$
\operatorname {R E G} = \mathcal {O} \left(\frac {| S | ^ {2} | A | \log (\iota)}{\eta} + \eta \left(L | S | C ^ {\mathcal {P}} \log T + L T\right) + \delta L T\right).
$$
Proof. We consider a specific transition  $P_0$  defined in Lee et al. [2020, Lemma C.4] and occupancy measure  $u$  such that
$$
u = \left(1 - \frac {1}{T}\right) q ^ {P, \stackrel {\circ} {\pi}} + \frac {1}{T | A |} \sum_ {a \in A} q ^ {P _ {0}, \pi_ {a}},
$$
where  $\pi_{a}$  is a policy such that action  $a$  is selected at every state.
$$
\begin{array}{l} D _ {\phi} (u, \widehat {q} _ {1}) = \sum_ {k = 0} ^ {L - 1} \sum_ {(s, a, s ^ {\prime}) \in W _ {k}} \left(\log \left(\frac {\widehat {q} _ {1} (s , a , s ^ {\prime})}{u (s , a , s ^ {\prime})}\right) + \frac {u (s , a , s ^ {\prime})}{\widehat {q} _ {1} (s , a , s ^ {\prime})} - 1\right) \\ = \sum_ {k = 0} ^ {L - 1} \sum_ {(s, a, s ^ {\prime}) \in W _ {k}} \log \left(\frac {\widehat {q} _ {1} (s , a , s ^ {\prime})}{u (s , a , s ^ {\prime})}\right) + \sum_ {k = 0} ^ {L - 1} \sum_ {(s, a, s ^ {\prime}) \in W _ {k}} \left(| S _ {k} | | A | | S _ {k + 1} | u (s, a, s ^ {\prime}) - 1\right) \\ = \sum_ {k = 0} ^ {L - 1} \sum_ {(s, a, s ^ {\prime}) \in W _ {k}} \log \left(\frac {\widehat {q} _ {1} (s , a , s ^ {\prime})}{u (s , a , s ^ {\prime})}\right) \\ \leq 3 | S | ^ {2} | A | \log (t), \\ \end{array}
$$
where the second step uses the definition  $\widehat{q}_1(s,a,s') = \frac{1}{|S_k||A||S_{k+1}|}$  for  $k = 0,\dots,L-1$  and the fourth step lower-bounds  $u(s,a,s') \geq \frac{1}{T^3|S|^2|A|}$  from [Lee et al., 2020, Lemma C.10], thereby  $u(s,a,s') \geq \frac{1}{\iota^3}$ , and upper-bounds  $\widehat{q}_1(s,a,s') \leq 1$ .
According to [Lee et al., 2020, Lemma C.4], we have  $q^{P_0,\pi_a} \in \cap_i \Omega(\mathcal{P}_i)$ . Therefore,  $u$  is a convex combination of points in that convex set, and we can use Lemma A.4.2 (included after this proof) to show
$$
\left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}} - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle
$$
$$
\leq \frac {3 | S | ^ {2} | A | \log (\iota)}{\eta} + 2 \eta \left(\sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \widehat {\ell} _ {t} (s, a) ^ {2} + \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} b _ {t} (s) ^ {2}\right). \tag {17}
$$
On the one hand, we bound the first summation in Eq. (17) as
$$
\sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \widehat {\ell} _ {t} (s, a) ^ {2} = \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \cdot \frac {\ell_ {t} (s , a) ^ {2} \mathbb {I} _ {t} \{s , a \}}{u _ {t} (s , a) ^ {2}} \leq L T,
$$
since  $q^{\widehat{P}_t,\pi_t}(s,a)\leq u_t(s,a)$  by definition of the upper occupancy bound.
On the other hand, we bound the second summation in Eq. (17) as
$$
\sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} b _ {t} (s) ^ {2} \leq 4 L \sum_ {t = 1} ^ {T} \sum_ {s} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s) b _ {t} (s) = \mathcal {O} \left(L | S | C ^ {\mathbb {P}} \log T\right), \tag {18}
$$
where the first step uses the facts that  $b_{t}(s)^{2} \leq 4Lb_{t}(s)$  and  $\sum_{a} q^{\widehat{P}_{t},\pi_{t}}(s,a)^{2} \leq q^{\widehat{P}_{t},\pi_{t}}(s)$ , and the last step applies Lemma 3.1.
Putting these inequalities together concludes the proof.
Lemma A.4.2. With  $\eta \in (0, \frac{1}{8L}]$ , Algorithm 1 ensures
$$
\sum_ {t = 1} ^ {T} \left\langle \widehat {q} _ {t} - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle \leq \frac {D _ {\phi} (u , \widehat {q} _ {1})}{\eta} + 2 \eta \sum_ {t = 1} ^ {T} \sum_ {s, a} \widehat {q} _ {t} (s, a) ^ {2} \left(\widehat {\ell} _ {t} (s, a) ^ {2} + b _ {t} (s) ^ {2}\right), \tag {19}
$$
for any  $u\in \cap_{i}\Omega (\mathcal{P}_{i})$
Proof. To use the standard analysis of OMD with log-barrier (e.g., see [Agarwal et al., 2017, Lemma 12]), we need to ensure that  $\eta \widehat{q}_t(s,a,s')\left(\widehat{\ell}_t(s,a) - b_t(s)\right) \geq -\frac{1}{2}$ , since  $\log (1 + x) \geq x - x^2$  holds for any  $x \geq -\frac{1}{2}$ . Clearly, by choosing  $\eta \in (0,\frac{1}{8L}]$ , we have
$$
\begin{array}{l} \eta \widehat {q} _ {t} (s, a, s ^ {\prime}) \left(\widehat {\ell} _ {t} (s, a) - b _ {t} (s)\right) \\ \geq - \eta \widehat {q} _ {t} (s, a, s ^ {\prime}) b _ {t} (s) \\ \geq - 4 L \eta \frac {\widehat {q} _ {t} (s , a , s ^ {\prime})}{u _ {t} (s)} \\ \geq - 4 L \eta \\ \geq - \frac {1}{2}, \\ \end{array}
$$
where the first step follows from the fact that  $\widehat{\ell_t}(s, a) \geq 0$  for all  $t, s, a$ ; the second step follows from the definition of amortized bonus in Eq. (7); the third step bounds  $\widehat{q_t}(s, a, s') \leq u_t(s)$ .
Now, we are ready to apply the standard analysis to show that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle \widehat {q} _ {t} - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle \\ \leq \frac {\sum_ {t = 1} ^ {T} \left(D _ {\phi} (u , \widehat {q} _ {t}) - D _ {\phi} (u , \widehat {q} _ {t + 1})\right)}{\eta} + \eta \sum_ {t = 1} ^ {T} \sum_ {s, a, s ^ {\prime}} \widehat {q} _ {t} (s, a, s ^ {\prime}) ^ {2} \left(\widehat {\ell} _ {t} (s, a) - b _ {t} (s)\right) ^ {2} \\ = \frac {D _ {\phi} (u , \widehat {q} _ {1}) - D _ {\phi} (u , \widehat {q} _ {T + 1})}{\eta} + \eta \sum_ {t = 1} ^ {T} \sum_ {s, a, s ^ {\prime}} \left(\widehat {q} _ {t} (s, a) \widehat {P} _ {t} (s ^ {\prime} | s, a)\right) ^ {2} \left(\widehat {\ell} _ {t} (s, a) - b _ {t} (s)\right) ^ {2} \\ \leq \frac {D _ {\phi} (u , \widehat {q} _ {1})}{\eta} + \eta \sum_ {t = 1} ^ {T} \sum_ {s, a} \widehat {q} _ {t} (s, a) ^ {2} \left(\widehat {\ell} _ {t} (s, a) - b _ {t} (s)\right) ^ {2} \cdot \left(\sum_ {s ^ {\prime}} \widehat {P} _ {t} (s ^ {\prime} | s, a) ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {D _ {\phi} (u , \widehat {q} _ {1})}{\eta} + \eta \sum_ {t = 1} ^ {T} \sum_ {s, a} \widehat {q} _ {t} (s, a) ^ {2} \left(\widehat {\ell} _ {t} (s, a) - b _ {t} (s)\right) ^ {2} \\ \leq \frac {D _ {\phi} (u , \widehat {q} _ {1})}{\eta} + 2 \eta \sum_ {t = 1} ^ {T} \sum_ {s, a} \widehat {q} _ {t} (s, a) ^ {2} \left(\widehat {\ell} _ {t} (s, a) ^ {2} + b _ {t} (s) ^ {2}\right), \\ \end{array}
$$
where the second step uses the fact that  $\widehat{q}_t(s,a,s') = \widehat{q}_t(s,a)\cdot \widehat{P}_t(s'|s,a)$  (see [Jin et al., 2020, Lemma 1] for more details); the third step follows from the fact that the Bregman divergence is non-negative; the fourth step follows from the fact  $\sum_{s'}\widehat{P}_t(s'|s,a) = 1$ ; the last step uses the fact that  $(x - y)^2\leq 2\left(x^2 +y^2\right)$  for any  $x,y\in \mathbb{R}$ .
A.5 Bounding  $\mathbf{BIAS}_2$
Lemma A.5.1 (Bound of  $\mathrm{BIAS}_2$ ). Algorithm 1 ensures
$$
\mathrm {B I A S} _ {2} = \mathcal {O} \left(| S | C ^ {\mathcal {P}} \log T + \delta L T\right).
$$
Proof. We first rewrite  $\mathrm{BIAS}_2$  as
$$
\mathrm {B I A S} _ {2} = \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle q ^ {P , \hat {\pi}}, \widehat {\ell} _ {t} - \ell_ {t} \right\rangle \right] - \sum_ {t = 1} ^ {T} \left\langle q ^ {P , \hat {\pi}} , b _ {t} \right\rangle \right]} _ {=: (\mathrm {I})} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}} , b _ {t} \right\rangle \right]} _ {=: (\mathrm {I I})},
$$
where (II) is bounded by  $\mathcal{O}\left(|S|C^{\mathsf{P}}\log T\right)$  by Lemma 3.1 (whose proof is included after this proof). Now, we show that (I) is bounded by  $\mathcal{O}(\delta LT)$ . Suppose that  $\mathcal{E}_{\mathrm{CON}}$  holds. We then have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left\langle q ^ {P, \hat {\pi}}, \widehat {\ell} _ {t} - \ell_ {t} \right\rangle \right] \\ = \sum_ {s, a} q ^ {P, \mathring {\pi}} (s, a) \left(\frac {q ^ {P _ {t} , \pi_ {t}} (s , a) - u _ {t} (s , a)}{u _ {t} (s , a)}\right) \\ \leq \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \left(\frac {\left| q ^ {P _ {t} , \pi_ {t}} (s , a) - q ^ {P , \pi_ {t}} (s , a) \right| + q ^ {P , \pi_ {t}} (s , a) - u _ {t} (s , a)}{u _ {t} (s , a)}\right) \\ = \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \cdot \frac {\left| q ^ {P _ {t} , \pi_ {t}} (s) - q ^ {P , \pi_ {t}} (s) \right|}{u _ {t} (s)} + \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \left(\frac {q ^ {P , \pi_ {t}} (s , a) - u _ {t} (s , a)}{u _ {t} (s , a)}\right) \\ \leq \sum_ {s, a} q ^ {P, \mathring {\pi}} (s, a) \cdot \frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)} + \sum_ {s, a} q ^ {P, \mathring {\pi}} (s, a) \left(\frac {q ^ {P , \pi_ {t}} (s , a) - u _ {t} (s , a)}{u _ {t} (s , a)}\right) \\ \leq \sum_ {s, a} q ^ {P, \mathring {\pi}} (s, a) \cdot \frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)}, \\ \end{array}
$$
where the fourth step applies Corollary D.3.6 to bound  $\left|q^{P_t,\pi_t}(s) - q^{P,\pi_t}(s)\right| \leq C_t^{\mathsf{P}}$ , and in the last step, the second term (in the fifth step) is bounded by 0 under  $\mathcal{E}_{\mathrm{CON}}$ .
Since  $q^{P,\hat{\pi}}(s,a)$  is fixed over all episodes, we apply Lemma 3.1 to show
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle q ^ {P, \hat {\pi}}, \widehat {\ell_ {t}} - \ell_ {t} \right\rangle \right] - \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \hat {\pi}}, b _ {t} \right\rangle \leq \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \sum_ {t = 1} ^ {T} \left(\frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)} - b _ {t} (s)\right) \leq 0,
$$
For the case that  $\mathcal{E}_{\mathrm{CON}}$  does not occur, we bound the expected regret by  $\mathcal{O}(\delta LT)$ . Combining two cases via Lemma D.1.1, we conclude the proof.
Lemma A.5.2 (Restatement of Lemma 3.1). The amortized bonus defined in Eq. (7) satisfies  $\sum_{t=1}^{T} \frac{C_t^P}{u_t(s)} \leq \sum_{t=1}^{T} b_t(s)$  and  $\sum_{t=1}^{T} \widehat{q}_t(s) b_t(s) = \mathcal{O}(C^P \log T)$  for any  $s$ .
Proof. In the following proof, we use the equivalent definition of  $b_{t}(s)$  given in Eq. (14). We first show  $\sum_{t=1}^{T} \frac{C_{t}^{\mathsf{P}}}{u_{t}(s)} \leq \sum_{t=1}^{T} b_{t}(s)$ . On the one hand, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)}\right) = \sum_ {t = 1} ^ {T} \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \mathbb {I} \{y _ {t} (s) = j \} \frac {C _ {t} ^ {\mathsf {P}}}{u _ {t} (s)} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \mathbb {I} \{y _ {t} (s) = j \} \frac {C _ {t} ^ {\mathsf {P}}}{2 ^ {- j - 1}} \\ = \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\sum_ {t = 1} ^ {T} \mathbb {I} \{y _ {t} (s) = j \} C _ {t} ^ {\mathsf {P}}}{2 ^ {- j - 1}} \\ \leq \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\min  \{2 L \sum_ {t = 1} ^ {T} \mathbb {I} \{y _ {t} (s) = j \} , C ^ {\mathsf {P}} \}}{2 ^ {- j - 1}}, \tag {20} \\ \end{array}
$$
where the second step uses the construction of the bin, i.e., if  $u_{t}(s)$  falls into a bin  $(2^{-j - 1}, 2^{-j}]$ , then, it is lower-bounded by  $2^{-j - 1}$ ; the fourth step follows the facts that  $C_t^{\mathsf{P}} \leq 2L$  and  $\sum_{t = 1}^{T}C_{t}^{\mathsf{P}} \leq C^{\mathsf{P}}$ . On the other hand, one can show
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} b _ {t} (s) = 4 L \sum_ {t = 1} ^ {T} \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\mathbb {I} \left\{y _ {t} (s) = j \right\} \mathbb {I} \left\{z _ {t} ^ {j} (s) \leq \frac {C ^ {\mathsf {P}}}{2 L} \right\}}{u _ {t} (s)} \\ \geq 4 L \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\sum_ {t = 1} ^ {T} \mathbb {I} \left\{y _ {t} (s) = j \right\} \mathbb {I} \left\{z _ {t} ^ {j} (s) \leq \frac {C ^ {\mathcal {P}}}{2 L} \right\}}{2 ^ {- j}} \\ \geq 2 L \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\min  \left\{\sum_ {t = 1} ^ {T} \mathbb {I} \left\{y _ {t} (s) = j \right\} , \frac {C ^ {\mathsf {P}}}{2 L} \right\}}{2 ^ {- j - 1}} \\ = \sum_ {j = 0} ^ {\lceil \log (| S | T) \rceil} \frac {\min  \left\{2 L \sum_ {t = 1} ^ {T} \mathbb {I} \left\{y _ {t} (s) = j \right\} , C ^ {\mathsf {P}} \right\}}{2 ^ {- j - 1}}, \tag {21} \\ \end{array}
$$
where the second step uses the construction of the bin, i.e., if  $u_{t}(s)$  falls into a bin  $(2^{-j - 1}, 2^{-j}]$ , then, it is upper-bounded by  $2^{-j}$ , and the third step follows the definitions of  $y_{t}(s)$  and  $z_{t}^{j}(s)$ .
Combining Eq. (20) and Eq. (21), we complete the proof of  $\sum_{t=1}^{T} \frac{C_t^{\mathsf{P}}}{u_t(s)} \leq \sum_{t=1}^{T} b_t(s)$ .
For the proof of  $\sum_{t=1}^{T} \widehat{q}_t(s) b_t(s) = \mathcal{O}(|S|C^P \log T)$ , one can show
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \widehat {q} _ {t} (s) b _ {t} (s) = 4 L \sum_ {t = 1} ^ {T} \sum_ {i = 0} ^ {\lceil \log (| S | T) \rceil} \frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (s) \mathbb {I} \{y _ {t} (s) = i \} \mathbb {I} \left\{z _ {t} ^ {i} (s) \leq \frac {C ^ {\mathsf {P}}}{2 L} \right\}}{u _ {t} (s)} \\ \leq 4 L \sum_ {t = 1} ^ {T} \sum_ {i = 0} ^ {\lceil \log (| S | T) \rceil} \mathbb {I} \{y _ {t} (s) = i \} \mathbb {I} \left\{z _ {t} ^ {i} (s) \leq \frac {C ^ {\mathsf {P}}}{2 L} \right\} \\ = 4 L \sum_ {i = 0} ^ {\lceil \log (| S | T) \rceil} \sum_ {t = 1} ^ {T} \mathbb {I} \{y _ {t} (s) = i \} \mathbb {I} \left\{z _ {t} ^ {i} (s) \leq \frac {C ^ {\mathsf {P}}}{2 L} \right\} \\ \leq 4 L \sum_ {i = 0} ^ {\lceil \log (| S | T) \rceil} \frac {C ^ {\mathsf {P}}}{2 L} \\ = \mathcal {O} \left(C ^ {\mathsf {P}} \log (| S | T)\right) = \mathcal {O} \left(C ^ {\mathsf {P}} \log (T)\right), \\ \end{array}
$$
where the first inequality uses the fact  $q^{\widehat{P}_t,\pi_t}(s)\leq u_t(s)$  , and the last equality uses  $|S|\le T$
A.6 Proof of Theorem 3.2
For REG, we choose  $\eta = \min \left\{\sqrt{\frac{|S|^2|A|\log(\iota)}{LT}},\frac{1}{8L}\right\}$ . First consider the case  $\eta \neq \frac{1}{8L}$ :
$$
\begin{array}{l} \operatorname {R E G} = \mathcal {O} \left(\frac {| S | ^ {2} | A | \log (\iota)}{\eta} + \eta (L | S | C ^ {\mathrm {P}} \log T + L T) + L T \delta\right) \\ \leq \mathcal {O} \left(\frac {| S | ^ {2} | A | \log (\iota)}{\eta} + \eta L T + | S | C ^ {\mathsf {P}} \log T + L T \delta\right) \\ = \mathcal {O} \left(| S | \sqrt {| A | \log (\iota) L T} + | S | C ^ {\mathsf {P}} \log T + L T \delta\right), \\ \end{array}
$$
where the second step uses  $\eta \leq \frac{1}{8L}$ , and the third step applies choice of  $\eta$  in the case of  $\eta \neq \frac{1}{8L}$ .
For the case of  $\eta = \frac{1}{8L}$ , we have  $T \leq 64L|S|^2 |A|\ln (\iota)$  and show
$$
\operatorname {R E G} = \mathcal {O} \left(L | S | ^ {2} | A | \log (\iota) + | S | C ^ {\mathsf {P}} \log T + L T \delta\right).
$$
Finally, choosing  $\delta = \frac{1}{T}$  and putting the bound above together with ERROR,  $\mathrm{BIAS}_1$ , and  $\mathrm{BIAS}_2$ , we complete the proof of Theorem 3.2.
B Omitted Details for Section 4
B.1 Bottom Layer Reduction: STABILISE
Proof of Theorem 4.3. Define indicators
$h_{t,j} = \mathbb{I}\{\mathsf{ALG}_j$  receives the feedback for episode  $t_j\}$
Now we consider the regret of  $\mathsf{ALG}_j$ . Notice that  $\mathsf{ALG}_j$  makes an update only when  $g_{t,j}h_{t,j} = 1$ . By the guarantee of the base algorithm (Definition 4.1), we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} (\ell_ {t} (\pi_ {t}) - \ell_ {t} (\pi)) g _ {t, j} h _ {t, j} \right] \\ \leq \mathbb {E} \left[ \sqrt {\beta_ {1} \sum_ {t = 1} ^ {T} g _ {t , j} h _ {t , j}} + \left(\beta_ {2} + \beta_ {3} \theta_ {j}\right) \max  _ {t \leq T} g _ {t, j} \right] + \Pr \left[ \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathrm {P}} g _ {t, j} h _ {t, j} > \theta_ {j} \right] L T. \tag {22} \\ \end{array}
$$
We first bound the last term: Notice that  $\mathbb{E}[h_{t,j}|g_{t,j}] = 2^{-j - 1}g_{t,j}$  by Algorithm 2. Therefore,
$$
\sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} \mathbb {E} \left[ h _ {t, j} \mid g _ {t, j} \right] = 2 ^ {- j - 1} \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} \leq 2 ^ {- j - 1} C ^ {\mathsf {P}} \tag {23}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} h _ {t, j} - \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] \\ \leq 2 \sqrt {2 \sum_ {t = 1} ^ {T} \left(C _ {t} ^ {\mathsf {P}}\right) ^ {2} g _ {t , j} \mathbb {E} \left[ h _ {t , j} \mid g _ {t , j} \right] \log (T)} + 4 L \log (T) \\ \leq 4 \sqrt {L \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t , j} \mathbb {E} [ h _ {t , j} | g _ {t , j} ] \log (T)} + 4 L \log (T) \quad \left(C _ {t} ^ {\mathsf {P}} \leq 2 L\right) \\ \leq \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathbb {P}} g _ {t, j} \mathbb {E} \left[ h _ {t, j} \mid g _ {t, j} \right] + 8 L \log (T) \quad (\text {A M - G M i n e q u a l i t y}) \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} h _ {t, j} \leq 2 \sum_ {t = 1} ^ {T} C _ {t} ^ {\mathsf {P}} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] + 8 L \log (T) \\ \leq 2 ^ {- j} C ^ {\mathsf {P}} + 8 L \log (T) \leq \theta_ {j} \\ \end{array}
$$
with probability at least  $1 - \frac{1}{T^2}$  using Eq. (23). Therefore, the last term in Eq. (22) is bounded by  $\frac{1}{T^2} LT \leq \frac{L}{T}$ .
Next, we deal with other terms in Eq. (22). Again, by  $\mathbb{E}[h_{t,j}|g_{t,j}] = 2^{-j - 1}g_{t,j}$ , Eq. (22) implies
$$
2 ^ {- j - 1} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} (\ell_ {t} (\pi_ {t}) - \ell_ {t} (\pi)) g _ {t, j} \right] \leq \mathbb {E} \left[ \sqrt {2 ^ {- j - 1} \beta_ {1} \sum_ {t = 1} ^ {T} g _ {t , j}} + (\beta_ {2} + \beta_ {3} \theta_ {j}) \max _ {t \leq T} g _ {t, j} \right] + \frac {L}{T}.
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(\pi_ {t}\right) - \ell_ {t} (\pi)\right) g _ {t, j} \right]
$$
$$
\leq \mathbb {E} \left[ \sqrt {\frac {1}{2 ^ {- j - 1}} \beta_ {1} \sum_ {t = 1} ^ {T} g _ {t , j}} + \left(\frac {\beta_ {2}}{2 ^ {- j - 1}} + \frac {\beta_ {3} \theta_ {j}}{2 ^ {- j - 1}}\right) \max _ {t \leq T} g _ {t, j} \right] + \frac {L}{T 2 ^ {- j - 1}}
$$
$$
\leq \mathbb {E} \left[ \sqrt {\beta_ {1} \sum_ {t = 1} ^ {T} \frac {2 g _ {t , j}}{w _ {t}}} + \left(\frac {2 \beta_ {2} + 1 6 \beta_ {3} L \log (T)}{2 ^ {- j}} + 4 \beta_ {3} C ^ {\mathsf {P}}\right) \max  _ {t \leq T} g _ {t, j} \right] + \frac {L}{T 2 ^ {- j - 1}}.
$$
(using that when  $g_{t,j} = 1, \frac{1}{2^{-j - 1}} \leq \frac{2}{w_t}$ , and the definition of  $\theta_j$ )
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(\pi_ {t}\right) - \ell_ {t} (\pi)\right) \mathbb {I} \left\{w _ {t} \geq \frac {1}{T} \right\} \right] \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sqrt {N \beta_ {1} \sum_ {t = 1} ^ {T} \frac {1}{w _ {t}}} + (\beta_ {2} + \beta_ {3} L \log (T)) \frac {1}{\min _ {t \leq T} w _ {t}} + N \beta_ {3} C ^ {\mathsf {P}} \right] + L\right) \\ \leq \mathcal {O} \left(\mathbb {E} \left[ \sqrt {\beta_ {1} T \log (T) \rho_ {T}} + (\beta_ {2} + \beta_ {3} L \log (T)) \rho_ {T} \right] + \beta_ {3} C ^ {\mathsf {P}} \log T + L\right) \\ \end{array}
$$
where  $N \leq \mathcal{O}(\log T)$  is the number of  $\mathsf{ALG}_j$ 's that has been executed at least once.
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(\pi_ {t}\right) - \ell_ {t} (\pi)\right) \mathbb {I} \left\{w _ {t} \leq \frac {1}{T} \right\} \right] \leq L T \mathbb {E} \left[ \mathbb {I} \left\{\rho_ {T} \geq T \right\} \right] \leq L \mathbb {E} \left[ \rho_ {T} \right].
$$
Combining the two parts and using the assumption  $\beta_{2} \geq L$  finishes the proof.
B.2 Top Layer Reduction: Corral
In this subsection, we use a base algorithm that satisfies Definition 4.2 to construct an algorithm with  $\sqrt{T} + C^{\mathsf{P}}$  regret under unknown  $C^{\mathsf{P}}$ . The idea is to run multiple base algorithms, each with a different hypothesis on  $C^{\mathsf{P}}$ ; on top of them, run another multi-armed bandit algorithm to adaptively choose among them. The goal is to let the top-level bandit algorithm perform almost as well as the best base algorithm. This is the Corral idea outlined in Agarwal et al. [2017], Foster et al. [2020], Luo et al. [2022], and the algorithm is presented in Algorithm 3.
Lemma B.2.1. The FTRL algorithm over a convex subset  $\Omega$  of the  $(M - 1)$ -dimensional simplex  $\Delta(M)$ :
Proof of Theorem 4.4. The Corral algorithm is essential an FTRL with log-barrier regularizer. To apply Lemma B.2.1, we first verify the condition  $\eta w_{t,i}|\ell_{t,i}| \leq \frac{1}{2}$  where  $\ell_{t,i} = \hat{c}_{t,i} - r_{t,i}$ . By our choice of  $\eta$ ,
$$
\eta w _ {t, i} \left| \hat {c} _ {t, i} \right| \leq \eta c _ {t, i} \leq \frac {1}{4}, \quad (\text {b e c a u s e} \beta_ {2} \geq L \text {b y D e f i n i t i o n 4 . 2})
$$
Algorithm 3 (A Variant of) Corral
Initialize: a log-barrier algorithm with each arm being an instance of any base algorithm satisfying Definition 4.2. The hypothesis on  $C^{\mathsf{P}}$  is set to  $2^{i}$  for arm  $i$  ( $i = 1, 2, \dots, M \triangleq \lceil \log_2 T \rceil$ ).