SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $g_{\tau}(s,a) = Q^{\widetilde{P}_i,\pi_\tau}(s,a;\widehat{\ell}_\tau) - V^{\widetilde{P}_i,\pi_\tau}(s;\widehat{\ell}_\tau)$  for any state-action pair  $(s,a)$  ( $Q$  is the standard  $Q$ -function; see Appendix C for formal definition). With this perspective, we follow the idea of Ito [2021] and propose the following learning rate schedule:
Definition 5.2. (Adaptive learning rate for log-barrier) For any  $t$ , if it is the starting episode of an epoch, we set  $\gamma_{t}(s,a) = 256L^{2}|S|$ ; otherwise, we set  $\gamma_{t + 1}(s,a) = \gamma_t(s,a) + \frac{D\nu_t(s,a)}{2\gamma_t(s,a)}$  where  $D = 1 / \log (\iota)$ ,  $\nu_{t}(s,a) = q^{\widetilde{P}_{i(t)},\pi_{t}}(s,a)^{2}\left(Q^{\widetilde{P}_{i(t)},\pi_{t}}(s,a;\widehat{\ell_{t}}) - V^{\widetilde{P}_{i(t)},\pi_{t}}(s;\widehat{\ell_{t}})\right)^{2}$ , and  $i(t)$  is the epoch index to which episode  $t$  belongs.
Such a learning rate schedule is critical for the analysis in obtaining a certain self-bounding quantity and eventually deriving the gap-dependent bound. This concludes the design of our algorithm; see Appendix C for more details.
In this work, we propose online RL algorithms that can handle both adversarial losses and adversarial transitions, with regret gracefully degrading in the degree of maliciousness of the adversary. Specifically, we achieve  $\tilde{\mathcal{O}} (\sqrt{T} +C^{\mathsf{P}})$  regret where  $C^{\mathsf{P}}$  measures how adversarial the transition functions are, even when  $C^{\mathsf{P}}$  is unknown. Moreover, we show that further refinements of the algorithm not only maintain the same regret bound, but also simultaneously adapt to easier environments, with the caveat that  $C^{\mathsf{P}}$  must be known ahead of time. We leave how to further remove this restriction as a key future direction.
TJ and HL are supported by NSF Award IIS-1943607 and a Google Research Scholar award. CR acknowledges partial support by the Independent Research Fund Denmark, grant number 9040-00361B.
Yasin Abbasi Yadjkori, Peter L Bartlett, Varun Kanade, Yevgeny Seldin, and Csaba Szepesvari. Online learning in markov decision processes with adversarially chosen transition probability distributions. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2017.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2008.
Yifang Chen, Simon Du, and Kevin Jamieson. Improved corruption robust algorithms for episodic reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Nonstationary reinforcement learning: The blessing of (more) optimism. Management Science, 2023.
Christoph Dann, Chen-Yu Wei, and Julian Zimmert. Best of both worlds policy optimization. arXiv preprint arXiv:2302.09408, 2023.
Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathematics of Operations Research, 2009.
Dylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Eva Tardos. Learning in games: Robustness of fast convergence. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Pratik Gajane, Ronald Ortner, and Peter Auer. A sliding-window algorithm for markov decision processes with arbitrarily changing rewards and transitions. arXiv preprint arXiv:1805.10066, 2018.
Shinji Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2021.
Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Tiancheng Jin and Haipeng Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tiancheng Jin, Longbo Huang, and Haipeng Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. PhD thesis, University College London, 2003.
Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Haipeng Luo, Mengxiao Zhang, Peng Zhao, and Zhi-Hua Zhou. Corralling a larger band of bandits: A case study on switching regret for linear bandits. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2022.
Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the Annual ACM SIGACT Symposium on Theory of Computing, 2018.
Thodoris Lykouris, Max Simchowitz, Aleksandrs Slivkins, and Wen Sun. Corruption robust exploration in episodic reinforcement learning. arXiv preprint arXiv:1911.08689, 2019.
Gergely Neu, Andras Antos, András György, and Csaba Szepesvári. Online markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2010a.
Gergely Neu, András György, and Csaba Szepesvári. The online loop-free stochastic shortest-path problem. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2010b.
Aldo Pacchiano, Christoph Dann, and Claudio Gentile. Best of both worlds model selection. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial markov decision processes. In Proceedings of the International Conference on Machine Learning (ICML), 2019a.
Aviv Rosenberg and Yishay Mansour. Online stochastic shortest path with bandit feedback and unknown transition function. Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Online learning in unknown markov games. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2018.
Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the International Conference on Computational Learning Theory (COLT), 2021.
Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corruption. In International Conference on Machine Learning, pages 12391-12401. PMLR, 2021.
Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adversarial bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
A Omitted Details for Section 3 14
A.1 Equivalent Definition of Amortized Bonuses 15
A.2 Bounding ERROR 15
A.3 Bounding  $\mathbf{BIAS}_1$  16
A.4 Bounding REG 18
A.5 Bounding  $\mathbf{BIAS}_2$  20
A.6 Proof of Theorem 3.2 22
B Omitted Details for Section 4 23
B.1 Bottom Layer Reduction: STABILISE 23
B.2 Top Layer Reduction: Corral 24
C Omitted Details for Section 5 27
C.1 Description of the Algorithm 28
C.2 Self-Bounding Properties of the Regret 28
C.3 Regret Decomposition of  $\operatorname{Reg}_T(\pi^\star)$  and Proof of Lemma C.2.3 31
C.4 Proof of Lemma C.3.1 33
C.5 Proof of Lemma C.3.2 33
C.6 Proof of Lemma C.3.3 34
C.6.1 Bounding Term 1 35
C.6.2 Bounding Term 2 36
C.6.3 Bounding Term 3 37
C.7 Proof of Lemma C.3.4 39
C.7.1 Properties of the Learning Rate 39
C.7.2 Bounding  $\mathrm{ESTREG}_i(\pi^{\star})$  for Varying Learning Rate 44
C.7.3 Bounding ESTREG 49
C.8 Properties of Optimistic Transition 50
D Supplementary Lemmas 52
D.1 Expectation 52
D.2 Confidence Bound with Known Corruption 52
D.3 Difference Lemma 57
D.4 Loss Shifting Technique with Optimistic Transition 62
D.5 Estimation Error 63
Important Notations Throughout the appendix, we denote the transition associated with the occupancy measure  $\widehat{q}_t$  by  $\widehat{P}_t$  so that  $\widehat{q}_t = q^{\widehat{P}_t,\pi_t}$ . Recall  $i(t)$  is the epoch index to which episode  $t$  belongs. Importantly, the notation  $m_i(s,a)$  (and similarly  $m_i(s,a,a')$ ), which is a changing variable in the algorithms, denotes the initial value of this counter in all analysis, that is, the total number of visits to  $(s,a)$  prior to epoch  $i$ . Finally, for convenience, we define  $\widehat{m}_i(s,a) = \max \left\{m_i(s,a),C^{\mathsf{P}} + \log (\iota)\right\}$ , and it can be verified that the confidence interval, defined in Eq. (4) as
$$
B _ {i} (s, a, s ^ {\prime}) = \min  \left\{1, 1 6 \sqrt {\frac {\bar {P} _ {i} (s ^ {\prime} | s , a) \log (\iota)}{m _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{m _ {i} (s , a)} \right\},
$$
can be equivalently written as
$$
B _ {i} (s, a, s ^ {\prime}) = \min \left\{1, 1 6 \sqrt {\frac {\bar {P} _ {i} (s ^ {\prime} | s , a) \log (\iota)}{\widehat {m} _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i} (s , a)} \right\}
$$
since whenever  $\widehat{m}_i(s,a)\neq m_i(s,a)$ , the two definitions both lead to a value of 1.
A Omitted Details for Section 3
In this section, we provide more details of the modified UOB-REPS algorithm as shown in Algorithm 1. In the algorithm, the occupancy measure for each episode  $t$  is computed as
$$
\widehat {q} _ {t + 1} = \underset {q \in \Omega \left(\mathcal {P} _ {i (t + 1)}\right)} {\operatorname {a r g m i n}} \eta \left\langle q, \widehat {\ell_ {t}} - b _ {t} \right\rangle + D _ {\phi} (q, \widehat {q} _ {t}), \tag {10}
$$
where  $\eta > 0$  is the learning rate and  $D_{\phi}(q, q')$  is the Bregman divergence defined as
$$
D _ {\phi} \left(q, q ^ {\prime}\right) = \phi (q) - \phi \left(q ^ {\prime}\right) - \left\langle \nabla \phi \left(q ^ {\prime}\right), q - q ^ {\prime} \right\rangle . \tag {11}
$$
The Bregman divergence is induced by the log-barrier regularizer  $\phi$  given as:
$$
\phi (q) = \sum_ {k = 0} ^ {L - 1} \sum_ {s \in S _ {k}} \sum_ {a \in A} \sum_ {s ^ {\prime} \in S _ {k + 1}} \log \left(\frac {1}{q (s , a , s ^ {\prime})}\right). \tag {12}
$$
We note that the loss estimator is constructed based upon upper occupancy bound  $u_{t}$ , which can be efficiently computed by COMP-UOB [Jin et al., 2020].
In the rest of this section, we prove Theorem 3.2 which shows that the expected regret is bounded by
$$
\mathcal {O} \left(L | S | \sqrt {| A | T \log (\iota)} + L | S | ^ {4} | A | \log^ {2} (\iota) + C ^ {\mathsf {P}} L | S | ^ {4} | A | \log (\iota)\right).
$$
Our analysis starts from a regret decomposition similar to Jin et al. [2020], but with the amortized bonuses taken into account:
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P _ {t}, \hat {\pi}}, \ell_ {t} \right\rangle \right] \\ = \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t} , \pi_ {t}} - q ^ {\widehat {P} _ {t} , \pi_ {t}}, \ell_ {t} \right\rangle \right]} _ {\text {E R R O R}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}}, \ell_ {t} - \widehat {\ell} _ {t} \right\rangle \right]} _ {\text {B I A S} _ {1}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}} - q ^ {P , \widehat {\pi}}, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right]} _ {\text {R E G}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P , \widehat {\pi}}, \widehat {\ell} _ {t} - \ell_ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\langle q ^ {\widehat {P} _ {t} , \pi_ {t}} - q ^ {P , \widehat {\pi}}, b _ {t} \right\rangle \right]} _ {\text {B I A S} _ {2}} \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \hat {\pi}} - q ^ {P _ {t}, \hat {\pi}}, \ell_ {t} \right\rangle \right], \\ \end{array}
$$
where the last term can be directly bounded by  $\mathcal{O}\left(LC^{\mathsf{P}}\right)$  using Corollary D.3.7, and ERROR,  $\mathrm{BIAS}_1$ , REG, and  $\mathrm{BIAS}_2$  are analyzed in Appendix A.2, Appendix A.3, Appendix A.4 and Appendix A.5 respectively.
A.1 Equivalent Definition of Amortized Bonuses
For ease of exposition, we show an alternative definition of the amortized bonus  $b_{t}(s)$ , which is equivalent to Eq. (7) but is useful for our following analysis. Recall from Eq. (7) that
$$
b _ {t} (s) = b _ {t} (s, a) = \left\{ \begin{array}{l l} \frac {4 L}{u _ {t} (s)} & \text {i f} \sum_ {\tau = 1} ^ {t} \mathbb {I} \{\lceil \log_ {2} u _ {\tau} (s) \rceil = \lceil \log_ {2} u _ {t} (s) \rceil \} \leq \frac {C ^ {\mathrm {P}}}{2 L}, \\ 0 & \text {e l s e ,} \end{array} \right. \tag {13}
$$
In Eq. (13), the value of  $b_{t}(s)$  relies on the cumulative sum of  $\mathbb{I}\{\lceil \log_2u_\tau (s)\rceil = \lceil \log_2u_t(s)\rceil \}$  and whether the cumulative sum exceeds the threshold  $\frac{C^{\mathrm{P}}}{2L}$ . To reconstruct this, we define  $y_{t}(s)$  as the unique integer value  $j = -\lceil \log_2u_t(s)\rceil$  such that  $u_{t}(s)\in (2^{-j - 1},2^{-j}]$  and define  $z_{t}^{j}(s)$  as the total number of times from episode  $\tau = 1$  to  $\tau = t$  where  $y_{\tau}(s) = j$  holds. With these definitions, the condition under which  $b_{t}(s) = \frac{4L}{u_{t}(s)}$  can be represented by  $\sum_{j}\mathbb{I}\{y_{t}(s) = j\} \mathbb{I}\left\{z_{t}^{j}(s)\leq C^{\mathrm{P}} / 2L\right\}$  where the summation is over all integers that  $-\lceil \log_2u_t(s)\rceil$  can take. Notice that since  $u_{t}(s)\geq \frac{1}{|S|T}$  holds for all  $s$  and  $t$  (see Lemma D.2.8), the largest possible integer value we need to consider is  $\lceil \log_2(|S|T)\rceil$ . Therefore,  $b_{t}(s)$  can be equivalently defined as:
$$
b _ {t} (s) = b _ {t} (s, a) = \frac {4 L}{u _ {t} (s)} \sum_ {j = 0} ^ {\lceil \log_ {2} (| S | T) \rceil} \mathbb {I} \left\{y _ {t} (s) = j \right\} \mathbb {I} \left\{z _ {t} ^ {j} (s) \leq \frac {C ^ {\mathrm {P}}}{2 L} \right\}, \forall s \in S, \forall t \in \{1, \dots , T \}. \tag {14}
$$
A.2 Bounding ERROR
Lemma A.2.1 (Bound of ERROR). Algorithm 1 ensures
$$
\operatorname {E R R O R} = \mathcal {O} \left(L | S | \sqrt {| A | T \log (\iota)} + L | S | ^ {2} | A | \log (\iota) \log (T) + C ^ {P} L | S | | A | \log (T) + \delta L T\right).
$$
Proof. For this proof, we consider two cases, whether the event  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  holds or not, where  $\mathcal{E}_{\mathrm{CON}}$ , defined above Eq. (4), and  $\mathcal{E}_{\mathrm{EST}}$ , defined in Proposition D.5.2, are both high probability events.
Suppose that  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  holds. We have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {\widehat {P} _ {t}, \pi_ {t}}, \ell_ {t} \right\rangle \\ \leq \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {s \in S _ {k}} \sum_ {a \in A} \left| q ^ {P _ {t}, \pi_ {t}} (s, a) - q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \right| \\ = \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {s \in S _ {k}} \left| q ^ {\widehat {P} _ {t}, \pi_ {t}} (s) - q ^ {P, \pi_ {t}} (s) \right| \\ \leq \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \sum_ {s \in S _ {k}} \sum_ {h = 0} ^ {k - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} \sum_ {w \in S _ {h + 1}} q ^ {P _ {t}, \pi_ {t}} (u, v) \left| \widehat {P} _ {t} (w | u, v) - P _ {t} (w | u, v) \right| q ^ {\widehat {P} _ {t}, \pi_ {t}} (s | w) \\ \leq L \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \left\| \widehat {P} _ {t} (\cdot | u, v) - P _ {t} (\cdot | u, v) \right\| _ {1} \\ \leq L \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \left(\| P _ {t} (\cdot | u, v) - P (\cdot | u, v) \| _ {1} + \left\| \widehat {P} _ {t} (\cdot | u, v) - P (\cdot | u, v) \right\| _ {1}\right) \\ \leq L \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \left\| \widehat {P} _ {t} (\cdot | u, v) - P (\cdot | u, v) \right\| _ {1} + L \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} C _ {t, h} ^ {\mathsf {P}} \\ = L \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \| \widehat {P} _ {t} (\cdot | u, v) - P (\cdot | u, v) \| _ {1} + L C ^ {\mathbf {P}}, \tag {15} \\ \end{array}
$$
where the first step follows from the fact that  $\ell_t(s,a)\in [0,1]$  for any  $(s,a)$ ; the third step applies Lemma D.3.3; the fourth step rearranges the summation and uses the fact that  $\sum_{s,a}q^{\widehat{P}_t,\pi_t}(s,a|w)\leq L$ ; and in the sixth step we define  $C_{t,h}^{\mathsf{P}} = \max_{s\in S_h,a\in A}\| P_t(\cdot |s,a) - P(\cdot |s,a)\| _1$ , so that  $C^\mathsf{P} = \sum_{t = 1}^{T}\sum_{h = 0}^{L - 1}C_{t,h}^{\mathsf{P}}$ . We continue to bound the first term above as:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \left\| \widehat {P} _ {t} (\cdot | u, v) - P (\cdot | u, v) \right\| _ {1} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {L - 1} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \left(\left\| \widehat {P} _ {t} (\cdot | u, v) - \bar {P} _ {i (t)} (\cdot | u, v) \right\| _ {1} + \left\| \bar {P} _ {i (t)} (\cdot | u, v) - P (\cdot | u, v) \right\| _ {1}\right) \\ \leq \sum_ {h = 0} ^ {L - 1} \sum_ {t = 1} ^ {T} \sum_ {u \in S _ {h}} \sum_ {v \in A} q ^ {P _ {t}, \pi_ {t}} (u, v) \mathcal {O} \left(\sqrt {\frac {| S _ {h + 1} | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}} + \frac {C ^ {\mathsf {P}} + | S _ {h + 1} | \log (\iota)}{\widehat {m} _ {i (t)} (u , v)}\right) \\ \leq \mathcal {O} \left(\sum_ {h = 0} ^ {L - 1} \left(\sqrt {\left| S _ {h} \right| \left| S _ {h + 1} \right| \left| A \right| T \log (\iota)} + \left| S _ {h} \right| \left| A \right| \log (T) \left(\left| S _ {h + 1} \right| \log (\iota) + C ^ {\mathbb {P}}\right) + \log (\iota)\right)\right) \\ \leq \mathcal {O} \left(\sum_ {h = 0} ^ {L - 1} \left(\left(| S _ {h} | + | S _ {h + 1} |\right) \sqrt {| A | T \log (\iota)} + | S _ {h} | | A | \log (T) \left(| S _ {h + 1} | \log (\iota) + C ^ {\mathbf {P}}\right) + \log (\iota)\right)\right) \\ \leq \mathcal {O} \left(| S | \sqrt {| A | T \log (\iota)} + | S | ^ {2} | A | \log (\iota) \log (T) + C ^ {\mathrm {P}} | S | | A | \log (T)\right), \tag {16} \\ \end{array}
$$
where the first step uses the triangle inequality; the second step applies Corollary D.2.6 to bound two norm terms based on the fact that  $\widehat{P}_t, \bar{P}_{i(t)} \in \mathcal{P}_{i(t)}$ ; the third step follows the definition of  $\mathcal{E}_{\mathrm{EST}}$  from Proposition D.5.2; the fourth step uses the AM-GM inequality. Putting the result of Eq. (16) into Eq. (15) yields the first three terms of the claimed bound.
Now suppose that  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  does not hold. We trivially bound  $\sum_{t=1}^{T} \left\langle q^{P_t, \pi_t} - q^{\widehat{P}_t, \pi_t}, \ell_t \right\rangle$  by  $LT$ . As the probability that this case occurs is at most  $\mathcal{O}(\delta)$ , this case contributes to at most  $\mathcal{O}(\delta LT)$  regret. Finally, combining these two cases and applying Lemma D.1.1 finishes the proof.
A.3 Bounding  $\mathbf{BIAS}_1$
Lemma A.3.1 (Bound of  $\mathrm{BIAS}_1$ ). Algorithm 1 ensures
$$
\operatorname {B I A S} _ {1} = \mathcal {O} \left(L | S | \sqrt {| A | T \log (\iota)} + | S | ^ {4} | A | \log^ {2} (\iota) + C ^ {P} L | S | ^ {4} | A | \log (\iota) + \delta L T\right).
$$
Proof. We can rewrite  $\mathrm{BIAS}_1$  as
$$
\operatorname {B I A S} _ {1} = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \ell_ {t} - \widehat {\ell} _ {t} \right\rangle \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\langle q ^ {\widehat {P} _ {t}, \pi_ {t}}, \mathbb {E} _ {t} \left[ \ell_ {t} - \widehat {\ell} _ {t} \right] \right\rangle \right],
$$
where the second step uses the law of total expectation and the fact that  $q^{\widehat{P}_t,\pi_t}$  is deterministic given all the history up to  $t$ . For any state-action pair  $(s,a)$ , we have
$$
\mathbb {E} _ {t} \left[ \ell_ {t} (s, a) - \widehat {\ell} _ {t} (s, a) \right] = \ell_ {t} (s, a) \left(1 - \frac {q ^ {P _ {t} , \pi_ {t}} (s , a)}{u _ {t} (s , a)}\right) = \ell_ {t} (s, a) \left(\frac {u _ {t} (s , a) - q ^ {P _ {t} , \pi_ {t}} (s , a)}{u _ {t} (s , a)}\right).
$$
Then, we can further rewrite and bound  $\mathrm{BIAS}_1$  as:
$$
\begin{array}{l} \mathbf {B I A S} _ {1} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \ell_ {t} (s, a) \left(\frac {u _ {t} (s , a) - q ^ {P _ {t} , \pi_ {t}} (s , a)}{u _ {t} (s , a)}\right) \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \left(\frac {\left| u _ {t} (s , a) - q ^ {P _ {t} , \pi_ {t}} (s , a) \right|}{u _ {t} (s , a)}\right) \right] \\ \end{array}
$$
$$
\leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \in S} \left| u _ {t} (s) - q ^ {P _ {t}, \pi_ {t}} (s) \right| \right],
$$
where the third step follows from the fact that  $q^{\widehat{P}_t,\pi_t}(s,a)\leq u_t(s,a)$  according to the definition of the upper occupancy measure.
Similar to the proof of Lemma A.2.1, we first consider the case when the high probability event  $\mathcal{E}_{\mathrm{CON}} \wedge \mathcal{E}_{\mathrm{EST}}$  holds:
$$
\begin{array}{l} \sum_ {s \in S} \left| u _ {t} (s) - q ^ {P _ {t}, \pi_ {t}} (s) \right| \\ \leq \sum_ {s \in S} | u _ {t} (s) - q ^ {P, \pi_ {t}} (s) | + \sum_ {s \in S} | q ^ {P, \pi_ {t}} (s) - q ^ {P _ {t}, \pi_ {t}} (s) | \\ \leq \sum_ {s \in S} \left| u _ {t} (s) - q ^ {P, \pi_ {t}} (s) \right| + L C _ {t} ^ {\mathsf {P}} \\ \leq \mathcal {O} \left(L C _ {t} ^ {\mathsf {P}} + \sum_ {s \in S} \sum_ {k = 0} ^ {k (s) - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}} q ^ {P, \pi_ {t}} (s | w)\right) \\ + \mathcal {O} \left(| S | ^ {3} \sum_ {s, a} q ^ {P, \pi_ {t}} (s, a) \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right) \\ \leq \mathcal {O} \left(L C _ {t} ^ {\mathsf {P}} + L \sum_ {k = 0} ^ {L - 1} \sum_ {(u, v, w) \in W _ {k}} q ^ {P, \pi_ {t}} (u, v) \sqrt {\frac {P (w | u , v) \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}}\right) \\ + \mathcal {O} \left(\left| S \right| ^ {3} \sum_ {s, a} q ^ {P, \pi_ {t}} (s, a) \frac {C ^ {\mathsf {P}} + \log (\iota)}{\widehat {m} _ {i (t)} (s , a)}\right), \\ \end{array}
$$
where the first step uses the triangle inequality; the second step uses Corollary D.3.6; the third step uses Lemma D.3.8; the last step follows from the fact that  $\sum_{s\in S}q^{P,\pi_t}(s|w)\leq L$
Taking the summation over all episodes yields the following: