SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Initialize:  $\rho_{0,i} = M$ ,  $\forall i$
$$
w _ {t} = \operatorname * {a r g m i n} _ {w \in \Delta (M), w _ {i} \geq \frac {1}{T}, \forall i} \left\{\left\langle w, \sum_ {\tau = 1} ^ {t - 1} \left(\hat {c} _ {\tau} - r _ {\tau}\right) \right\rangle + \frac {1}{\eta} \sum_ {i = 1} ^ {M} \log \frac {1}{w _ {i}} \right\}
$$
where  $\eta = \frac{1}{4(\sqrt{\beta_1T} + \beta_2)}$ .
Execute the  $\pi_t$  output by instance  $i_t$
Receive the loss  $c_{t,i_t}$  for policy  $\pi_t$  (whose expectation is  $\ell_t(\pi_t)$ ) and send it to instance  $i_t$ .
Define for all  $i$ :
$$
\hat {c} _ {t, i} = \frac {c _ {t , i} \mathbb {I} [ i _ {t} = i ]}{w _ {t , i}},
$$
$$
r _ {t, i} = \sqrt {\beta_ {1} T} \left(\sqrt {\rho_ {t , i}} - \sqrt {\rho_ {t - 1 , i}}\right) + \beta_ {2} \left(\rho_ {t, i} - \rho_ {t - 1, i}\right).
$$
$$
\eta w _ {t, i} r _ {t, i} = \eta \sqrt {\beta_ {1} T} w _ {t, i} \left(\sqrt {\rho_ {t , i}} - \sqrt {\rho_ {t - 1 , i}}\right) + \eta \beta_ {2} w _ {t, i} \left(\rho_ {t, i} - \rho_ {t - 1, i}\right).
$$
$$
\begin{array}{l} \eta w _ {t, i} r _ {t, i} \\ \leq \eta \sqrt {\beta_ {1} T} \frac {1}{\rho_ {t , i}} \left(\sqrt {\rho_ {t , i}} - \sqrt {\rho_ {t - 1 , i}}\right) + \eta \beta_ {2} \frac {1}{\rho_ {t , i}} \left(\rho_ {t, i} - \rho_ {t - 1, i}\right) \\ = \eta \sqrt {\beta_ {1} T} \left(\frac {1}{\sqrt {\rho_ {t , i}}} - \frac {\sqrt {\rho_ {t - 1 , i}}}{\rho_ {t , i}}\right) + \eta \beta_ {2} \left(1 - \frac {\rho_ {t - 1 , i}}{\rho_ {t , i}}\right) \\ \leq \eta \sqrt {\beta_ {1} T} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) + \eta \beta_ {2} \left(1 - \frac {\rho_ {t - 1 , i}}{\rho_ {t , i}}\right) \quad \left(\frac {1}{\sqrt {a}} - \frac {\sqrt {b}}{a} \leq \frac {1}{\sqrt {b}} - \frac {1}{\sqrt {a}} \text {f o r} a, b > 0\right) \tag {24} \\ \leq \eta \sqrt {\beta_ {1} T} + \eta \beta_ {2} \quad (\rho_ {t, i} \geq 1) \\ = \frac {1}{4} \quad (\text {d e f i n i t i o n o f} \eta) \\ \end{array}
$$
which can be combined to get the desired property  $\eta w_{t,i}|\hat{c}_{t,i} - r_{t,i}|\leq \frac{1}{2}$
Hence, by the regret guarantee of log-barrier FTRL (Lemma B.2.1), we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(c _ {t, i _ {t}} - c _ {t, i ^ {\star}}\right) \right] \\ \leq \mathcal {O} \left(\frac {M \log T}{\eta} + \eta \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t , i} ^ {2} (\hat {c} _ {t , i} - r _ {t , i}) ^ {2}} _ {\text {s t a b i l i t y - t e r m}} \right]\right) + \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \left(\sum_ {i = 1} ^ {M} w _ {t , i} r _ {t , i} - r _ {t , i ^ {\star}}\right)} _ {\text {b o u n s - t e r m}} \right] \\ \end{array}
$$
where  $i^*$  is the smallest  $i$  such that  $2^i$  upper bounds the true corruption amount  $C^\mathsf{P}$ .
Bounding stability-term:
$$
\text {s t a b i l i t y} \leq 2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} \left(\hat {c} _ {t, i} ^ {2} + r _ {t, i} ^ {2}\right)
$$
$$
2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} \hat {c} _ {t, i} ^ {2} = 2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} c _ {t, i} ^ {2} \mathbb {I} \{i _ {t} = i \} \leq \mathcal {O} (\eta L ^ {2} T)
$$
$$
2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} r _ {t, i} ^ {2} \leq 4 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} (\sqrt {\beta_ {1} T}) ^ {2} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) ^ {2} + 4 \eta \beta_ {2} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(1 - \frac {\rho_ {t - 1 , i}}{\rho_ {t , i}}\right) ^ {2}
$$
(continue from Eq. (24))
$$
\leq 4 \eta \beta_ {1} T \times \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) + 4 \eta \beta_ {2} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \ln \frac {\rho_ {t , i}}{\rho_ {t - 1 , i}}
$$
$$
\left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}} \leq 1 \text {a n d} 1 - a \leq - \ln a\right)
$$
$$
\leq 4 \eta \beta_ {1} T M ^ {\frac {3}{2}} + 4 \eta \beta_ {2} M \ln T. (t e l e s c o p i n g a n d u s i n g \rho_ {0, i} = M a n d \rho_ {T, i} \leq T)
$$
Bounding bonus-term:
$$
\mathbf {b o u n s - t e r m} = \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} r _ {t, i} - \sum_ {t = 1} ^ {T} r _ {t, i ^ {\star}}
$$
$$
\leq \sqrt {\beta_ {1} T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) + \beta_ {2} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \log \frac {\rho_ {t , i}}{\rho_ {t - 1 , i}}
$$
$$
- \left(\sqrt {\rho_ {T , i ^ {\star}} \beta_ {1} T} + \rho_ {T, i ^ {\star}} \beta_ {2} - \sqrt {\rho_ {0 , i ^ {\star}} \beta_ {1} T} - \rho_ {0, i ^ {\star}} \beta_ {2}\right)
$$
(continue from Eq. (24) and using  $1 - a \leq -\ln a$ )
$$
\leq \mathcal {O} \left(\sqrt {\beta_ {1} T} M ^ {\frac {3}{2}} + \beta_ {2} M \log T\right) - \left(\sqrt {\rho_ {T , i ^ {\star}} \beta_ {1} T} + \rho_ {T, i ^ {\star}} \beta_ {2}\right).
$$
Combining the two terms and using  $\eta = \Theta \left(\frac{1}{\sqrt{\beta_1T} + \beta_2}\right)$ ,  $M = \Theta (\log T)$ , we get
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(\pi_ {t}\right) - c _ {t, i ^ {\star}}\right) \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(c _ {t, i _ {t}} - c _ {t, i ^ {\star}}\right) \right] \\ = \mathcal {O} \left(\sqrt {\beta_ {1} T \log^ {3} T} + \beta_ {2} \log^ {2} T\right) - \mathbb {E} \left[ \sqrt {\rho_ {T , i ^ {\star}} \beta_ {1} T} + \rho_ {T, i ^ {\star}} \beta_ {2} \right] \tag {25} \\ \end{array}
$$
On the other hand, by Definition 4.2 and that  $C^{\mathsf{P}} \in [2^{i^{\star} - 1}, 2^{i^{\star}}]$ , we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(c _ {t, i ^ {\star}} - \ell_ {t} (\stackrel {\circ} {\pi})\right) \right] \leq \mathbb {E} \left[ \sqrt {\rho_ {T , i ^ {\star}} \beta_ {1} T} + \rho_ {T, i ^ {\star}} \beta_ {2} \right] + 2 \beta_ {3} C ^ {\mathsf {P}}. \tag {26}
$$
Combining Eq. (25) and Eq. (26), we get
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} (\ell_ {t} (\pi_ {t}) - \ell_ {t} (\stackrel {\circ} {\pi})) \right] \leq \mathcal {O} \left(\sqrt {\beta_ {1} T \log^ {3} T} + \beta_ {2} \log^ {2} T\right) + 2 \beta_ {3} C ^ {\mathsf {P}},
$$
Algorithm 4 Algorithm with Optimistic Transition Achieving Gap-Dependent Bounds (Known  $C^{\mathbb{P}}$ )
Input: confidence parameter  $\delta \in (0,1)$ .
Initialize:  $\forall (s,a)$ , learning rate  $\gamma_{1}(s,a) = 256L^{2}|S|$ ; epoch index  $i = 1$  and epoch starting time  $t_i = 1$ ;  $\forall (s,a,s')$ , set counters  $m_{1}(s,a) = m_{1}(s,a,s') = m_{0}(s,a) = m_{0}(s,a,s') = 0$ ; empirical transition  $\bar{P}_1$  and confidence width  $B_1$  based on Eq. (2); optimistic transition  $\widetilde{P}_i$  by Definition C.1.1. for  $t = 1,\dots,T$  do
Let  $\phi_t$  be defined in Eq. (28) and compute
$$
\widehat {q} _ {t} = \underset {q \in \Omega (\bar {P} _ {i})} {\operatorname {a r g m i n}} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} \left(\widehat {\ell} _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q). \tag {27}
$$
Compute policy  $\pi_t$  from  $\widehat{q}_t$  such that  $\pi_t(a|s) \propto \widehat{q}_t(s,a)$ .
Execute policy  $\pi_t$  and obtain trajectory  $(s_{t,k},a_{t,k})$  for  $k = 0,\ldots ,L - 1$ .
Construct loss estimator  $\widehat{\ell_t}$  as defined in Eq. (5).
Compute amortized bonus  $b_{t}$  based on Eq. (29).
Compute learning rate  $\gamma_{t + 1}$  according to Definition 5.2.
Increment counters: for each  $k < L$ ,  $m_{i}(s_{t,k}, a_{t,k}, s_{t,k+1}) \stackrel{+}{\leftarrow} 1$ ,  $m_{i}(s_{t,k}, a_{t,k}) \stackrel{+}{\leftarrow} 1$ .
if  $\exists k$ $m_{i}(s_{t,k},a_{t,k})\geq \max \{1,2m_{i - 1}(s_{t,k},a_{t,k})\}$  then
Increment epoch index  $i \stackrel{+}{\leftarrow} 1$  and set new epoch starting time  $t_i = t + 1$ .
Initialize new counters:  $\forall (s,a,s^{\prime}),m_{i}(s,a,s^{\prime}) = m_{i - 1}(s,a,s^{\prime}),m_{i}(s,a) = m_{i - 1}(s,a)$
Update empirical transition  $P_{i}$  and confidence width  $B_{i}$  based on Eq. (2).
Update optimistic transition  $\widetilde{P}_i$  based on Definition C.1.1
C Omitted Details for Section 5
In this section, we consider a variant of the algorithm proposed in Jin et al. [2021], which instead ensures exploration via optimistic transitions and also switch the regularizer from Tsallis entropy to log-barrier. We present the pseudocode in Algorithm 4, and show that it automatically adapts to easier environments with improved gap-dependent regret bounds.
Remark C.1. Throughout the analysis, we denote by  $Q^{P,\pi}(s,a;r)$  the state-action value of state-action pair  $(s,a)$  with respect to transition function  $P$  ( $P$  could be an optimistic transition function), policy  $\pi$  and loss function  $r$ ; similarly, we denote by  $V^{P,\pi}(s;r)$  the corresponding state-value function.
Specifically, the state value function  $V^{P,\pi}(s;r)$  is computed in a backward manner from layer  $L$  to layer 0 as following:
$$
V ^ {P, \pi} (s; r) = \left\{ \begin{array}{l l} 0, & s = s _ {L}, \\ \sum_ {a \in A} \pi (a | s) \cdot \Big (r (s, a) + \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} P (s ^ {\prime} | s, a) V ^ {P, \pi} (s ^ {\prime}; r) \Big), & s \neq s _ {L}. \end{array} \right.
$$
Similarly, the state-action value function  $Q^{P,\pi}(s,a;r)$  is calculated in the same manner:
$$
Q ^ {P, \pi} (s, a; r) = \left\{ \begin{array}{l l} 0, & s = s _ {L}, \\ r (s, a) + \sum_ {s ^ {\prime} \in S _ {k (s) + 1}} P (s ^ {\prime} | s, a) \sum_ {a ^ {\prime} \in A} \pi (a ^ {\prime} | s ^ {\prime}) Q ^ {P, \pi} (s ^ {\prime}, a ^ {\prime}; r), & s \neq s _ {L}. \end{array} \right.
$$
Clearly,  $V^{P,\pi}(s_0;r)$ , the state value of  $s_0$ , is equal to
$$
V ^ {P, \pi} (s _ {0}; r) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi} (s, a) r (s, a) \triangleq \left\langle q ^ {P, \pi}, r \right\rangle .
$$
This equality can be further extended to the other state  $u$  as:
$$
V ^ {P, \pi} (u; r) = \sum_ {s \neq s _ {L}} \sum_ {a \in A} q ^ {P, \pi} (s, a | u) r (s, a),
$$
where  $q^{P,\pi}(s,a|u)$  denotes the probability of arriving at  $(s,a)$  starting at state  $u$  under policy  $\pi$  and transition function  $P$ .
C.1 Description of the Algorithm
The construction of this algorithm follows similar ideas to the work of Jin et al. [2021], while also including several key differences. One of them is that we rely on the log-barrier regularizer, defined with a positive learning rate  $\gamma_t(s,a)$  as
$$
\phi_ {t} (q) = - \sum_ {s, a} \gamma_ {t} (s, a) \log (q (s, a)). \tag {28}
$$
The choice of log-barrier is important for adversarial transitions as discussed in Section 3, while the adaptive choice of learning rate is important for adapting to easy environments. The formal definition of the learning rate is given in Definition 5.2, and further details and properties of the learning rate are provided in Section C.7.
As the algorithm runs a new instance of FTRL for each epoch  $i$ , we modify the definition of the amortized bonus  $b_{t}$  accordingly,
$$
b _ {t} (s) = b _ {t} (s, a) = \left\{ \begin{array}{l l} \frac {4 L}{u _ {t} (s)} & \text {i f} \sum_ {\tau = t _ {i (t)}} ^ {t} \mathbb {I} \{\lceil \log_ {2} u _ {\tau} (s) \rceil = \lceil \log_ {2} u _ {t} (s) \rceil \} \leq \frac {C ^ {\mathrm {P}}}{2 L}, \\ 0 & \text {e l s e .} \end{array} \right. \tag {29}
$$
The bonus  $b_{t}(s)$  in Eq. (29) is defined based on each epoch  $i$ , in the sense that  $\sum_{\tau=1}^{t} \mathbb{I}\{\lceil \log_{2} u_{\tau}(s) \rceil = \lceil \log_{2} u_{t}(s) \rceil\}$  counts, among all previous rounds  $\tau = t_{i}, \ldots, t$  in epoch  $i$ , the number of times that the value of  $u_{\tau}(s)$  falls into the same bin as  $u_{t}(s)$ .
Again, for the ease of analysis, in the analysis we use the equivalent definition of the bonus  $b_{t}(s)$  defined in Eq. (14), except that the counter  $z_{t}^{j}(s)$  now will be reset to zero at the beginning of each epoch  $i$ .
Next, we formally define the optimistic transition as follows.
Definition C.1.1 (Optimistic Transition). For epoch  $i$ , the optimistic transition  $\widetilde{P}_i: S \times A \times S \to [0,1]$  is defined as:
$$
\widetilde {P} _ {i} (s ^ {\prime} | s, a) = \left\{ \begin{array}{l l} \max  \left\{0, \bar {P} _ {i} (s ^ {\prime} | s, a) - B _ {i} (s, a, s ^ {\prime}) \right\}, & (s, a, s ^ {\prime}) \in W _ {k}, \\ \sum_ {s ^ {\prime} \in S _ {k + 1}} \min  \left\{\bar {P} _ {i} (s ^ {\prime} | s, a), B _ {i} (s, a, s ^ {\prime}) \right\}, & (s, a) \in S _ {k} \times A a n d s ^ {\prime} = s _ {L}, \end{array} \right.
$$
where  $\bar{P}_i$  is the empirical transition defined in Eq. (2).
Note that the optimistic transition  $\tilde{P}_i(\cdot |s,a)$  is a valid distribution as we have
$$
\sum_ {s ^ {\prime} \in S _ {k + 1}} \min  \left\{\bar {P} _ {i} (s ^ {\prime} | s, a), B _ {i} (s, a, s ^ {\prime}) \right\} = 1 - \sum_ {s ^ {\prime} \in S _ {k + 1}} \max  \left\{0, \bar {P} _ {i} (s ^ {\prime} | s, a) - B _ {i} (s, a, s ^ {\prime}) \right\}.
$$
We summarize the properties of the optimistic transition functions in Appendix C.8.
C.2 Self-Bounding Properties of the Regret
In order to achieve best-of-both-worlds guarantees, our goal is to bound  $\mathrm{Reg}_T(\pi^\star)$  in terms of two self-bounding quantities for some  $x > 0$  (plus other minor terms):
$$
\mathbb {S} _ {1} (x) = \sqrt {x \cdot \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]}, \tag {30}
$$
$$
\mathbb {S} _ {2} (x) = \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \sqrt {x \cdot \mathbb {E} \left[ \sum_ {t = 1} ^ {T} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]}.
$$
These two quantities enjoy a certain self-bounding property which is critical to achieve the gap-dependent bound under Condition (9), as they can be related back to the regret against policy  $\pi^{\star}$  itself. To see this, we first show the following implication of Condition (9).
Lemma C.2.1. Under Condition (9), the following holds.
$$
\operatorname {R e g} _ {T} \left(\pi^ {\star}\right) \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t}, \pi_ {t}} (s, a) \Delta (s, a) \right] - C ^ {L} - 2 L C ^ {P} - L ^ {2} C ^ {P}. \tag {31}
$$
Proof. From the definition of  $\mathrm{Reg}_T(\pi^\star)$ , we first note that:
$$
\begin{array}{l} \operatorname {R e g} _ {T} (\pi^ {\star}) \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}} - q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P _ {t}, \pi_ {t}} - q ^ {P, \pi_ {t}}, \ell_ {t} \right\rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi^ {\star}} - q ^ {P _ {t}, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \\ \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}} - q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] - 2 L C ^ {\mathsf {P}} \\ \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {*} (s)} q ^ {P, \pi_ {t}} (s, a) \Delta (s, a) \right] - C ^ {\mathsf {L}} - 2 L C ^ {\mathsf {P}} \\ \end{array}
$$
where the third step applies Corollary D.3.7 and the last step uses the Condition (9). We continue to bound the first term above as:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P, \pi_ {t}} (s, a) \Delta (s, a) \right] \\ \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t}, \pi_ {t}} (s, a) \Delta (s, a) \right] \\ - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \left| q ^ {P _ {t}, \pi_ {t}} (s, a) - q ^ {P, \pi_ {t}} (s, a) \right| \Delta (s, a) \right] \\ \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t}, \pi_ {t}} (s, a) \Delta (s, a) \right] - L ^ {2} C ^ {\mathsf {P}}, \\ \end{array}
$$
where the last step uses  $\Delta (s,a)\in (0,L]$  and Corollary D.3.6.
We are now ready to show the following important self-bounding properties (recall  $\Delta_{\mathrm{MIN}} = \min_{s \neq s_L, a \neq \pi^*(s)} \Delta(s, a)$ ).
Lemma C.2.2 (Self-Bounding Quantities). Under Condition (31), we have for any  $z > 0$ :
$$
\mathbb {S} _ {1} (x) \leq z \left(\operatorname {R e g} _ {T} (\pi^ {\star}) + C ^ {L} + 2 L C ^ {P} + L ^ {2} C ^ {P}\right) + \frac {1}{z} \left(\frac {x}{4 \Delta_ {\mathrm {M I N}}}\right),
$$
$$
\mathbb {S} _ {2} (x) \leq z \left(\operatorname {R e g} _ {T} (\pi^ {\star}) + C ^ {L} + 2 L C ^ {P} + L ^ {2} C ^ {P}\right) + \frac {1}{z} \left(\sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \frac {x}{4 \Delta (s , a)}\right).
$$
Besides, it always holds that  $\mathbb{S}_1(x) \leq \sqrt{x \cdot LT}$  and  $\mathbb{S}_2(x) \leq \sqrt{x \cdot L|S||A|T}$ .
$$
\mathbb {S} _ {1} (x) = \sqrt {\frac {x}{2 z \Delta_ {\mathrm {M I N}}} \cdot \mathbb {E} \left[ \left(2 z \Delta_ {\mathrm {M I N}} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t} , \pi_ {t}} (s , a)\right) \right]}
$$
$$
\begin{array}{l} \leq \mathbb {E} \left[ z \Delta_ {\mathrm {M I N}} \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {*} (s)} q ^ {P _ {t}, \pi_ {t}} (s, a) \right] + \frac {x}{4 z \Delta_ {\mathrm {M I N}}} \\ \leq z \left(\operatorname {R e g} _ {T} \left(\pi^ {\star}\right) + C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}}\right) + \frac {x}{4 z \Delta_ {\mathrm {M I N}}}, \\ \end{array}
$$
where the second step follows from the AM-GM inequality:  $2\sqrt{xy} \leq x + y$  for any  $x, y \geq 0$ , and the last step follows from Condition (31).
$$
\begin{array}{l} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \sqrt {\frac {x}{2 z \Delta (s , a)} \cdot \mathbb {E} \left[ 2 z \Delta (s , a) \sum_ {t = 1} ^ {T} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]} \\ \leq \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \frac {x}{4 z \Delta (s , a)} + z \mathbb {E} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \sum_ {t = 1} ^ {T} q ^ {P _ {t}, \pi_ {t}} (s, a) \Delta (s, a) \right] \\ = z \left(\operatorname {R e g} _ {T} \left(\pi^ {\star}\right) + C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}}\right) + \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \frac {x}{4 z \Delta (s , a)}. \\ \end{array}
$$
Finally, by direct calculation, we can show that
$$
\mathbb {S} _ {1} (x) = \sqrt {x \cdot \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]} \leq \sqrt {x \cdot L T},
$$
according to the fact that  $\sum_{s\neq s_L}\sum_{a\in A}q^{P_t,\pi_t}(s,a)\leq L$ . On the other hand,  $\mathbb{S}_2(x)$  is bounded as
$$
\mathbb {S} _ {2} (x) \leq \sqrt {x \cdot | S | | A | \mathbb {E} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {*} (s)} \sum_ {t = 1} ^ {T} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]} \leq \sqrt {x \cdot L | S | | A | T},
$$
with the help of the Cauchy-Schwarz inequality in the first step.
Finally, we show that Algorithm 4 achieves the following adaptive regret bound, which directly leads to the best-of-both-worlds guarantee in Theorem 5.1.
Lemma C.2.3. Algorithm 4 with  $\delta = 1 / T^2$  and learning rate defined in Definition 5.2 ensures that, for any mapping  $\pi^{\star}:S\to A$ , the regret  $\mathrm{Reg}_T(\pi^\star)$  is bounded by  $\mathcal{O}\left((C^P +1)L^2 |S|^4 |A|^2\log^2 (\iota)\right)$  plus
$$
\left. \mathcal {O} \left(\mathbb {S} _ {1} \left(L ^ {3} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \mathbb {S} _ {2} \left(L ^ {2} | S | | A | \log^ {2} (\iota)\right)\right). \right.
$$
The proof of this result is detailed in Section C.3. We emphasize that this bound holds for any mapping  $\pi^{\star}$ , and is not limited to that policy in Eq. (9). This is important for proving the robustness result when losses are arbitrary, as shown in the following proof of Theorem 5.1.
Proof of Theorem 5.1. When losses are arbitrary, we simply select  $\pi^{\star} = \mathring{\pi}$  where  $\mathring{\pi}$  is one of the optimal deterministic policies in hindsight and obtain the following bound of  $\operatorname{Reg}_T(\mathring{\pi})$ :
$$
\begin{array}{l} \mathcal {O} \left(\left(C ^ {\mathsf {P}} + 1\right) L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota) + \mathbb {S} _ {1} \left(L ^ {3} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \mathbb {S} _ {2} \left(L ^ {2} | S | | A | \log^ {2} (\iota)\right)\right) \\ = \mathcal {O} \left(\left(C ^ {\mathsf {P}} + 1\right) L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota) + \sqrt {L ^ {4} | S | ^ {2} | A | T \log^ {2} (\iota)} + \sqrt {L ^ {3} | S | ^ {2} | A | ^ {2} T \log^ {2} (\iota)}\right), \\ \end{array}
$$
where the first step follows from Lemma C.2.3 and the second step follows from Lemma C.2.2.
Next, suppose that Condition (9) holds. We set  $\pi^{\star}$  as defined in the condition and use Lemma C.2.3 to write the regret against  $\pi^{\star}$  as:
$$
\operatorname {R e g} _ {T} \left(\pi^ {\star}\right) \leq \mu \left(\mathbb {S} _ {1} \left(L ^ {3} | S | ^ {2} | A | \log^ {2} (\iota)\right) + \mathbb {S} _ {2} \left(L ^ {2} | S | | A | \log^ {2} (\iota)\right)\right) + \xi \left(C ^ {\mathsf {P}} + 1\right),
$$
where  $\mu > 0$  is an absolute constant and  $\xi = \mathcal{O}\left(L^{2}|S|^{4}|A|^{2}\log^{2}(\iota)\right)$ .
For any  $z > 0$ , according to Lemma C.2.2 (where we set the  $z$  there as  $z / 2\mu$ ), we have:
$$
\begin{array}{l} \operatorname {R e g} _ {T} \left(\pi^ {\star}\right) \leq z \left(\operatorname {R e g} _ {T} \left(\pi^ {\star}\right) + C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}}\right) \\ + \frac {\mu^ {2}}{z} \left(\frac {L ^ {3} | S | ^ {2} | A | \log^ {2} (\iota)}{\Delta_ {\mathrm {M I N}}} + \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \frac {L ^ {2} | S | | A | \log^ {2} (\iota)}{\Delta (s , a)}\right) + \xi \left(C ^ {\mathsf {P}} + 1\right). \\ \end{array}
$$
Let  $x = \frac{1 - z}{z}$  and  $U = \frac{L^3|S|^2|A|\log^2(\iota)}{\Delta_{\mathrm{MIN}}} +\sum_{s\neq s_L}\sum_{a\neq \pi^* (s)}\frac{L^2|S||A|\log^2(\iota)}{\Delta(s,a)}$  Rearranging the above inequality leads to
$$
\begin{array}{l} \operatorname {R e g} _ {T} (\pi^ {\star}) \leq \frac {z \left(C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}}\right)}{1 - z} + \frac {\mu^ {2} U}{z (1 - z)} + \frac {\xi \left(C ^ {\mathsf {P}} + 1\right)}{1 - z} \\ = \frac {\left(C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}}\right)}{x} + \left(x + 2 + \frac {1}{x}\right) \mu^ {2} U + \left(1 + \frac {1}{x}\right) \xi \left(C ^ {\mathsf {P}} + 1\right) \\ = \frac {1}{x} \left(C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}} + \mu^ {2} U + \xi \left(C ^ {\mathsf {P}} + 1\right)\right) + x \cdot \mu^ {2} U + 2 \mu^ {2} U + \xi \left(C ^ {\mathsf {P}} + 1\right). \\ \end{array}
$$
Picking the optimal  $x$  to minimize the upper bound of  $\operatorname{Reg}_T(\pi^\star)$  yields
$$
\begin{array}{l} \operatorname {R e g} _ {T} (\pi^ {\star}) = 2 \sqrt {\left(C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}} + \mu^ {2} U + \xi \left(C ^ {\mathsf {P}} + 1\right)\right) \mu^ {2} U} + 2 \mu^ {2} U + \xi \left(C ^ {\mathsf {P}} + 1\right) \\ \leq 2 \sqrt {\left(C ^ {\mathsf {L}} + 2 L C ^ {\mathsf {P}} + L ^ {2} C ^ {\mathsf {P}} + \xi \left(C ^ {\mathsf {P}} + 1\right)\right) \mu^ {2} U} + 4 \mu^ {2} U + \xi \left(C ^ {\mathsf {P}} + 1\right) \\ = \mathcal {O} \left(\sqrt {\left(C ^ {\mathsf {L}} + L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota) C ^ {\mathsf {P}}\right) U} + U + L ^ {2} | S | ^ {4} | A | \log^ {2} (\iota) \left(C ^ {\mathsf {P}} + 1\right)\right) \\ \leq \mathcal {O} \left(\sqrt {U C ^ {\mathsf {L}}} + U + L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota) \left(C ^ {\mathsf {P}} + 1\right)\right), \\ \end{array}
$$