SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which we also write as  $b_{t}(s)$  since it is independent of  $a$ . To understand this definition, note that  $-\lceil \log_2 u_t(s) \rceil$  is exactly the unique integer  $j$  such that  $u_{t}(s)$  falls into the bin  $(2^{-j-1}, 2^{-j}]$ . Therefore, the expression  $\sum_{\tau=1}^{t} \mathbb{I}\{\lceil \log_2 u_{\tau}(s) \rceil = \lceil \log_2 u_{t}(s) \rceil\}$  counts, among all previous rounds  $\tau = 1, \ldots, t$ , how many times we have encountered a  $u_{\tau}(s)$  value that falls into the same bin as  $u_{t}(s)$ . If this number does not exceed  $\frac{C_t^P}{2L}$ , we apply a bonus of  $\frac{4L}{u_t(s)}$ , which is (two times of) the maximum possible value of the unknown quantity  $\frac{C_t^P}{u_t(s)}$ ; otherwise, we do not apply any bonus. The idea is that by enlarging the bonus to its maximum value and stopping it after enough times, even though each  $b_{t}(s)$  might be quite different from  $\frac{C_t^P}{u_t(s)}$ , overall they behave similarly after  $T$  episodes:
<sup>4</sup>The original loss estimator of Jin et al. [2020] is slightly different, but that difference is only for the purpose of obtaining a high probability regret guarantee, which we do not consider in this work for simplicity.
Algorithm 1 Algorithm for Adversarial Transitions (with Known  $C^{\mathsf{P}}$ )
for  $t = 1,\dots ,T$  do
Input: confidence parameter  $\delta \in (0,1)$ , learning rate  $\eta > 0$ .  
Initialize: epoch index  $i = 1$ ; counters  $m_{1}(s,a) = m_{1}(s,a,s') = m_{0}(s,a) = m_{0}(s,a,s') = 0$  for all  $(s,a,s')$ ; empirical transition  $\bar{P}_{1}$  and confidence width  $B_{1}$  based on Eq. (2) and Eq. (4); occupancy measure  $\widehat{q}_{1}(s,a,s') = \frac{1}{|S_{k}||A||S_{k + 1}|}$  for all  $(s,a,s')$ ; and initial policy  $\pi_{1} = \pi^{\widehat{q}_{1}}$ .
Execute policy  $\pi_t$  and obtain trajectory  $(s_{t,k},a_{t,k})$  for  $k = 0,\dots ,L - 1$ .  
Construct loss estimator  $\widehat{\ell}_t$  as defined in Eq. (5).  
Update  $b_{t}(s)$  for all  $s$  based on Eq. (7).  
Increase counters: for each  $k < L$ ,  $m_i(s_{t,k},a_{t,k},s_{t,k + 1})\stackrel {+}{\leftarrow}1$ ,  $m_i(s_{t,k},a_{t,k})\stackrel {+}{\leftarrow}1$ .  
if  $\exists k$ ,  $m_i(s_{t,k},a_{t,k})\geq \max \{1,2m_{i - 1}(s_{t,k},a_{t,k})\}$  then ▷ entering a new epoch  
Increase epoch index  $i\stackrel {\pm}{\leftarrow}1$ .  
Initialize new counters:  $\forall (s,a,s^{\prime}),m_i(s,a,s^{\prime}) = m_{i - 1}(s,a,s^{\prime}),m_i(s,a) = m_{i - 1}(s,a)$ .  
Update confidence set  $\mathcal{P}_i$  based on Eq. (3).  
Let  $D_{\phi}(\cdot ,\cdot)$  be the Bregman divergence with respect to log barrier (Eq. (11)) and compute  
\[
\widehat{q}_{t + 1} = \operatorname *{argmin}_{q\in \Omega (\mathcal{P}_i)}\eta \Bigl {\langle}q,\widehat{\ell}_t - b_t\bigr {\rangle} + D_\phi (q,\widehat{q}_t).
\]  
Update policy  $\pi_{t + 1} = \pi^{\widehat{q}_{t + 1}}$ .
Lemma 3.1. The amortized bonus defined in Eq. (7) satisfies  $\sum_{t=1}^{T} \frac{C_t^P}{u_t(s)} \leq \sum_{t=1}^{T} b_t(s)$  and  $\sum_{t=1}^{T} \widehat{q}_t(s) b_t(s) = \mathcal{O}(C^P \log T)$  for any  $s$ .
Therefore, the problematic term Eq. (6) is at most  $\mathbb{E}[\sum_t\langle q^{P,\hat{\pi}},b_t\rangle ]$  , which, if "converted" to  $\mathbb{E}[\sum_t\langle \widehat{q}_t,b_t\rangle ]$  (change of measure), is nicely bounded by  $\mathcal{O}(|S|C^P\log T)$  . As mentioned, such a change of measure can be realized by feeding  $\widehat{\ell}_t - b_t$  instead of  $\widehat{\ell}_t$  to OMD, because now standard analysis of OMD ensures a bound on  $\mathrm{REG} = \mathbb{E}[\sum_t\langle \widehat{q}_t - q^{P,\hat{\pi}},\widehat{\ell}_t - b_t\rangle ]$  , which, compared to the earlier definition of REG, leads to a difference of  $\mathbb{E}[\sum_t\langle \widehat{q}_t - q^{P,\hat{\pi}},b_t\rangle ]$  (see Appendix A.5 for details).
Second Issue The second issue comes from analyzing REG (which exists even if no bonuses are used). Specifically, standard analysis of OMD requires bounding a "stability" term, which, for the negative entropy regularizer, is in the form of  $\mathbb{E}[\sum_t\sum_{s,a}\widehat{q}_t(s,a)\widehat{\ell}_t(s,a)^2] = \mathbb{E}[\sum_t\sum_{s,a}\widehat{q}_t(s,a)\frac{q^{P_t,\pi_t}(s,a)\ell_t(s,a)^2}{u_t(s,a)^2}]\leq \mathbb{E}[\sum_t\sum_{s,a}\frac{q^{P_t,\pi_t}(s,a)}{u_t(s,a)}]$ . Once again, when  $C^{\mathsf{P}} = 0$  and  $P_{t} = P$ , we have  $q^{P_t,\pi_t}$  bounded by  $u_{t}(s,a)$  with high probability, and thus the stability term is  $\mathcal{O}(T|S||A|)$ ; but this breaks if  $C^{\mathsf{P}}\neq 0$  and  $P_{t}$  can be arbitrarily different from transitions in  $\mathcal{P}_{i(t)}$ .
Solution: Log-Barrier Regularizer Resolving this second issue, however, is relatively straightforward — it suffices to switch the regularizer from negative entropy to log-barrier:  $\phi(q) = -\sum_{k=0}^{L-1}\sum_{(s,a,s') \in W_k}\log q(s,a,s')$ , which is first used by Lee et al. [2020] in the context of learning adversarial MDPs but dates back to earlier work such as Foster et al. [2016] for multi-armed bandits. An important property of log-barrier is that it leads to a smaller stability term in the form of  $\mathbb{E}[\sum_t\sum_{s,a}\widehat{q}_t(s,a)^2\widehat{\ell}_t(s,a)^2]$  (with an extra  $\widehat{q}_t(s,a)$ ), which is at most  $\mathbb{E}[\sum_t\sum_{s,a}q^{P_t,\pi_t}(s,a)\ell_t(s,a)^2] = \mathcal{O}(TL)$  since  $\widehat{q}_t(s,a) \leq u_t(s,a)$ . In fact, this also helps control the extra stability term when bonuses are used, which is in the form of  $\mathbb{E}[\sum_t\sum_{s,a}\widehat{q}_t(s,a)^2b_t(s,a)^2]$  and is at most  $4L\mathbb{E}[\sum_t\langle\widehat{q}_t,b_t\rangle]=\mathcal{O}(L|S|C^{\mathsf{P}}\log T)$  according to Lemma 3.1.
Putting these two ideas together leads to our final algorithm (see Algorithm 1). We prove the following regret bound in Appendix A, which recovers that of Jin et al. [2020] when  $C^{\mathsf{P}} = 0$  and increases linearly in  $C^{\mathsf{P}}$  as desired.
Theorem 3.2. With  $\delta = 1 / T$  and  $\eta = \min \left\{\sqrt{\frac{|S|^2|A|\log(\iota)}{LT}},\frac{1}{8L}\right\}$ , Algorithm 1 ensures
$$
\operatorname {R e g} _ {T} = \mathcal {O} \left(L | S | \sqrt {| A | T \log (\iota)} + L | S | ^ {4} | A | \log^ {2} (\iota) + C ^ {P} L | S | ^ {4} | A | \log (\iota)\right).
$$
4 Achieving  $\mathcal{O}(\sqrt{T} + C^{\mathbf{P}})$  with Unknown  $C^{\mathbf{P}}$
In this section, we address the case when the amount of corruption is unknown. We develop a black-box reduction which turns an algorithm that only deals with known  $C^{\mathsf{P}}$  to one that handles unknown  $C^{\mathsf{P}}$ . This is similar to Wei et al. [2022] but additionally handles adversarial losses using a different approach. A byproduct of our reduction is that we develop an entirely black-box model selection approach for adversarial online learning problems, as opposed to the gray-box approach developed by the "Corral" literature [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022] which requires checking if the base algorithm is stable. To achieve this, we essentially develop another layer of reduction that turns any standard algorithm with sublinear regret into a stable algorithm. This result itself might be of independent interest and useful for solving other model selection problems.
More specifically, our reduction has two layers. The bottom layer is where our novelty lies: it takes as input an arbitrary corruption-robust algorithm that operates under known  $C^{\mathsf{P}}$  (e.g., the one we developed in Section 3), and outputs a stable corruption-robust algorithm (formally defined later) that still operates under known  $C^{\mathsf{P}}$ . The top layer, on the other hand, follows the standard Corral idea and takes as input a stable algorithm that operates under known  $C^{\mathsf{P}}$ , and outputs an algorithm that operates under unknown  $C^{\mathsf{P}}$ . Below, we explain these two layers of reduction in details.
Bottom Layer (from an Arbitrary Algorithm to a Stable Algorithm) The input of the bottom layer is an arbitrary corruption-robust algorithm, formally defined as:
Definition 4.1. An adversarial MDP algorithm is corruption-robust if it takes  $\theta$  (a guess on the corruption amount) as input, and achieves the following regret for any random stopping time  $t' \leq T$ :
$$
\max _ {\pi} \mathbb {E} \left[ \sum_ {t = 1} ^ {t ^ {\prime}} (\ell_ {t} (\pi_ {t}) - \ell_ {t} (\pi)) \right] \leq \mathbb {E} \left[ \sqrt {\beta_ {1} t ^ {\prime}} + (\beta_ {2} + \beta_ {3} \theta) \mathbb {I} \{t ^ {\prime} \geq 1 \} \right] + \operatorname * {P r} [ C _ {1: t ^ {\prime}} ^ {\boldsymbol {\mathcal {P}}} > \theta ] L T
$$
for problem-dependent constants and  $\log (T)$  factors  $\beta_{1}\geq L^{2},\beta_{2}\geq L,\beta_{3}\geq 1$  , where  $C_{1:t^{\prime}}^{P} =$ $\sum_{\tau = 1}^{t^{\prime}}C_{\tau}^{P}$  is the total corruption up to time  $t^\prime$
While the regret bound in Definition 4.1 might look cumbersome, it is in fact fairly reasonable: if the guess  $\theta$  is not smaller than the true corruption amount, the regret should be of order  $\sqrt{t'} + \theta$ ; otherwise, the regret bound is vacuous since  $LT$  is its largest possible value. The only extra requirement is that the algorithm needs to be anytime (i.e., the regret bound holds for any stopping time  $t'$ ), but even this is known to be easily achievable by using a doubling trick over a fixed-time algorithm. It is then clear that our algorithm in Section 3 (together with a doubling trick) indeed satisfies Definition 4.1.
In such a learning protocol, Agarwal et al. [2017] defines a stable algorithm as one whose regret smoothly degrades with  $\rho_{T} = \frac{1}{\min_{t\in[T]}w_{t}}$ . For our purpose here, we additionally require that the dependence on  $C^{\mathsf{P}}$  in the regret bound is linear, which results in the following definition:
Definition 4.2 ( $\frac{1}{2}$ -stable corruption-robust algorithm). A  $\frac{1}{2}$ -stable corruption-robust algorithm is one that, with prior knowledge on  $C^P$ , achieves  $\mathrm{Reg}_T \leq \mathbb{E}\left[\sqrt{\beta_1\rho_T T} + \beta_2\rho_T\right] + \beta_3 C^P$  under Protocol 1 for problem-dependent constants and  $\log(T)$  factors  $\beta_1 \geq L^2, \beta_2 \geq L$ , and  $\beta_3 \geq 1$ .
For simplicity, we only define and discuss the  $\frac{1}{2}$ -stability notion here (the parameter  $\frac{1}{2}$  refers to the exponent of  $T$ ), but our result can be straightforwardly extended to the general  $\alpha$ -stability notion for
Algorithm 2 STable Algorithm By Independent Learners and Instance SElection (STABILISE)
Input:  $C^{\mathsf{P}}$  and a base algorithm satisfying Definition 4.1. Initialize:  $\lceil \log_2 T \rceil$  instances of the base algorithm  $\mathsf{ALG}_1, \ldots, \mathsf{ALG}_{\lceil \log_2 T \rceil}$ , where  $\mathsf{ALG}_j$  is configured with the parameter
$$
\theta = \theta_ {j} \triangleq 2 ^ {- j + 1} C ^ {\mathsf {P}} + 1 6 L \log (T).
$$
for  $t = 1,2,\ldots$  do  
Receive  $w_{t}$   
if  $w_{t} \leq \frac{1}{T}$  then  
 $\begin{array}{l}\text{play an arbitrary policy } \pi_t \\ \text{continue (without updating any instances)} \end{array}$   
Let  $j_{t}$  be such that  $w_{t} \in (2^{-j_{t} - 1}, 2^{-j_{t}}]$ .  
Let  $\pi_t$  be the policy suggested by  $\mathsf{ALG}_{j_t}$ .  
Output  $\pi_t$ .  
If feedback is received, send it to  $\mathsf{ALG}_{j_t}$  with probability  $\frac{2^{-j_t - 1}}{w_t}$ , and discard it otherwise.
$\alpha \in [\frac{1}{2}, 1)$  as in Agarwal et al. [2017]. Our main result in this section is then that one can convert any corruption-robust algorithm into a  $\frac{1}{2}$ -stable corruption-robust algorithm:
Theorem 4.3. If an algorithm is corruption robust according to Definition 4.1 for some constants  $(\beta_{1},\beta_{2},\beta_{3})$ , then one can convert it to a  $\frac{1}{2}$ -stable corruption-robust algorithm (Definition 4.2) with constants  $(\beta_1',\beta_2',\beta_3')$  where  $\beta_1' = \mathcal{O}(\beta_1\log T)$ ,  $\beta_2' = \mathcal{O}(\beta_2 + \beta_3L\log T)$ , and  $\beta_3' = \mathcal{O}(\beta_3\log T)$ .
This conversion is achieved by a procedure that we call STABILISE (see Algorithm 2 for details). The high-level idea of STABILISE is as follows. Noticing that the challenge when learning in Protocol 1 is that  $w_{t}$  varies over time, we discretize the value of  $w_{t}$  and instantiate one instance of the input algorithm to deal with one possible discretized value, so that it is learning in Protocol 1 but with a fixed  $w_{t}$ , making it straightforward to bound its regret based on what it promises in Definition 4.1.
More concretely, STABILISE instantiates  $\mathcal{O}(\log_2T)$  instances  $\{\mathsf{ALG}_j\}_{j = 0}^{\lceil \log_2T\rceil}$  of the input algorithm that satisfies Definition 4.1, each with a different parameter  $\theta_{j}$ . Upon receiving  $w_{t}$  from the environment, it dispatches round  $t$  to the  $j$ -th instance where  $j$  is such that  $w_{t}\in (2^{-j - 1},2^{-j}]$ , and uses the policy generated by  $\mathsf{ALG}_j$  to interact with the environment (if  $w_{t}\leq \frac{1}{T}$ , simply ignore this round). Based on Protocol 1, the feedback for this round is received with probability  $w_{t}$ . To equalize the probability of  $\mathsf{ALG}_j$  receiving feedback as mentioned in the high-level idea, when the feedback is actually obtained, STABILISE sends it to  $\mathsf{ALG}_j$  only with probability  $\frac{2^{-j - 1}}{w_t}$  (and discards it otherwise). This way, every time  $\mathsf{ALG}_j$  is assigned to a round, it always receives the desired feedback with probability  $w_{t}\cdot \frac{2^{-j - 1}}{w_{t}} = 2^{-j - 1}$ . This equalization step is the key that allows us to use the original guarantee of the base algorithm (Definition 4.1) and run it as it is, without requiring it to perform extra importance weighting steps as in Agarwal et al. [2017].
The choice of  $\theta_{j}$  is crucial in making sure that STABILISE only has  $C^{\mathsf{P}}$  regret overhead instead of  $\rho_T C^{\mathsf{P}}$ . Since  $\mathsf{ALG}_j$  only receives feedback with probability  $2^{-j - 1}$ , the expected total corruption it experiences is on the order of  $2^{-j - 1} C^{\mathsf{P}}$ . Therefore, its input parameter  $\theta_{j}$  only needs to be of this order instead of the total corruption  $C^{\mathsf{P}}$ . This is similar to the key idea of Wei et al. [2022] and Lykouris et al. [2018]. See Appendix B.1 for more details and the full proof of Theorem 4.3.
Top Layer (from Known  $C^{\mathbf{P}}$  to Unknown  $C^{\mathbf{P}}$ ) With a stable algorithm and a regret guarantee in Definition 4.2, it is relatively standard to convert it to an algorithm with  $\widetilde{\mathcal{O}}(\sqrt{T} + C^{\mathbf{P}})$  regret without knowing  $C^{\mathbf{P}}$ . Similar arguments have been made in Foster et al. [2020], and the idea is to have another specially designed OMD/FTRL-based master algorithm to choose on the fly among a set of instances of this stable base algorithm, each with a different guess on  $C^{\mathbf{P}}$  (the probability  $w_{t}$  in Protocol 1 is then decided by this master algorithm). We defer all details to Appendix B. The final regret guarantee is the following  $(\widetilde{\mathcal{O}}(\cdot)$  hides  $\log(T)$  factors).
Theorem 4.4. Using an algorithm satisfying Definition 4.2 as a base algorithm, Algorithm 3 (in the appendix) ensures  $\mathrm{Reg}_T = \tilde{\mathcal{O}}\left(\sqrt{\beta_1T} + \beta_2 + \beta_3C^P\right)$  without knowing  $C^P$ .
5 Gap-Dependent Refinements with Known  $C^{\mathbf{P}}$
Finally, we discuss how to further improve our algorithm so that it adapts to easier environments and enjoys a better bound when the loss functions satisfy a certain gap condition, while still maintaining the  $\mathcal{O}(\sqrt{T} + C^{\mathsf{P}})$  robustness guarantee. This result unfortunately requires the knowledge of  $C^{\mathsf{P}}$  because the black-box approach introduced in the last section leads to  $\sqrt{T}$  regret overhead already. We leave the possibility of removing this limitation for future work.
More concretely, following prior work such as Jin and Luo [2020], we consider the following general condition: there exists a mapping  $\pi^{\star}:S\to A$ , a gap function  $\Delta :S\times A\to (0,L]$ , and a constant  $C^\mathbb{L}\geq 0$ , such that for any policies  $\pi_1,\ldots ,\pi_T$  generated by the learner, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \pi_ {t}} - q ^ {P, \pi^ {\star}}, \ell_ {t} \right\rangle \right] \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} q ^ {P, \pi_ {t}} (s, a) \Delta (s, a) \right] - C ^ {\mathrm {L}}. \tag {9}
$$
It has been shown that this condition subsumes the case when the loss functions are drawn from a fixed distribution (in which case  $\pi^{\star}$  is simply the optimal policy with respect to the loss mean and  $P$ ,  $\Delta$  is the gap function with respect to the optimal  $Q$ -function, and  $C^{\mathsf{L}} = 0$ ), or further corrupted by an adversary in an arbitrary manner subject to a budget of  $C^{\mathsf{L}}$ ; we refer the readers to Jin and Luo [2020] for detailed explanation. Our main result for this section is a novel algorithm (whose pseudocode is deferred to Appendix C due to space limit) that achieves the following best-of-both-world guarantee.
Theorem 5.1. Algorithm 4 (with  $\delta = 1 / T^2$  and  $\gamma_t$  defined as in Definition 5.2) ensures
$$
\operatorname {R e g} _ {T} (\stackrel {\circ} {\pi}) = \mathcal {O} \left(L ^ {2} | S | | A | \log (\iota) \sqrt {T} + \left(C ^ {P} + 1\right) L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota)\right)
$$
always, and simultaneously the following gap-dependent bound under Condition (9):
$$
\operatorname {R e g} _ {T} (\pi^ {\star}) = \mathcal {O} \left(U + \sqrt {U C ^ {L}} + \left(C ^ {P} + 1\right) L ^ {2} | S | ^ {4} | A | ^ {2} \log^ {2} (\iota)\right),
$$
where  $U = \frac{L^3|S|^2|A|\log^2(\iota)}{\Delta_{\mathrm{MIN}}} +\sum_{s\neq s_L}\sum_{a\neq \pi^* (s)}\frac{L^2|S||A|\log^2(\iota)}{\Delta(s,a)}$  and  $\Delta_{\mathrm{MIN}} = \min_{s\neq s_L,a\neq \pi^* (s)}\Delta (s,a)$
Aside from having larger dependence on parameters  $L$ ,  $S$ , and  $A$ , Algorithm 4 maintains the same  $\mathcal{O}(\sqrt{T} + C^{\mathsf{P}})$  regret as before, no matter how losses/transitions are generated; additionally, the  $\sqrt{T}$  part can be significantly improved to  $\mathcal{O}(U + \sqrt{UC^{\mathsf{L}}})$  (which can be of order only  $\log^2 T$  when  $C^{\mathsf{L}}$  is small) under Condition (9). This result not only generalizes that of Jin et al. [2021], Dann et al. [2023] from  $C^{\mathsf{P}} = 0$  to any  $C^{\mathsf{P}}$ , but in fact also improves their results by having smaller dependence on  $L$  in the definition of  $U$ . In the rest of this section, we describe the main ideas of our algorithm.
FTRL with Epoch Schedule Our algorithm follows a line of research originated from Wei and Luo [2018], Zimmert and Seldin [2019] for multi-armed bandits and uses FTRL (instead of OMD) together with a certain self-bounding analysis technique. Since FTRL does not deal with varying decision sets easily, similar to Jin et al. [2021], we restart FTRL from scratch at the beginning of each epoch  $i$  (recall the epoch schedule described in Section 2). More specifically, in an episode  $t$  that belongs to epoch  $i$ , we now compute  $\widehat{q}_t$  as  $\mathrm{argmin}_q\left\langle q,\sum_{\tau = t_i}^{t - 1}\left(\widehat{\ell}_{\tau} - b_{\tau}\right)\right\rangle +\phi_t(q)$ , where  $t_i$  is the first episode of epoch  $i$ ,  $\widehat{\ell}_t$  is the same loss estimator defined in Eq. (5),  $b_{t}$  is the amortized bonus defined in Eq. (7) (except that  $\tau = 1$  there is also changed to  $\tau = t_i$  due to restarting),  $\phi_t$  is a time-varying regularizer to be specified later, and the set that  $q$  is optimized over is also a key element to be discussed next. As before, the learner then simply executes  $\pi_t = \pi^{\widehat{q}_t}$  for this episode.
Optimistic Transition An important idea from Jin et al. [2021] is that if FTRL optimizes  $q$  over  $\Omega(\bar{P}_i)$  (occupancy measures with respect to a fixed transition  $\bar{P}_i$ ) instead of  $\Omega(\mathcal{P}_i)$  (occupancy measures with respect to a set of plausible transitions) as in UOB-REPS, then a critical loss-shifting technique can be applied in the analysis. However, the algorithm lacks "optimism" when not using a confidence set, which motivates Jin et al. [2021] to instead incorporate optimism by subtracting
a bonus term BONUS from the loss estimator (not to be confused with the amortized bonus  $b_{t}$  we propose in this work). Indeed, if we define the value function  $V^{\bar{P},\pi}(s;\ell)$  as the expected loss one suffers when starting from  $s$  and following  $\pi$  in an MDP with transition  $\bar{P}$  and loss  $\ell$ , then they show that the BONUS term is such that  $V^{\bar{P}_i,\pi}(s;\ell -\mathrm{BONUS})\leq V^{P,\pi}(s;\ell)$  for any state  $s$  and any loss function  $\ell$ , that is, the performance of any policy is never underestimated.
Instead of following the same idea, here, we propose a simpler and better way to incorporate optimism via what we call optimistic transitions. Specifically, for each epoch  $i$ , we simply define an optimistic transition function  $\widetilde{P}_i$  such that  $\widetilde{P}_i(s'|s,a) = \max \left\{0,\widetilde{P}_i(s'|s,a) - B_i(s,a,s')\right\}$  (recall the confidence interval  $B_i$  defined in Eq. (4)). Since this makes  $\sum_{s'}\widetilde{P}_i(s'|s,a)$  less than 1, we allocate all the remaining probability to the terminal state  $s_L$  (which breaks the layer structure but does not really affect anything). This is a form of optimism because reaching the terminate state earlier can only lead to smaller loss. More formally, under the high probability event  $\mathcal{E}_{\mathrm{CON}}$ , we prove  $V^{\widetilde{P}_i,\pi}(s;\ell)\leq V^{P,\pi}(s;\ell)$  for any policy  $\pi$ , any state  $s$ , and any loss function  $\ell$  (see Lemma C.8.3).
With such an optimistic transition, we simply perform FTRL over  $\Omega (\widetilde{P}_i)$  without adding any additional bonus term (other than  $b_{t}$ ), making both the algorithm and the analysis much simpler than Jin et al. [2021]. Moreover, it can also be shown that  $V^{\widetilde{P}_i,\pi}(s;\ell -\mathrm{BONUS})\leq V^{\widetilde{P}_i,\pi}(s;\ell)$  (see Lemma C.8.4), meaning that while both loss estimation schemes are optimistic, ours is tighter than that of Jin et al. [2021]. This eventually leads to the aforementioned improvement in the  $U$  definition.
Time-Varying Log-Barrier Regularizers The final element to be specified in our algorithm is the time-varying regularizer  $\phi_t$ . Recall from discussions in Section 3 that using log-barrier as the regularizer is critical for bounding some stability terms in the presence of adversarial transitions. We thus consider the following log-barrier regularizer with an adaptive learning rate  $\gamma_t: S \times A \to \mathbb{R}_+$ :  $\phi_t(q) = -\sum_{s \neq s_L} \sum_{a \in A} \gamma_t(s, a) \cdot \log q(s, a)$ . The learning rate design requires combining the loss-shifting idea of [Jin et al., 2021] and the idea from [Ito, 2021], the latter of which is the first work to show that with adaptive learning rate tuning, the log-barrier regularizer leads to near-optimal best-of-both-world guarantee for multi-armed bandits.
More specifically, following the same loss-shifting argument of Jin et al. [2021], we first observe that our FTRL update can be equivalently written as
$$
\widehat {q} _ {t} = \underset {q \in \Omega (\widetilde {P} _ {i})} {\operatorname {a r g m i n}} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} \left(\widehat {\ell} _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q) = \underset {x \in \Omega (\widetilde {P} _ {i})} {\operatorname {a r g m i n}} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} \left(g _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q),
$$