SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tiancheng Jin*  
University of Southern California  
tiancheng.jin@usc.edu
Chloe Rouyer  
University of Copenhagen  
chloe@di.ku.dk
Chen-Yu Wei  
MIT Institute for Data, Systems, and Society  
chenyuw@mit.edu
William Chang  
University of California, Los Angeles  
chang314@g.ucla.edu
Existing online learning algorithms for adversarial Markov Decision Processes achieve  $\mathcal{O}(\sqrt{T})$  regret after  $T$  rounds of interactions even if the loss functions are chosen arbitrarily by an adversary, with the caveat that the transition function has to be fixed. This is because it has been shown that adversarial transition functions make no-regret learning impossible. Despite such impossibility results, in this work, we develop algorithms that can handle both adversarial losses and adversarial transitions, with regret increasing smoothly in the degree of maliciousness of the adversary. More concretely, we first propose an algorithm that enjoys  $\widetilde{\mathcal{O}} (\sqrt{T} +C^{\mathsf{P}})$  regret where  $C^{\mathbb{P}}$  measures how adversarial the transition functions are and can be at most  $\mathcal{O}(T)$ . While this algorithm itself requires knowledge of  $C^{\mathbb{P}}$ , we further develop a black-box reduction approach that removes this requirement. Moreover, we also show that further refinements of the algorithm not only maintains the same regret bound, but also simultaneously adapts to easier environments (where losses are generated in a certain stochastically constrained manner as in Jin et al. [2021]) and achieves  $\widetilde{\mathcal{O}} (U + \sqrt{UC^{\mathsf{L}}} +C^{\mathsf{P}})$  regret, where  $U$  is some standard gap-dependent coefficient and  $C^{\mathsf{L}}$  is the amount of corruption on losses.
Markov Decision Processes (MDPs) are widely-used models for reinforcement learning. They are typically studied under a fixed transition function and a fixed loss function, which fails to capture scenarios where the environment is time-evolving or susceptible to adversarial corruption. This motivates many recent studies to overcome these challenges. In particular, one line of research, originated from Even-Dar et al. [2009] and later improved or generalized by e.g. Neu et al. [2010a,b], Zimin and Neu [2013], Rosenberg and Mansour [2019a], Jin et al. [2020], takes inspiration from the online learning literature and considers interacting with a sequence of  $T$  MDPs, each with an adversarily chosen loss function. Despite facing such a challenging environment, the learner can still ensure  $\mathcal{O}(\sqrt{T})$  regret (ignoring other dependence; same below) as shown by these works, that is, the learner's average performance is close to that of the best fixed policy up to  $\mathcal{O}(1 / \sqrt{T})$ .
However, one caveat of these studies is that they all still require the MDPs to have to the same transition function. This is not for no reason — Abbasi Yadmori et al. [2013] shows that even with full information feedback, achieving sub-linear regret with adversarial transition functions is computationally hard, and Tian et al. [2021] complements this result by showing that under the more challenging bandit feedback, this goal becomes even information-theoretically impossible without paying exponential dependence on the episode length.
To get around such impossibility results, one natural idea is to allow the regret to depend on some measure of maliciousness  $C^{\mathsf{P}}$  of the transition functions, which is 0 when the transitions remain the same over time and  $\mathcal{O}(T)$  in the worst case when they are completely arbitrary. We review several such attempts at the end of this section, and point out here that they all suffer one issue: even when  $C^{\mathsf{P}} = 0$ , the algorithms developed in these works all suffer linear regret when the loss functions are completely arbitrary, while, as mentioned above,  $\mathcal{O}(\sqrt{T})$  regret is achievable in this case. This begs the question: when learning with completely adversarial MDPs, is  $\mathcal{O}(\sqrt{T} + C^{\mathsf{P}})$  regret achievable?
In this work, we not only answer this question affirmatively, but also show that one can perform even better sometimes. More concretely, our results are as follows.
1. In Section 3, we develop a variant of the UOB-REPS algorithm [Jin et al., 2020], achieving  $\widetilde{\mathcal{O}} (\sqrt{T} +C^{\mathsf{P}})$  regret in completely adversarial environments when  $C^\mathbb{P}$  is known. The algorithmic modifications we propose include an enlarged confidence set, using the log-barrier regularizer, and a novel amortized bonus term that leads to a critical "change of measure" effect in the analysis.
2. We then remove the requirement on the knowledge of  $C^{\mathsf{P}}$  in Section 4 by proposing a black-box reduction that turns any algorithm with  $\widetilde{\mathcal{O}}(\sqrt{T} + C^{\mathsf{P}})$  regret under known  $C^{\mathsf{P}}$  into another algorithm with the same guarantee (up to logarithmic factors) even if  $C^{\mathsf{P}}$  is unknown. Our reduction improves that of Wei et al. [2022] by allowing adversarial losses, which presents extra challenges as discussed in Pacchiano et al. [2022]. The idea of our reduction builds on top of previous adversarial model selection framework (a.k.a. Corral [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022]), but is even more general and is of independent interest: it shows that the requirement from previous work on having a stable input algorithm is actually redundant, since our method can turn any algorithm into a stable one.
3. Finally, in Section 5 we also further refine our algorithm so that it simultaneously adapts to the maliciousness of the loss functions and achieves  $\tilde{\mathcal{O}} (\min \{\sqrt{T},U + \sqrt{UC^{\mathsf{L}}}\} +C^{\mathsf{P}})$  regret, where  $U$  is some standard gap-dependent coefficient and  $C^\mathrm{L}\leq T$  is the amount of corruption on losses (this result unfortunately requires the knowledge of  $C^\mathbb{P}$ , but not  $U$  or  $C^\mathbb{L}$ ). This generalizes the so-called best-of-both-worlds guarantee of Jin and Luo [2020], Jin et al. [2021], Dann et al. [2023] from  $C^{\mathbb{P}} = 0$  to any  $C^\mathbb{P}$ , and is achieved by combining the ideas from Jin et al. [2021] and Ito [2021] with a novel optimistic transition technique. In fact, this technique also leads to improvement on the dependence of episode length even when  $C^\mathbb{P} = 0$ .
Related Work Here, we review how existing studies deal with adversarially chosen transition functions and how our results compare to theirs. The closest line of research is usually known as corruption robust reinforcement learning [Lykouris et al., 2019, Chen et al., 2021, Zhang et al., 2021, Wei et al., 2022], which assumes a ground truth MDP and measures the maliciousness of the adversary via the amount of corruption to the ground truth — the amount of corruption is essentially our  $C^{\mathsf{L}}$ , while the amount of corruption is essentially our  $C^{\mathsf{P}}$  (these will become clear after we provide their formal definitions later). Naturally, the regret in these works is defined as the difference between the learner's total loss and that of the best policy with respect to the ground truth MDP, in which case  $\widetilde{O}(\sqrt{T} + C^{\mathsf{P}} + C^{\mathsf{L}})$  regret is unavoidable and is achieved by the state-of-the-art [Wei et al., 2022].
On the other hand, following the canonical definition in online learning, we define regret with respect to the corrupted MDPs, in which case  $\widetilde{\mathcal{O}} (\sqrt{T} +C^{\mathsf{P}})$  is achievable as we show (regardless how large  $C^\mathrm{L}$  is). To compare these results, note that the two regret definitions differ from each other by an amount of at most  $\mathcal{O}(C^{\mathsf{P}} + C^{\mathsf{L}})$ . Therefore, our result implies that of Wei et al. [2022], but not vice versa — what Wei et al. [2022] achieves in our definition of regret is again  $\widetilde{\mathcal{O}} (\sqrt{T} +C^{\mathsf{P}} + C^{\mathsf{L}})$ , which is never better than ours and could be  $\Omega (T)$  even when  $C^{\mathsf{P}} = 0$ .
In fact, our result also improves upon that of Wei et al. [2022] in terms of the gap-dependent refinement — their refined bound is  $\widetilde{\mathcal{O}} (\min \{\sqrt{T},G\} +C^{\mathsf{P}} + C^{\mathsf{L}})$  for some gap-dependent measure
$G$  that is known to be no less than our gap-dependent measure  $U$ ; on the other hand, based on earlier discussion, our refined bound in their regret definition is  $\widetilde{\mathcal{O}} (\min \{\sqrt{T},U + \sqrt{UC^{\mathsf{L}}}\} +C^{\mathsf{P}} + C^{\mathsf{L}}) = \widetilde{\mathcal{O}} (\min \{\sqrt{T},U\} +C^{\mathsf{P}} + C^{\mathsf{L}})$  and thus better. The caveat is that, as mentioned, for this refinement our result requires the knowledge of  $C^{\mathsf{P}}$ , but Wei et al. [2022] does not. However, we emphasize again that for the gap-independent bound, our result does not require knowledge of  $C^{\mathsf{P}}$  and is achieved via an even more general black-box reduction compared to the reduction of Wei et al. [2022].
Finally, we mention that another line of research, usually known as non-stationary reinforcement learning, also allows arbitrary transition/loss functions and measures the difficulty by either the number of changes in the environment [Auer et al., 2008, Gajane et al., 2018] or some smoother measure such as the total variation across time [Wei and Luo, 2021, Cheung et al., 2023]. These results are less comparable to ours since their regret (known as dynamic regret) measures the performance of the learner against the best sequence of policies, while ours (known as static regret) measures the performance against the best fixed policy.
We consider the problem of sequentially learning  $T$  episodic MDPs, all with the same state space  $S$  and action space  $A$ . We assume without loss of generality (similarly to Jin et al. [2021]) that the state space  $S$  has a layered structure and is partitioned into  $L + 1$  subsets  $S_0, \ldots, S_L$  such that  $S_0$  and  $S_L$  only contain the initial state  $s_0$  and the terminal state  $s_L$  respectively, and transitions are only possible between consecutive layers. For notational convenience, for any  $k < L$ , we denote the set of tuples  $S_k \times A \times S_{k+1}$  by  $W_k$ . We also denote by  $k(s)$  the layer to which state  $s \in S$  belongs.
Ahead of time, knowing the learner's algorithm, the environment decides the transition functions  $\{P_t\}_{t=1}^T$  and the loss functions  $\{\ell_t\}_{t=1}^T$  for these  $T$  MDPs in an arbitrary manner (unknown to the learner). Then the learner sequentially interacts with these  $T$  MDPs: for each episode  $t = 1, \dots, T$ , the learner first decides a stochastic policy  $\pi_t : S \times A \to [0,1]$  where  $\pi_t(a|s)$  is the probability of taking action  $a$  when visiting state  $s$ ; then, starting from the initial state  $s_{t,0} = s_0$ , for each  $k = 0, \dots, L-1$ , the learner repeatedly selects an action  $a_{t,k}$  sampled from  $\pi_t(\cdot | s_{t,k})$ , suffers loss  $\ell_t(s_{t,k}, a_{t,k}) \in [0,1]$ , and transits to the next state  $s_{t,k+1}$  sampled from  $P_t(\cdot | s_{t,k}, a_{t,k})$  (until reaching the terminal state); finally, the learner observes the losses of those visited state-action pairs (a.k.a. bandit feedback).
Let  $\ell_t(\pi) = \mathbb{E}\big[\sum_{h=0}^{L-1} \ell_t(s_h, a_h) | P_t, \pi\big]$  be the expected loss of executing policy  $\pi$  in the  $t$ -th MDP (that is,  $\{(s_h, a_h)\}_{h=0}^{L-1}$  is a stochastic trajectory generated according to transition  $P_t$  and policy  $\pi$ ). Then, the regret of the learner against any policy  $\pi$  is defined as  $\mathrm{Reg}_T(\pi) = \mathbb{E}\big[\sum_{t=1}^T \ell_t(\pi_t) - \ell_t(\pi)\big]$ . We denote by  $\mathring{\pi}$  one of the optimal policies in hindsight such that  $\mathrm{Reg}_T(\mathring{\pi}) = \max_{\pi} \mathrm{Reg}_T(\pi)$  and use  $\mathrm{Reg}_T \triangleq \mathrm{Reg}_T(\mathring{\pi})$  as a shorthand.
Maliciousness Measure of the Transitions If the transition functions are all the same, Jin et al. [2020] shows that  $\mathrm{Reg}_T = \widetilde{\mathcal{O}}(L|S|\sqrt{|A|T})$  is achievable, no matter how the loss functions are decided. However, when the transition functions are also arbitrary, Tian et al. [2021] shows that  $\mathrm{Reg}_T = \Omega(\min\{T, \sqrt{2^L T}\})$  is unavoidable. Therefore, a natural goal is to allow the regret to smoothly increase from order  $\sqrt{T}$  to  $T$  when some maliciousness measure of the transitions increases. Specifically, the measure we use is
$$
C ^ {\mathbb {P}} \triangleq \min  _ {P ^ {\prime} \in \mathcal {P}} \sum_ {t = 1} ^ {T} \sum_ {k = 0} ^ {L - 1} \max  _ {(s, a) \in S _ {k} \times A} \| P _ {t} (\cdot | s, a) - P ^ {\prime} (\cdot | s, a) \| _ {1}, \tag {1}
$$
where  $\mathcal{P}$  denotes the set of all valid transition functions. Let  $P$  be the transition that realizes the minimum in this definition. Then  $C^{\mathsf{P}}$  can be regarded as the same corruption measure used in Chen et al. [2021]; there, it is assumed that a ground truth MDP with transition  $P$  exists, and the adversary corrupts it arbitrarily in each episode to obtain  $P_{t}$ , making  $C^{\mathsf{P}}$  the total amount of corruption measured in a certain norm. For simplicity, in the rest of this paper, we will also take this perspective and call
2When  $C^{\mathsf{P}} + C^{\mathsf{L}}$  is known, Lykouris et al. [2019] also achieves  $\widetilde{\mathcal{O}} (\min \{\sqrt{T},U\} +C^{\mathsf{P}} + C^{\mathsf{L}})$  regret, but similar to earlier discussions on gap-independent bounds, their regret definition is weaker than ours.
$C^{\mathsf{P}}$  the transition corruption. We also use  $C_t^{\mathsf{P}} = \sum_{k=0}^{L-1} \max_{(s,a) \in S_k \times A} \| P_t(\cdot | s, a) - P(\cdot | s, a) \|_1$  to denote the per-round corruption (so  $C^{\mathsf{P}} = \sum_{t=1}^{T} C_t^{\mathsf{P}}$ ). It is clear that  $C^{\mathsf{P}} = 0$  when the transition stays the same for all MDPs, while in the worst case it is at most  $2TL$ . Our goal is to achieve  $\operatorname{Reg}_T = \mathcal{O}(\sqrt{T} + C^{\mathsf{P}})$  (ignoring other dependence), which smoothly interpolates between the result of Jin et al. [2020] for  $C^{\mathsf{P}} = 0$  and that of Tian et al. [2021] for  $C^{\mathsf{P}} = \mathcal{O}(T)$ .
Enlarged Confidence Set A central technique to deal with unknown transitions is to maintain a shrinking confidence set that contains the ground truth with high probability [Rosenberg and Mansour, 2019a,b, Jin et al., 2020]. With a properly enlarged confidence set, the same idea extends to the case with adversarial transitions [Lykouris et al., 2019]. Specifically, all our algorithms deploy the following transition estimation procedure. It proceeds in epochs, indexed by  $i = 1, 2, \dots$ , and each epoch  $i$  includes some consecutive episodes. An epoch ends whenever we encounter a state-action pair whose total number of visits doubles itself when compared to the beginning of that epoch. At the beginning of each epoch  $i$ , we calculate an empirical transition  $\bar{P}_i$  as:
$$
\bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) = m _ {i} \left(s, a, s ^ {\prime}\right) / m _ {i} (s, a), \quad \forall (s, a, s ^ {\prime}) \in W _ {k}, k = 0, \dots L - 1, \tag {2}
$$
where  $m_{i}(s,a)$  and  $m_{i}(s,a,s^{\prime})$  are the total number of visits to  $(s,a)$  and  $(s,a,s^{\prime})$  prior to epoch  $i$ . In addition, we calculate the following transition confidence set.
Definition 2.1. (Confidence Set of Transition Functions) Let  $\delta \in (0,1)$  be a confidence parameter. With known corruption  $C^P$ , we define the confidence set of transition functions for epoch  $i$  as
$$
\mathcal {P} _ {i} = \left\{P \in \mathcal {P}: \left| P \left(s ^ {\prime} \mid s, a\right) - \bar {P} _ {i} \left(s ^ {\prime} \mid s, a\right) \right| \leq B _ {i} (s, a, s ^ {\prime}), \forall (s, a, s ^ {\prime}) \in W _ {k}, k = 0, \dots , L - 1 \right\}, \tag {3}
$$
where the confidence interval  $B_{i}(s,a,s^{\prime})$  is defined, with  $\iota = |S||A|T / \delta$  as
$$
B _ {i} (s, a, s ^ {\prime}) = \min  \left\{1, 1 6 \sqrt {\frac {\bar {P} _ {i} \left(s ^ {\prime} \mid s , a\right) \log (\iota)}{m _ {i} (s , a)}} + 6 4 \cdot \frac {C ^ {P} + \log (\iota)}{m _ {i} (s , a)} \right\}. \tag {4}
$$
Note that the confidence interval is enlarged according to how large the corruption  $C^{\mathbb{P}}$  is. We denote by  $\mathcal{E}_{\mathrm{CON}}$  the event that  $P \in \mathcal{P}_i$  for all epoch  $i$ , which is guaranteed to happen with high-probability.
Lemma 2.2. With probability at least  $1 - 2\delta$ , the event  $\mathcal{E}_{\mathrm{CON}}$  holds.
Occupancy Measure and Upper Occupancy Bound Similar to previous work [Rosenberg and Mansour, 2019a, Jin et al., 2020], given a stochastic policy  $\pi$  and a transition function  $\bar{P}$ , we define the occupancy measure  $q^{\bar{P},\pi}$  as the mapping from  $S \times A \times S$  to  $[0,1]$  such that  $q^{\bar{P},\pi}(s,a,s')$  is the probability of visiting  $(s,a,s')$  when executing policy  $\pi$  in an MDP with transition  $\bar{P}$ . Further define  $q^{\bar{P},\pi}(s,a) = \sum_{s' \in S} q^{\bar{P},\pi}(s,a,s')$  (the probability of visiting  $(s,a)$ ) and  $q^{\bar{P},\pi}(s) = \sum_{a \in A} q^{\bar{P},\pi}(s,a)$  (the probability of visiting  $s$ ). Given an occupancy measure  $q$ , the corresponding policy that defines it, denoted by  $\pi^q$ , can be extracted via  $\pi^q(a|s) \propto q(s,a)$ .
Importantly, the expected loss  $\ell_t(\pi)$  defined earlier equals  $\langle q^{P_t,\pi},\ell_t\rangle$ , making our problem a variant of online linear optimization and enabling the usage of standard algorithmic frameworks such as Online Mirror Descent (OMD) or Follow-the-Regularized-Leader (FTRL). These frameworks operate over a set of occupancy measures in the form of either  $\Omega (\bar{P}) = \{q^{\bar{P},\pi}:\pi$  is a stochastic policy\} for some transition function  $\bar{P}$ , or  $\Omega (\bar{\mathcal{P}}) = \{q^{\bar{P},\pi}:\bar{P}\in \bar{\mathcal{P}},\pi$  is a stochastic policy\} for some set of transition functions  $\bar{\mathcal{P}}$ .
Following Jin et al. [2020], to handle partial feedback on the loss function  $\ell_t$ , we need to construct loss estimators  $\widehat{\ell}_t$  using the (efficiently computable) upper occupancy bound  $u_t$ :
$$
\widehat {\ell} _ {t} (s, a) = \frac {\mathbb {I} _ {t} (s , a) \ell_ {t} (s , a)}{u _ {t} (s , a)}, \text {w h e r e} u _ {t} (s, a) = \max  _ {\widehat {P} \in \mathcal {P} _ {i (t)}} q ^ {\widehat {P}, \pi_ {t}} (s, a), \tag {5}
$$
$\mathbb{I}_t(s,a)$  is 1 if  $(s,a)$  is visited during episode  $t$  (so that  $\ell_t(s,a)$  is revealed), and 0 otherwise, and  $i(t)$  denotes the epoch index to which episode  $t$  belongs. We also define  $u_{t}(s) = \sum_{a\in A}u_{t}(s,a)$ .
3When  $m_{i}(s,a) = 0$  , we simply let the transition function to be uniform, that is,  $\bar{P}_i(s'|s,a) = 1 / \left|S_{k(s')}\right|$
3 Achieving  $\mathcal{O}(\sqrt{T} + C^{\mathbf{P}})$  with Known  $C^{\mathbf{P}}$
As the first step, we develop an algorithm that achieves our goal when  $C^{\mathsf{P}}$  is known. To introduce our solution, we first briefly review the UOB-REPS algorithm of Jin et al. [2020] (designed for  $C^{\mathsf{P}} = 0$ ) and point out why simply using the enlarged confidence set Eq. (3) when  $C^{\mathsf{P}} \neq 0$  is far away from solving the problem. Specifically, UOB-REPS maintains a sequence of occupancy measures  $\{\widehat{q}_t\}_{t=1}^T$  via OMD:  $\widehat{q}_{t+1} = \operatorname*{argmin}_{q \in \Omega(\mathcal{P}_{i(t+1)})} \eta\langle q, \widehat{\ell}_t \rangle + D_\phi(q, \widehat{q}_t)$ . Here,  $\eta > 0$  is a learning rate,  $\widehat{\ell}_t$  is the loss estimator defined in Eq. (5),  $\phi$  is the negative entropy regularizer, and  $D_\phi$  is the corresponding Bregman divergence. With  $\widehat{q}_t$  at hand, in episode  $t$ , the learner simply executes  $\pi_t = \pi^{\widehat{q}_t}$ . Standard analysis of OMD ensures a bound on the estimated regret  $\mathrm{REG} = \mathbb{E}\left[\sum_t \langle \widehat{q}_t - q^{P,\widehat{\pi}}, \widehat{\ell}_t \rangle\right]$ , and the rest of the analysis of Jin et al. [2020] boils down to bounding the difference between  $\mathrm{REG}$  and  $\mathrm{Reg}_T$ .
First Issue This difference between  $\mathrm{REG}$  and  $\mathrm{Reg}_T$  leads to the first issue when one tries to analyze UOB-REPS against adversarial transitions — it contains the following bias term that measures the difference between the optimal policy's estimated loss and its true loss:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle q ^ {P, \hat {\pi}}, \widehat {\ell} _ {t} - \ell_ {t} \right\rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \ell_ {t} (s, a) \left(\frac {q ^ {P _ {t} , \pi_ {t}} (s , a) - u _ {t} (s , a)}{u _ {t} (s , a)}\right) \right]. \tag {6}
$$
When  $C^{\mathsf{P}} = 0$ , we have  $P = P_{t}$ , and thus under the high probability event  $\mathcal{E}_{\mathrm{CON}}$  and by the definition of upper occupancy bound, we know  $q^{P_t,\pi_t}(s,a)\leq u_t(s,a)$ , making Eq. (6) negligible. However, this argument breaks when  $C^{\mathsf{P}}\neq 0$  and  $P\neq P_{t}$ . In fact,  $P_{t}$  can be highly different from any transitions in  $\mathcal{P}_{i(t)}$  with respect to which  $u_{t}$  is defined, making Eq. (6) potentially huge.
Solution: Change of Measure via Amortized Bonuses Given that  $q^{P,\pi_t}(s,a) \leq u_t(s,a)$  does still hold with high probability, Eq. (6) is (approximately) bounded by
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s, a} q ^ {P, \hat {\pi}} (s, a) \frac {| q ^ {P _ {t} , \pi_ {t}} (s , a) - q ^ {P , \pi_ {t}} (s , a) |}{u _ {t} (s , a)} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {s} q ^ {P, \hat {\pi}} (s) \frac {| q ^ {P _ {t} , \pi_ {t}} (s) - q ^ {P , \pi_ {t}} (s) |}{u _ {t} (s)} \right]
$$
which is at most  $\mathbb{E}\left[\sum_{t=1}^{T}\sum_{s}q^{P,\hat{\pi}}(s)\frac{C_t^{\mathbb{P}}}{u_t(s)}\right]$  since  $\left|q^{P_t,\pi_t}(s) - q^{P,\pi_t}(s)\right|$  is bounded by the per-round corruption  $C_t^{\mathbb{P}}$  (see Corollary D.3.6). While this quantity is potentially huge, if we could "change the measure" from  $q^{P,\hat{\pi}}$  to  $\widehat{q}_t$ , then the resulting quantity  $\mathbb{E}\left[\sum_{t=1}^{T}\sum_{s}\widehat{q}_t(s)\frac{C_t^{\mathbb{P}}}{u_t(s)}\right]$  is at most  $|S|C^{\mathbb{P}}$  since  $\widehat{q}_t(s) \leq u_t(s)$  by definition. The general idea of such a change of measure has been extensively used in the online learning literature (see Luo et al. [2021] in a most related context) and can be realized by changing the loss fed to OMD from  $\widehat{\ell}_t$  to  $\widehat{\ell}_t - b_t$  for some bonus term  $b_t$ , which, in our case, should satisfy  $b_t(s,a) \approx \frac{C_t^{\mathbb{P}}}{u_t(s)}$ . However, the challenge here is that  $C_t^{\mathbb{P}}$  is unknown!
Our solution is to introduce a type of efficiently computable amortized bonuses that do not change the measure per round, but do so overall. Specifically, our amortized bonus  $b_{t}$  is defined as
$$
b _ {t} (s, a) = \left\{ \begin{array}{l l} \frac {4 L}{u _ {t} (s)} & \text {i f} \sum_ {\tau = 1} ^ {t} \mathbb {I} \{\lceil \log_ {2} u _ {\tau} (s) \rceil = \lceil \log_ {2} u _ {t} (s) \rceil \} \leq \frac {C ^ {\mathcal {P}}}{2 L}, \\ 0 & \text {e l s e ,} \end{array} \right. \tag {7}
$$