Title: No-Regret Online Reinforcement Learning with Adversarial Losses and Transitions
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement Learning

best of both worlds

MDP

robust RL

adversarial corruption
Abstract: Existing online learning algorithms for adversarial Markov Decision Processes 
achieve $\mathcal{O}(\sqrt{T})$ regret after $T$ rounds of interactions even if the loss functions are chosen arbitrarily by an adversary, with the caveat that the transition function has to be fixed.
This is because it has been shown that adversarial transition functions make no-regret learning impossible.
Despite such impossibility results, in this work, we develop algorithms that can handle both adversarial losses and adversarial transitions, with regret increasing smoothly in the degree of maliciousness of the adversary.
More concretely, we first propose an algorithm that enjoys $\widetilde{\mathcal{O}}(\sqrt{T} + C^{P})$ regret where $C^{P}$ measures how adversarial the transition functions are and can be at most $\mathcal{O}(T)$.
While this algorithm itself requires knowledge of $C^{P}$, we further develop a black-box reduction approach that removes this requirement.
Moreover, we also show that further refinements of the algorithm not only maintains the same regret bound, but also simultaneously adapts to easier environments (where losses are generated in a certain stochastically constrained manner as in [Jin et al. 2021]) and achieves $\widetilde{\mathcal{O}}(U + \sqrt{UC^{L}}  + C^{P})$ regret, where $U$ is some standard gap-dependent coefficient and $C^{L}$ is the amount of corruption on losses.

No-Regret Online Reinforcement Learning with Adversarial Losses and Transitions

Abstract

Existing online learning algorithms for adversarial Markov Decision Processes achieve O ( √ T ) regret after T rounds of interactions even if the loss functions are chosen arbitrarily by an adversary, with the caveat that the transition function has to be fixed. This is because it has been shown that adversarial transition functions make no-regret learning impossible. Despite such impossibility results, in this work, we develop algorithms that can handle both adversarial losses and adversarial transitions, with regret increasing smoothly in the degree of maliciousness of the adversary. More concretely, we first propose an algorithm that enjoys ˜ O ( √ T + C P ) regret where C P measures how adversarial the transition functions are and can be at most O ( T ) . While this algorithm itself requires knowledge of C P , we further develop a black-box reduction approach that removes this requirement. Moreover, we also show that further refinements of the algorithm not only maintains the same regret bound, but also simultaneously adapts to easier environments (where losses are generated in a certain stochastically constrained manner as in Jin et al. [2021]) and achieves ˜ O ( U + √ UC L + C P ) regret, where U is some standard gap-dependent coefficient and C L is the amount of corruption on losses.

1 Introduction

Markov Decision Processes (MDPs) are widely-used models for reinforcement learning. They are typically studied under a fixed transition function and a fixed loss function, which fails to capture scenarios where the environment is time-evolving or susceptible to adversarial corruption. This motivates many recent studies to overcome these challenges. In particular, one line of research, originated from Even-Dar et al. [2009] and later improved or generalized by e.g. Neu et al. [2010a,b], Zimin and Neu [2013], Rosenberg and Mansour [2019a], Jin et al. [2020], takes inspiration from the online learning literature and considers interacting with a sequence of T MDPs, each with an adversarially chosen loss function. Despite facing such a challenging environment, the learner can still ensure O ( √ T ) regret (ignoring other dependence; same below) as shown by these works, that is, the learner's average performance is close to that of the best fixed policy up to O (1 / √ T ) .

However, one caveat of these studies is that they all still require the MDPs to have to the same transition function. This is not for no reason - Abbasi Yadkori et al. [2013] shows that even with full information feedback, achieving sub-linear regret with adversarial transition functions is computationally hard, and Tian et al. [2021] complements this result by showing that under the more challenging bandit feedback, this goal becomes even information-theoretically impossible without paying exponential dependence on the episode length.

To get around such impossibility results, one natural idea is to allow the regret to depend on some measure of maliciousness C P of the transition functions, which is 0 when the transitions remain the same over time and O ( T ) in the worst case when they are completely arbitrary. We review several such attempts at the end of this section, and point out here that they all suffer one issue: even when C P = 0 , the algorithms developed in these works all suffer linear regret when the loss functions are completely arbitrary, while, as mentioned above, O ( √ T ) regret is achievable in this case. This begs the question: when learning with completely adversarial MDPs, is O ( √ T + C P ) regret achievable?

In this work, we not only answer this question affirmatively, but also show that one can perform even better sometimes. More concretely, our results are as follows.

In Section 3, we develop a variant of the UOB-REPS algorithm [Jin et al., 2020], achieving ˜ O ( √ T + C P ) regret in completely adversarial environments when C P is known. The algorithmic modifications we propose include an enlarged confidence set, using the log-barrier regularizer, and a novel amortized bonus term that leads to a critical 'change of measure' effect in the analysis.

We then remove the requirement on the knowledge of C P in Section 4 by proposing a blackbox reduction that turns any algorithm with ˜ O ( √ T + C P ) regret under known C P into another algorithm with the same guarantee (up to logarithmic factors) even if C P is unknown. Our reduction improves that of Wei et al. [2022] by allowing adversarial losses, which presents extra challenges as discussed in Pacchiano et al. [2022]. The idea of our reduction builds on top of previous adversarial model selection framework (a.k.a. Corral [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022]), but is even more general and is of independent interest: it shows that the requirement from previous work on having a stable input algorithm is actually redundant, since our method can turn any algorithm into a stable one.

Finally, in Section 5 we also further refine our algorithm so that it simultaneously adapts to the maliciousness of the loss functions and achieves ˜ O (min { √ T, U + √ UC L } + C P ) regret, where U is some standard gap-dependent coefficient and C L ≤ T is the amount of corruption on losses (this result unfortunately requires the knowledge of C P , but not U or C L ). This generalizes the so-called best-of-both-worlds guarantee of Jin and Luo [2020], Jin et al. [2021], Dann et al. [2023] from C P = 0 to any C P , and is achieved by combining the ideas from Jin et al. [2021] and Ito [2021] with a novel optimistic transition technique. In fact, this technique also leads to improvement on the dependence of episode length even when C P = 0 .

Related Work Here, we review how existing studies deal with adversarially chosen transition functions and how our results compare to theirs. The closest line of research is usually known as corruption robust reinforcement learning [Lykouris et al., 2019, Chen et al., 2021, Zhang et al., 2021, Wei et al., 2022], which assumes a ground truth MDP and measures the maliciousness of the adversary via the amount of corruption to the ground truth - the amount of corruption is essentially our C L , while the amount of corruption is essentially our C P (these will become clear after we provide their formal definitions later). Naturally, the regret in these works is defined as the difference between the learner's total loss and that of the best policy with respect to the ground truth MDP , in which case ˜ O ( √ T + C P + C L ) regret is unavoidable and is achieved by the state-of-the-art [Wei et al., 2022].

On the other hand, following the canonical definition in online learning, we define regret with respect to the corrupted MDPs , in which case ˜ O ( √ T + C P ) is achievable as we show (regardless how large C L is). To compare these results, note that the two regret definitions differ from each other by an amount of at most O ( C P + C L ) . Therefore, our result implies that of Wei et al. [2022], but not vice versa -what Wei et al. [2022] achieves in our definition of regret is again ˜ O ( √ T + C P + C L ) , which is never better than ours and could be Ω( T ) even when C P = 0 .

In fact, our result also improves upon that of Wei et al. [2022] in terms of the gap-dependent refinement - their refined bound is ˜ O (min { √ T, G } + C P + C L ) for some gap-dependent measure

G that is known to be no less than our gap-dependent measure U ; on the other hand, based on earlier discussion, our refined bound in their regret definition is ˜ O (min { √ T, U + √ UC L } + C P + C L ) = ˜ O (min { √ T, U } + C P + C L ) and thus better. The caveat is that, as mentioned, for this refinement our result requires the knowledge of C P , but Wei et al. [2022] does not. 2 However, we emphasize again that for the gap-independent bound, our result does not require knowledge of C P and is achieved via an even more general black-box reduction compared to the reduction of Wei et al. [2022].

Finally, we mention that another line of research, usually known as non-stationary reinforcement learning , also allows arbitrary transition/loss functions and measures the difficulty by either the number of changes in the environment [Auer et al., 2008, Gajane et al., 2018] or some smoother measure such as the total variation across time [Wei and Luo, 2021, Cheung et al., 2023]. These results are less comparable to ours since their regret (known as dynamic regret) measures the performance of the learner against the best sequence of policies, while ours (known as static regret) measures the performance against the best fixed policy.

2 Preliminaries

We consider the problem of sequentially learning T episodic MDPs, all with the same state space S and action space A . We assume without loss of generality (similarly to Jin et al. [2021]) that the state space S has a layered structure and is partitioned into L +1 subsets S 0 , . . . , S L such that S 0 and S L only contain the initial state s 0 and the terminal state s L respectively, and transitions are only possible between consecutive layers. For notational convenience, for any k < L , we denote the set of tuples S k × A × S k +1 by W k . We also denote by k ( s ) the layer to which state s ∈ S belongs.

Ahead of time, knowing the learner's algorithm, the environment decides the transition functions { P t } T t =1 and the loss functions { ℓ t } T t =1 for these T MDPs in an arbitrary manner (unknown to the learner). Then the learner sequentially interacts with these T MDPs: for each episode t = 1 , . . . , T , the learner first decides a stochastic policy π t : S × A → [0 , 1] where π t ( a | s ) is the probability of taking action a when visiting state s ; then, starting from the initial state s t, 0 = s 0 , for each k = 0 , . . . , L -1 , the learner repeatedly selects an action a t,k sampled from π t ( ·| s t,k ) , suffers loss ℓ t ( s t,k , a t,k ) ∈ [0 , 1] , and transits to the next state s t,k +1 sampled from P t ( ·| s t,k , a t,k ) (until reaching the terminal state); finally, the learner observes the losses of those visited state-action pairs (a.k.a. bandit feedback).

Let ℓ t ( π ) = E [ ∑ L -1 h =0 ℓ t ( s h , a h ) | P t , π ] be the expected loss of executing policy π in the t -th MDP (that is, { ( s h , a h ) } L -1 h =0 is a stochastic trajectory generated according to transition P t and policy π ). Then, the regret of the learner against any policy π is defined as Reg T ( π ) = E [ ∑ T t =1 ℓ t ( π t ) -ℓ t ( π ) ] . We denote by ˚ π one of the optimal policies in hindsight such that Reg T (˚ π ) = max π Reg T ( π ) and use Reg T ≜ Reg T (˚ π ) as a shorthand.

Maliciousness Measure of the Transitions If the transition functions are all the same, Jin et al. [2020] shows that Reg T = ˜ O ( L | S | √ | A | T ) is achievable, no matter how the loss functions are decided. However, when the transition functions are also arbitrary, Tian et al. [2021] shows that Reg T = Ω(min { T, √ 2 L T } ) is unavoidable. Therefore, a natural goal is to allow the regret to smoothly increase from order √ T to T when some maliciousness measure of the transitions increases. Specifically, the measure we use is

where P denotes the set of all valid transition functions. Let P be the transition that realizes the minimum in this definition. Then C P can be regarded as the same corruption measure used in Chen et al. [2021]; there, it is assumed that a ground truth MDP with transition P exists, and the adversary corrupts it arbitrarily in each episode to obtain P t , making C P the total amount of corruption measured in a certain norm. For simplicity, in the rest of this paper, we will also take this perspective and call

C P the transition corruption. We also use C P t = ∑ L -1 k =0 max ( s,a ) ∈ S k × A ∥ P t ( ·| s, a ) -P ( ·| s, a ) ∥ 1 to denote the per-round corruption (so C P = ∑ T t =1 C P t ). It is clear that C P = 0 when the transition stays the same for all MDPs, while in the worst case it is at most 2 TL . Our goal is to achieve Reg T = O ( √ T + C P ) (ignoring other dependence), which smoothly interpolates between the result of Jin et al. [2020] for C P = 0 and that of Tian et al. [2021] for C P = O ( T ) .

Enlarged Confidence Set A central technique to deal with unknown transitions is to maintain a shrinking confidence set that contains the ground truth with high probability [Rosenberg and Mansour, 2019a,b, Jin et al., 2020]. With a properly enlarged confidence set, the same idea extends to the case with adversarial transitions [Lykouris et al., 2019]. Specifically, all our algorithms deploy the following transition estimation procedure. It proceeds in epochs, indexed by i = 1 , 2 , · · · , and each epoch i includes some consecutive episodes. An epoch ends whenever we encounter a state-action pair whose total number of visits doubles itself when compared to the beginning of that epoch. At the beginning of each epoch i , we calculate an empirical transition ¯ P i as:

where m i ( s, a ) and m i ( s, a, s ′ ) are the total number of visits to ( s, a ) and ( s, a, s ′ ) prior to epoch i . 3 In addition, we calculate the following transition confidence set.

Definition 2.1. (Confidence Set of Transition Functions) Let δ ∈ (0 , 1) be a confidence parameter. With known corruption C P , we define the confidence set of transition functions for epoch i as

where the confidence interval B i ( s, a, s ′ ) is defined, with ι = | S || A | T / δ as

P

Note that the confidence interval is enlarged according to how large the corruption C is. We denote by E CON the event that P ∈ P i for all epoch i , which is guaranteed to happen with high-probability. Lemma 2.2. With probability at least 1 -2 δ , the event E CON holds.

Occupancy Measure and Upper Occupancy Bound Similar to previous work [Rosenberg and Mansour, 2019a, Jin et al., 2020], given a stochastic policy π and a transition function ¯ P , we define the occupancy measure q ¯ P,π as the mapping from S × A × S to [0 , 1] such that q ¯ P,π ( s, a, s ′ ) is the probability of visiting ( s, a, s ′ ) when executing policy π in an MDP with transition ¯ P . Further define q ¯ P,π ( s, a ) = ∑ s ′ ∈ S q ¯ P,π ( s, a, s ′ ) (the probability of visiting ( s, a ) ) and q ¯ P,π ( s ) = ∑ a ∈ A q ¯ P,π ( s, a ) (the probability of visiting s ). Given an occupancy measure q , the corresponding policy that defines it, denoted by π q , can be extracted via π q ( a | s ) ∝ q ( s, a ) .

Importantly, the expected loss ℓ t ( π ) defined earlier equals 〈 q P t ,π , ℓ t 〉 , making our problem a variant of online linear optimization and enabling the usage of standard algorithmic frameworks such as Online Mirror Descent (OMD) or Follow-the-Regularized-Leader (FTRL). These frameworks operate over a set of occupancy measures in the form of either Ω( ¯ P ) = { q ¯ P,π : π is a stochastic policy } for some transition function ¯ P , or Ω( ¯ P ) = { q ¯ P,π : ¯ P ∈ ¯ P , π is a stochastic policy } for some set of transition functions ¯ P .

Following Jin et al. [2020], to handle partial feedback on the loss function ℓ t , we need to construct loss estimators ̂ ℓ t using the (efficiently computable) upper occupancy bound u t :

I t ( s, a ) is 1 if ( s, a ) is visited during episode t (so that ℓ t ( s, a ) is revealed), and 0 otherwise, and i ( t ) denotes the epoch index to which episode t belongs. We also define u t ( s ) = ∑ a ∈ A u t ( s, a ) .

3 Achieving O ( √ T + C P ) with Known C P

As the first step, we develop an algorithm that achieves our goal when C P is known. To introduce our solution, we first briefly review the UOB-REPS algorithm of Jin et al. [2020] (designed for C P = 0 ) and point out why simply using the enlarged confidence set Eq. (3) when C P = 0 is far away from solving the problem. Specifically, UOB-REPS maintains a sequence of occupancy measures { ̂ q t } T t =1 via OMD: ̂ q t +1 = argmin q ∈ Ω ( P i ( t +1) ) η ⟨ q, ̂ ℓ t ⟩ + D ϕ ( q, ̂ q t ) . Here, η > 0 is a learning rate, ̂ ℓ t is the loss estimator defined in Eq. (5), ϕ is the negative entropy regularizer, and D ϕ is the corresponding Bregman divergence. 4 With ̂ q t at hand, in episode t , the learner simply executes π t = π ̂ q t . Standard analysis of OMD ensures a bound on the estimated regret REG = E [ ∑ t ⟨ ̂ q t -q P, ˚ π , ̂ ℓ t ⟩ ] , and the rest of the analysis of Jin et al. [2020] boils down to bounding the difference between REG and Reg T .

̸

First Issue This difference between REG and Reg T leads to the first issue when one tries to analyze UOB-REPS against adversarial transitions - it contains the following bias term that measures the difference between the optimal policy's estimated loss and its true loss:

̸

When C P = 0 , we have P = P t , and thus under the high probability event E CON and by the definition of upper occupancy bound, we know q P t ,π t ( s, a ) ≤ u t ( s, a ) , making Eq. (6) negligible. However, this argument breaks when C P = 0 and P = P t . In fact, P t can be highly different from any transitions in P i ( t ) with respect to which u t is defined, making Eq. (6) potentially huge.

̸

Solution: Change of Measure via Amortized Bonuses Given that q P,π t ( s, a ) ≤ u t ( s, a ) does still hold with high probability, Eq. (6) is (approximately) bounded by

which is at most E [ ∑ T t =1 ∑ s q P, ˚ π ( s ) C P t u t ( s ) ] since ∣ ∣ q P t ,π t ( s ) -q P,π t ( s ) ∣ ∣ is bounded by the perround corruption C P t (see Corollary D.3.6). While this quantity is potentially huge, if we could 'change the measure' from q P, ˚ π to ̂ q t , then the resulting quantity E [ ∑ T t =1 ∑ s ̂ q t ( s ) C P t u t ( s ) ] is at most | S | C P since ̂ q t ( s ) ≤ u t ( s ) by definition. The general idea of such a change of measure has been extensively used in the online learning literature (see Luo et al. [2021] in a most related context) and can be realized by changing the loss fed to OMD from ̂ ℓ t to ̂ ℓ t -b t for some bonus term b t , which, in our case, should satisfy b t ( s, a ) ≈ C P t u t ( s ) . However, the challenge here is that C P t is unknown !

Our solution is to introduce a type of efficiently computable amortized bonuses that do not change the measure per round, but do so overall. Specifically, our amortized bonus b t is defined as

which we also write as b t ( s ) since it is independent of a . To understand this definition, note that -⌈ log 2 u t ( s ) ⌉ is exactly the unique integer j such that u t ( s ) falls into the bin (2 -j -1 , 2 -j ] . Therefore, the expression ∑ t τ =1 I {⌈ log 2 u τ ( s ) ⌉ = ⌈ log 2 u t ( s ) ⌉} counts, among all previous rounds τ = 1 , . . . , t , how many times we have encountered a u τ ( s ) value that falls into the same bin as u t ( s ) . If this number does not exceed C P 2 L , we apply a bonus of 4 L u t ( s ) , which is (two times of) the maximum possible value of the unknown quantity C P t u t ( s ) ; otherwise, we do not apply any bonus. The idea is that by enlarging the bonus to it maximum value and stopping it after enough times, even though each b t ( s ) might be quite different from C P t u t ( s ) , overall they behave similarly after T episodes:

Algorithm 1 Algorithm for Adversarial Transitions (with Known C P )

Input: confidence parameter δ ∈ (0 , 1) , learning rate η > 0 . Initialize: epoch index i = 1 ; counters m 1 ( s, a ) = m 1 ( s, a, s ′ ) = m 0 ( s, a ) = m 0 ( s, a, s ′ ) = 0 for all ( s, a, s ′ ) ; empirical transition ¯ P 1 and confidence width B 1 based on Eq. (2) and Eq. (4); occupancy measure ̂ q 1 ( s, a, s ′ ) = 1 | S k || A || S k +1 | for all ( s, a, s ′ ) ; and initial policy π 1 = π ̂ q 1 .

Execute policy π and obtain trajectory ( s , a ) for k = 0 , . . . , L -1 .

Construct loss estimator ℓ

Update b ( s ) for all s based on Eq. (7).

t t,k t,k ̂ t as defined in Eq. (5). t

Increase counters: for each k < L , m i ( s t,k , a t,k , s t,k +1 ) + ← 1 , m i ( s t,k , a t,k ) + ← 1 .

Increase epoch index i ← 1 .

Initialize new counters: ∀ ( s, a, s ′ ) , m i ( s, a, s ′ ) = m i -1 ( s, a, s ′ ) , m i ( s, a ) = m i -1 ( s, a ) . Update confidence set P i based on Eq. (3).

Let D ϕ ( · , · ) be the Bregman divergence with respect to log barrier (Eq. (11)) and compute

Update policy π t +1 = π ̂ q t +1 .

Lemma 3.1. The amortized bonus defined in Eq. (7) satisfies ∑ T t =1 C P t u t ( s ) ≤ ∑ T t =1 b t ( s ) and ∑ T t =1 ̂ q t ( s ) b t ( s ) = O ( C P log T ) for any s .

Therefore, the problematic term Eq. (6) is at most E [ ∑ t ⟨ q P, ˚ π , b t ⟩ ] , which, if 'converted' to E [ ∑ t ⟨ ̂ q t , b t ⟩ ] (change of measure), is nicely bounded by O ( | S | C P log T ) . As mentioned, such a change of measure can be realized by feeding ̂ ℓ t -b t instead of ̂ ℓ t to OMD, because now standard analysis of OMD ensures a bound on REG = E [ ∑ t ⟨ ̂ q t -q P, ˚ π , ̂ ℓ t -b t ⟩ ] , which, compared to the earlier definition of REG, leads to a difference of E [ ∑ t ⟨ ̂ q t -q P, ˚ π , b t ⟩ ] (see Appendix A.5 for details).

̸

Second Issue The second issue comes from analyzing REG (which exists even if no bonuses are used). Specifically, standard analysis of OMD requires bounding a 'stability' term, which, for the negative entropy regularizer, is in the form of E [ ∑ t ∑ s,a ̂ q t ( s, a ) ̂ ℓ t ( s, a ) 2 ] = E [ ∑ t ∑ s,a ̂ q t ( s, a ) q Pt ,π t ( s,a ) ℓ t ( s,a ) 2 u t ( s,a ) 2 ] ≤ E [ ∑ t ∑ s,a q Pt ,π t ( s,a ) u t ( s,a ) ] . Once again, when C P = 0 and P t = P , we have q P t ,π t bounded by u t ( s, a ) with high probability, and thus the stability term is O ( T | S || A | ) ; but this breaks if C P = 0 and P t can be arbitrarily different from transitions in P i ( t ) .

Solution: Log-Barrier Regularizer Resolving this second issue, however, is relatively straightforward - it suffices to switch the regularizer from negative entropy to log-barrier: ϕ ( q ) = -∑ L -1 k =0 ∑ ( s,a,s ′ ) ∈ W k log q ( s, a, s ′ ) , which is first used by Lee et al. [2020] in the context of learning adversarial MDPs but dates back to earlier work such as Foster et al. [2016] for multi-armed bandits. An important property of log-barrier is that it leads to a smaller stability term in the form of E [ ∑ t ∑ s,a ̂ q t ( s, a ) 2 ̂ ℓ t ( s, a ) 2 ] (with an extra ̂ q t ( s, a ) ), which is at most E [ ∑ t ∑ s,a q P t ,π t ( s, a ) ℓ t ( s, a ) 2 ] = O ( TL ) since ̂ q t ( s, a ) ≤ u t ( s, a ) . In fact, this also helps control the extra stability term when bonuses are used, which is in the form of E [ ∑ t ∑ s,a ̂ q t ( s, a ) 2 b t ( s, a ) 2 ] and is at most 4 L E [ ∑ t ⟨ ̂ q t , b t ⟩ ] = O ( L | S | C P log T ) according to Lemma 3.1.

Putting these two ideas together leads to our final algorithm (see Algorithm 1). We prove the following regret bound in Appendix A, which recovers that of Jin et al. [2020] when C P = 0 and increases linearly in C P as desired.

▷ entering a new epoch

4 Achieving O ( √ T + C P ) with Unknown C P

In this section, we address the case when the amount of corruption is unknown. We develop a black-box reduction which turns an algorithm that only deals with known C P to one that handles unknown C P . This is similar to Wei et al. [2022] but additionally handles adversarial losses using a different approach. A byproduct of our reduction is that we develop an entirely black-box model selection approach for adversarial online learning problems, as opposed to the gray-box approach developed by the 'Corral' literature [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022] which requires checking if the base algorithm is stable . To achieve this, we essentially develop another layer of reduction that turns any standard algorithm with sublinear regret into a stable algorithm. This result itself might be of independent interest and useful for solving other model selection problems.

More specifically, our reduction has two layers. The bottom layer is where our novelty lies: it takes as input an arbitrary corruption-robust algorithm that operates under known C P (e.g., the one we developed in Section 3), and outputs a stable corruption-robust algorithm (formally defined later) that still operates under known C P . The top layer, on the other hand, follows the standard Corral idea and takes as input a stable algorithm that operates under known C P , and outputs an algorithm that operates under unknown C P . Below, we explain these two layers of reduction in details.

Bottom Layer (from an Arbitrary Algorithm to a Stable Algorithm) The input of the bottom layer is an arbitrary corruption-robust algorithm, formally defined as:

Definition 4.1. An adversarial MDP algorithm is corruption-robust if it takes θ (a guess on the corruption amount) as input, and achieves the following regret for any random stopping time t ′ ≤ T :

for problem-dependent constants and log( T ) factors β 1 ≥ L 2 , β 2 ≥ L, β 3 ≥ 1 , where C P 1: t ′ = ∑ t ′ τ =1 C P τ is the total corruption up to time t ′ .

While the regret bound in Definition 4.1 might look cumbersome, it is in fact fairly reasonable: if the guess θ is not smaller than the true corruption amount, the regret should be of order √ t ′ + θ ; otherwise, the regret bound is vacuous since LT is its largest possible value. The only extra requirement is that the algorithm needs to be anytime (i.e., the regret bound holds for any stopping time t ′ ), but even this is known to be easily achievable by using a doubling trick over a fixed-time algorithm. It is then clear that our algorithm in Section 3 (together with a doubling trick) indeed satisfies Definition 4.1.

As mentioned, the output of the bottom layer is a stable robust algorithm. To characterize stability, we follow Agarwal et al. [2017] and define a new learning protocol that abstracts the interaction between the output algorithm of the bottom layer and the master algorithm from the top layer:

Protocol 1. In every round t , before the learner makes a decision, a probability w t ∈ [0 , 1] is revealed to the learner. After making a decision, the learner sees the desired feedback from the environment with probability w t , and sees nothing with probability 1 -w t .

In such a learning protocol, Agarwal et al. [2017] defines a stable algorithm as one whose regret smoothly degrades with ρ T = 1 . For our purpose here, we additionally require that the dependence on C in the regret bound is linear, which results in the following definition:

min t ∈ [ T ] w t P

Definition 4.2 ( 1 2 -stable corruption-robust algorithm) . A 1 2 -stable corruption-robust algorithm is one that, with prior knowledge on C P , achieves Reg T ≤ E [ √ β 1 ρ T T + β 2 ρ T ] + β 3 C P under Protocol 1 for problem-dependent constants and log( T ) factors β 1 ≥ L 2 , β 2 ≥ L , and β 3 ≥ 1 .

For simplicity, we only define and discuss the 1 2 -stability notion here (the parameter 1 2 refers to the exponent of T ), but our result can be straightforwardly extended to the general α -stability notion for

Algorithm 2 ST able A lgorithm B y I ndependent L earners and I nstance SE lection (STABILISE)

Input : C P and a base algorithm satisfying Definition 4.1. Initialize : ⌈ log 2 T ⌉ instances of the base algorithm ALG 1 , . . . , ALG ⌈ log 2 T ⌉ , where ALG j is configured with the parameter

α ∈ [ 1 2 , 1) as in Agarwal et al. [2017]. Our main result in this section is then that one can convert any corruption-robust algorithm into a 1 2 -stable corruption-robust algorithm:

Theorem 4.3. If an algorithm is corruption robust according to Definition 4.1 for some constants ( β 1 , β 2 , β 3 ) , then one can convert it to a 1 2 -stable corruption-robust algorithm (Definition 4.2) with constants ( β ′ 1 , β ′ 2 , β ′ 3 ) where β ′ 1 = O ( β 1 log T ) , β ′ 2 = O ( β 2 + β 3 L log T ) , and β ′ 3 = O ( β 3 log T ) .

This conversion is achieved by a procedure that we call STABILISE (see Algorithm 2 for details). The high-level idea of STABILISE is as follows. Noticing that the challenge when learning in Protocol 1 is that w t varies over time, we discretize the value of w t and instantiate one instance of the input algorithm to deal with one possible discretized value, so that it is learning in Protocol 1 but with a fixed w t , making it straightforward to bound its regret based on what it promises in Definition 4.1.

More concretely, STABILISE instantiates O (log 2 T ) instances { ALG j } ⌈ log 2 T ⌉ j =0 of the input algorithm that satisfies Definition 4.1, each with a different parameter θ j . Upon receiving w t from the environment, it dispatches round t to the j -th instance where j is such that w t ∈ (2 -j -1 , 2 -j ] , and uses the policy generated by ALG j to interact with the environment (if w t ≤ 1 T , simply ignore this round). Based on Protocol 1, the feedback for this round is received with probability w t . To equalize the probability of ALG j receiving feedback as mentioned in the high-level idea, when the feedback is actually obtained, STABILISE sends it to ALG j only with probability 2 -j -1 w t (and discards it otherwise). This way, every time ALG j is assigned to a round, it always receives the desired feedback with probability w t · 2 -j -1 w t = 2 -j -1 . This equalization step is the key that allows us to use the original guarantee of the base algorithm (Definition 4.1) and run it as it is, without requiring it to perform extra importance weighting steps as in Agarwal et al. [2017].

The choice of θ j is crucial in making sure that STABILISE only has C P regret overhead instead of ρ T C P . Since ALG j only receives feedback with probability 2 -j -1 , the expected total corruption it experiences is on the order of 2 -j -1 C P . Therefore, its input parameter θ j only needs to be of this order instead of the total corruption C P . This is similar to the key idea of Wei et al. [2022] and Lykouris et al. [2018]. See Appendix B.1 for more details and the full proof of Theorem 4.3.

Top Layer (from Known C P to Unknown C P ) With a stable algorithm and a regret guarantee in Definition 4.2, it is relatively standard to convert it to an algorithm with ˜ O ( √ T + C P ) regret without knowing C P . Similar arguments have been made in Foster et al. [2020], and the idea is to have another specially designed OMD/FTRL-based master algorithm to choose on the fly among a set of instances of this stable base algorithm, each with a different guess on C P (the probability w t in Protocol 1 is then decided by this master algorithm). We defer all details to Appendix B. The final regret guarantee is the following ( ˜ O ( · ) hides log( T ) factors).

Theorem 4.4. Using an algorithm satisfying Definition 4.2 as a base algorithm, Algorithm 3 (in the appendix) ensures Reg T = ˜ O ( √ β 1 T + β 2 + β 3 C P ) without knowing C P .

5 Gap-Dependent Refinements with Known C P

Finally, we discuss how to further improve our algorithm so that it adapts to easier environments and enjoys a better bound when the loss functions satisfy a certain gap condition, while still maintaining the O ( √ T + C P ) robustness guarantee. This result unfortunately requires the knowledge of C P because the black-box approach introduced in the last section leads to √ T regret overhead already. We leave the possibility of removing this limitation for future work.

More concretely, following prior work such as Jin and Luo [2020], we consider the following general condition: there exists a mapping π ⋆ : S → A , a gap function ∆ : S × A → (0 , L ] , and a constant C L ≥ 0 , such that for any policies π 1 , . . . , π T generated by the learner, we have

̸

̸

It has been shown that this condition subsumes the case when the loss functions are drawn from a fixed distribution (in which case π ⋆ is simply the optimal policy with respect to the loss mean and P , ∆ is the gap function with respect to the optimal Q -function, and C L = 0 ), or further corrupted by an adversary in an arbitrary manner subject to a budget of C L ; we refer the readers to Jin and Luo [2020] for detailed explanation. Our main result for this section is a novel algorithm (whose pseudocode is deferred to Appendix C due to space limit) that achieves the following best-of-both-world guarantee.

Theorem 5.1. Algorithm 4 (with δ = 1 / T 2 and γ t defined as in Definition 5.2) ensures

always, and simultaneously the following gap-dependent bound under Condition (9):

̸

̸

̸

̸

Aside from having larger dependence on parameters L , S , and A , Algorithm 4 maintains the same O ( √ T + C P ) regret as before, no matter how losses/transitions are generated; additionally, the √ T part can be significantly improved to O ( U + √ UC L ) (which can be of order only log 2 T when C L is small) under Condition (9). This result not only generalizes that of Jin et al. [2021], Dann et al. [2023] from C P = 0 to any C P , but in fact also improves their results by having smaller dependence on L in the definition of U . In the rest of this section, we describe the main ideas of our algorithm.

FTRL with Epoch Schedule Our algorithm follows a line of research originated from Wei and Luo [2018], Zimmert and Seldin [2019] for multi-armed bandits and uses FTRL (instead of OMD) together with a certain self-bounding analysis technique. Since FTRL does not deal with varying decision sets easily, similar to Jin et al. [2021], we restart FTRL from scratch at the beginning of each epoch i (recall the epoch schedule described in Section 2). More specifically, in an episode t that belongs to epoch i , we now compute ̂ q t as argmin q 〈 q, ∑ t -1 τ = t i ( ̂ ℓ τ -b τ ) 〉 + ϕ t ( q ) , where t i is the first episode of epoch i , ̂ ℓ t is the same loss estimator defined in Eq. (5), b t is the amortized bonus defined in Eq. (7) (except that τ = 1 there is also changed to τ = t i due to restarting), ϕ t is a time-varying regularizer to be specified later, and the set that q is optimized over is also a key element to be discussed next. As before, the learner then simply executes π t = π ̂ q t for this episode.

Optimistic Transition An important idea from Jin et al. [2021] is that if FTRL optimizes q over Ω( ¯ P i ) (occupancy measures with respect to a fixed transition ¯ P i ) instead of Ω( P i ) (occupancy measures with respect to a set of plausible transitions) as in UOB-REPS, then a critical loss-shifting technique can be applied in the analysis. However, the algorithm lacks 'optimism' when not using a confidence set, which motivates Jin et al. [2021] to instead incorporate optimism by subtracting

a bonus term BONUS from the loss estimator (not to be confused with the amortized bonus b t we propose in this work). Indeed, if we define the value function V ¯ P,π ( s ; ℓ ) as the expected loss one suffers when starting from s and following π in an MDP with transition ¯ P and loss ℓ , then they show that the BONUS term is such that V ¯ P i ,π ( s ; ℓ -BONUS ) ≤ V P,π ( s ; ℓ ) for any state s and any loss function ℓ , that is, the performance of any policy is never underestimated.

Instead of following the same idea, here, we propose a simpler and better way to incorporate optimism via what we call optimistic transitions . Specifically, for each epoch i , we simply define an optimistic transition function ˜ P i such that ˜ P i ( s ′ | s, a ) = max { 0 , ¯ P i ( s ′ | s, a ) -B i ( s, a, s ′ ) } (recall the confidence interval B i defined in Eq. (4)). Since this makes ∑ s ′ ˜ P i ( s ′ | s, a ) less than 1 , we allocate all the remaining probability to the terminal state s L (which breaks the layer structure but does not really affect anything). This is a form of optimism because reaching the terminate state earlier can only lead to smaller loss. More formally, under the high probability event E CON , we prove V ˜ P i ,π ( s ; ℓ ) ≤ V P,π ( s ; ℓ ) for any policy π , any state s , and any loss function ℓ (see Lemma C.8.3).

With such an optimistic transition, we simply perform FTRL over Ω( ˜ P i ) without adding any additional bonus term (other than b t ), making both the algorithm and the analysis much simpler than Jin et al. [2021]. Moreover, it can also be shown that V ¯ P i ,π ( s ; ℓ -BONUS ) ≤ V ˜ P i ,π ( s ; ℓ ) (see Lemma C.8.4), meaning that while both loss estimation schemes are optimistic, ours is tighter than that of Jin et al. [2021]. This eventually leads to the aforementioned improvement in the U definition.

̸

Time-Varying Log-Barrier Regularizers The final element to be specified in our algorithm is the time-varying regularizer ϕ t . Recall from discussions in Section 3 that using log-barrier as the regularizer is critical for bounding some stability terms in the presence of adversarial transitions. We thus consider the following log-barrier regularizer with an adaptive learning rate γ t : S × A → R + : ϕ t ( q ) = -∑ s = s L ∑ a ∈ A γ t ( s, a ) · log q ( s, a ) . The learning rate design requires combining the loss-shifting idea of [Jin et al., 2021] and the idea from [Ito, 2021], the latter of which is the first work to show that with adaptive learning rate tuning, the log-barrier regularizer leads to near-optimal best-of-both-world gaurantee for multi-armed bandits.

More specifically, following the same loss-shifting argument of Jin et al. [2021], we first observe that our FTRL update can be equivalently written as

where g τ ( s, a ) = Q ˜ P i ,π τ ( s, a ; ̂ ℓ τ ) -V ˜ P i ,π τ ( s ; ̂ ℓ τ ) for any state-action pair ( s, a ) ( Q is the standard Q -function; see Appendix C for formal definition). With this perspective, we follow the idea of Ito [2021] and propose the following learning rate schedule:

Definition 5.2. (Adaptive learning rate for log-barrier) For any t , if it is the starting episode of an epoch, we set γ t ( s, a ) = 256 L 2 | S | ; otherwise, we set γ t +1 ( s, a ) = γ t ( s, a ) + Dν t ( s,a ) 2 γ t ( s,a ) where

Such a learning rate schedule is critical for the analysis in obtaining a certain self-bounding quantity and eventually deriving the gap-dependent bound. This concludes the design of our algorithm; see Appendix C for more details.

6 Conclusions

In this work, we propose online RL algorithms that can handle both adversarial losses and adversarial transitions, with regret gracefully degrading in the degree of maliciousness of the adversary. Specifically, we achieve ˜ O ( √ T + C P ) regret where C P measures how adversarial the transition functions are, even when C P is unknown. Moreover, we show that further refinements of the algorithm not only maintain the same regret bound, but also simultaneously adapt to easier environments, with the caveat that C P must be known ahead of time. We leave how to further remove this restriction as a key future direction.

Acknowledgments and Disclosure of Funding

TJ and HL are supported by NSF Award IIS-1943607 and a Google Research Scholar award. CR acknowledges partial support by the Independent Research Fund Denmark, grant number 904000361B.

References

Yasin Abbasi Yadkori, Peter L Bartlett, Varun Kanade, Yevgeny Seldin, and Csaba Szepesvari. Online learning in markov decision processes with adversarially chosen transition probability distributions. In Advances in Neural Information Processing Systems (NeurIPS) , 2013.

Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2017.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 2008.

Yifang Chen, Simon Du, and Kevin Jamieson. Improved corruption robust algorithms for episodic reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML) , 2021.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Nonstationary reinforcement learning: The blessing of (more) optimism. Management Science , 2023.

Christoph Dann, Chen-Yu Wei, and Julian Zimmert. Best of both worlds policy optimization. arXiv preprint arXiv:2302.09408 , 2023.

Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathematics of Operations Research , 2009.

Dylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Eva Tardos. Learning in games: Robustness of fast convergence. Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Pratik Gajane, Ronald Ortner, and Peter Auer. A sliding-window algorithm for markov decision processes with arbitrarily changing rewards and transitions. arXiv preprint arXiv:1805.10066 , 2018.

Shinji Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2021.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Tiancheng Jin and Haipeng Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Tiancheng Jin, Longbo Huang, and Haipeng Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . PhD thesis, University College London, 2003.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Haipeng Luo, Mengxiao Zhang, Peng Zhao, and Zhi-Hua Zhou. Corralling a larger band of bandits: A case study on switching regret for linear bandits. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2022.

Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the Annual ACM SIGACT Symposium on Theory of Computing , 2018.

Thodoris Lykouris, Max Simchowitz, Aleksandrs Slivkins, and Wen Sun. Corruption robust exploration in episodic reinforcement learning. arXiv preprint arXiv:1911.08689 , 2019.

Gergely Neu, Andras Antos, András György, and Csaba Szepesvári. Online markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2010a.

Gergely Neu, András György, and Csaba Szepesvári. The online loop-free stochastic shortest-path problem. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2010b.

Aldo Pacchiano, Christoph Dann, and Claudio Gentile. Best of both worlds model selection. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial markov decision processes. In Proceedings of the International Conference on Machine Learning (ICML) , 2019a.

Aviv Rosenberg and Yishay Mansour. Online stochastic shortest path with bandit feedback and unknown transition function. Advances in Neural Information Processing Systems (NeurIPS) , 2019b.

Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Online learning in unknown markov games. In Proceedings of the International Conference on Machine Learning (ICML) , 2021.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2018.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the International Conference on Computational Learning Theory (COLT) , 2021.

Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML) , 2022.

Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corruption. In International Conference on Machine Learning , pages 12391-12401. PMLR, 2021.

Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems (NeurIPS) , 2013.

Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adversarial bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Appendix

Important Notations Throughout the appendix, we denote the transition associated with the occupancy measure ̂ q t by ̂ P t so that ̂ q t = q ̂ P t ,π t . Recall i ( t ) is the epoch index to which episode t belongs. Importantly, the notation m i ( s, a ) (and similarly m i ( s, a, a ′ ) ), which is a changing variable in the algorithms, denotes the initial value of this counter in all analysis, that is, the total number of visits to ( s, a ) prior to epoch i . Finally, for convenience, we define ̂ m i ( s, a ) = max { m i ( s, a ) , C P +log( ι ) } , and it can be verified that the confidence interval, defined in Eq. (4) as

can be equivalently written as

̸

since whenever ̂ m i ( s, a ) = m i ( s, a ) , the two definitions both lead to a value of 1 .

A Omitted Details for Section 3

In this section, we provide more details of the modified UOB-REPS algorithm as shown in Algorithm 1. In the algorithm, the occupancy measure for each episode t is computed as

where η > 0 is the learning rate and D ϕ ( q, q ′ ) is the Bregman divergence defined as

The Bregman divergence is induced by the log-barrier regularizer ϕ given as:

We note that the loss estimator is constructed based upon upper occupancy bound u t , which can be efficiently computed by COMP-UOB [Jin et al., 2020].

In the rest of this section, we prove Theorem 3.2 which shows that the expected regret is bounded by

Our analysis starts from a regret decomposition similar to Jin et al. [2020], but with the amortized bonuses taken into account:

where the last term can be directly bounded by O ( LC P ) using Corollary D.3.7, and ERROR, BIAS 1 , REG, and BIAS 2 are analyzed in Appendix A.2, Appendix A.3, Appendix A.4 and Appendix A.5 respectively.

A.1 Equivalent Definition of Amortized Bonuses

For ease of exposition, we show an alternative definition of the amortized bonus b t ( s ) , which is equivalent to Eq. (7) but is useful for our following analysis. Recall from Eq. (7) that

In Eq. (13), the value of b t ( s ) relies on the cumulative sum of I {⌈ log 2 u τ ( s ) ⌉ = ⌈ log 2 u t ( s ) ⌉} and whether the cumulative sum exceeds the threshold C P 2 L . To reconstruct this, we define y t ( s ) as the unique integer value j = -⌈ log 2 u t ( s ) ⌉ such that u t ( s ) ∈ (2 -j -1 , 2 -j ] and define z j t ( s ) as the total number of times from episode τ = 1 to τ = t where y τ ( s ) = j holds. With these definitions, the condition under which b t ( s ) = 4 L u t ( s ) can be represented by ∑ j I { y t ( s ) = j } I { z j t ( s ) ≤ C P / 2 L } where the summation is over all integers that -⌈ log 2 u t ( s ) ⌉ can take. Notice that since u t ( s ) ≥ 1 | S | T holds for all s and t (see Lemma D.2.8), the largest possible integer value we need to consider is ⌈ log 2 ( | S | T ) ⌉ . Therefore, b t ( s ) can be equivalently defined as:

A.2 Bounding ERROR

Lemma A.2.1 (Bound of ERROR) . Algorithm 1 ensures

Proof. For this proof, we consider two cases, whether the event E CON ∧E EST holds or not, where E CON , defined above Eq. (4), and E EST , defined in Proposition D.5.2, are both high probability events.

Suppose that E CON ∧ E EST holds. We have

where the first step follows from the fact that ℓ t ( s, a ) ∈ [0 , 1] for any ( s, a ) ; the third step applies Lemma D.3.3; the fourth step rearranges the summation and uses the fact that ∑ s,a q ̂ P t ,π t ( s, a | w ) ≤ L ; and in the sixth step we define C P t,h = max s ∈ S h ,a ∈ A ∥ P t ( ·| s, a ) -P ( ·| s, a ) ∥ 1 , so that C P = ∑ T t =1 ∑ L -1 h =0 C P t,h . We continue to bound the first term above as:

where the first step uses the triangle inequality; the second step applies Corollary D.2.6 to bound two norm terms based on the fact that ̂ P t , ¯ P i ( t ) ∈ P i ( t ) ; the third step follows the definition of E EST from Proposition D.5.2; the fourth step uses the AM-GM inequality. Putting the result of Eq. (16) into Eq. (15) yields the first three terms of the claimed bound.

Now suppose that E CON ∧ E EST does not hold. We trivially bound ∑ T t =1 〈 q P t ,π t -q ̂ P t ,π t , ℓ t 〉 by LT . As the probability that this case occurs is at most O ( δ ) , this case contributes to at most O ( δLT ) regret. Finally, combining these two cases and applying Lemma D.1.1 finishes the proof.

A.3 Bounding BIAS 1

Lemma A.3.1 (Bound of BIAS 1 ) . Algorithm 1 ensures

Proof. We can rewrite BIAS 1 as

where the second step uses the law of total expectation and the fact that q ̂ P t ,π t is deterministic given all the history up to t . For any state-action pair ( s, a ) , we have

Then, we can further rewrite and bound BIAS 1 as:

where the third step follows from the fact that q ̂ P t ,π t ( s, a ) ≤ u t ( s, a ) according to the definition of the upper occupancy measure.

Similar to the proof of Lemma A.2.1, we first consider the case when the high probability event E CON ∧ E EST holds:

where the first step uses the triangle inequality; the second step uses Corollary D.3.6; the third step uses Lemma D.3.8; the last step follows from the fact that ∑ s ∈ S q P,π t ( s | w ) ≤ L .

Taking the summation over all episodes yields the following:

where the second step uses the Cauchy-Schwartz inequality: ∑ w ∈ S k +1 √ P ( w | u, v ) ≤ √ | S k +1 | and also applies Lemma D.5.3; the fourth step bounds ̂ m i ( t ) ( u, v ) ≥ log( ι ) for any ( u, v ) and | S k +1 | ≤ | S | ; the fifth applies Proposition D.5.2, and Corollary D.3.7 to bound ∑ T t =1 ∑ L -1 k =0 ∑ ( u,v ) ∈ S k × A ∣ ∣ q P,π t ( u, v ) -q P t ,π t ( u, v ) ∣ ∣ by O ( LC P ) ; the last step uses the fact that √ xy ≤ x + y for any x, y ≥ 0 , and thus we have ∑ L -1 k =0 √ | S k | | S k +1 | ≤ 2 ∑ L -1 k =0 | S k | = 2 | S | .

Finally, using a similar argument in the proof of Lemma A.2.1 to bound the case that E CON ∧ E EST does not hold, we complete the proof.

A.4 Bounding REG

Lemma A.4.1 (Bound of REG) . If learning rate η satisfies η ∈ (0 , 1 8 L ] , then, Algorithm 1 ensures

Proof. We consider a specific transition P 0 defined in Lee et al. [2020, Lemma C.4] and occupancy measure u such that

where π a is a policy such that action a is selected at every state.

By direct calculation, we have

where the second step uses the definition ̂ q 1 ( s, a, s ′ ) = 1 | S k || A || S k +1 | for k = 0 , · · · , L -1 and the fourth step lower-bounds u ( s, a, s ′ ) ≥ 1 T 3 | S | 2 | A | from [Lee et al., 2020, Lemma C.10], thereby u ( s, a, s ′ ) ≥ 1 ι 3 , and upper-bounds ̂ q 1 ( s, a, s ′ ) ≤ 1 .

According to [Lee et al., 2020, Lemma C.4], we have q P 0 ,π a ∈ ∩ i Ω( P i ) . Therefore, u is a convex combination of points in that convex set, and we can use Lemma A.4.2 (included after this proof) to show

On the one hand, we bound the first summation in Eq. (17) as

since q ̂ P t ,π t ( s, a ) ≤ u t ( s, a ) by definition of the upper occupancy bound.

On the other hand, we bound the second summation in Eq. (17) as

where the first step uses the facts that b t ( s ) 2 ≤ 4 Lb t ( s ) and ∑ a q ̂ P t ,π t ( s, a ) 2 ≤ q ̂ P t ,π t ( s ) , and the last step applies Lemma 3.1.

Putting these inequalities together concludes the proof.

Lemma A.4.2. With η ∈ (0 , 1 8 L ] , Algorithm 1 ensures

for any u ∈ ∩ i Ω( P i ) .

Proof. To use the standard analysis of OMD with log-barrier (e.g.,see [Agarwal et al., 2017, Lemma 12]), we need to ensure that η ̂ q t ( s, a, s ′ ) ( ̂ ℓ t ( s, a ) -b t ( s ) ) ≥ -1 2 , since log (1 + x ) ≥ x -x 2 holds for any x ≥ -1 2 . Clearly, by choosing η ∈ (0 , 1 8 L ] , we have

where the first step follows from the fact that ̂ ℓ t ( s, a ) ≥ 0 for all t, s, a ; the second step follows from the definition of amortized bonus in Eq. (7); the third step bounds ̂ q t ( s, a, s ′ ) ≤ u t ( s ) .

Now, we are ready to apply the standard analysis to show that

where the second step uses the fact that ̂ q t ( s, a, s ′ ) = ̂ q t ( s, a ) · ̂ P t ( s ′ | s, a ) (see [Jin et al., 2020, Lemma 1] for more details); the third step follows from the fact that the Bregman divergence is non-negative; the fourth step follows from the fact ∑ s ′ ̂ P t ( s ′ | s, a ) = 1 ; the last step uses the fact that ( x -y ) 2 ≤ 2 ( x 2 + y 2 ) for any x, y ∈ R .

A.5 Bounding BIAS 2

Lemma A.5.1 (Bound of BIAS 2 ) . Algorithm 1 ensures

Proof. We first rewrite BIAS 2 as

where (II) is bounded by O ( | S | C P log T ) by Lemma 3.1 (whose proof is included after this proof). Now, we show that (I) is bounded by O ( δLT ) . Suppose that E CON holds. We then have

where the fourth step applies Corollary D.3.6 to bound ∣ ∣ q P t ,π t ( s ) -q P,π t ( s ) ∣ ∣ ≤ C P t , and in the last step, the second term (in the fifth step) is bounded by 0 under E CON .

Since q P, ˚ π ( s, a ) is fixed over all episodes, we apply Lemma 3.1 to show

For the case that E CON does not occur, we bound the expected regret by O ( δLT ) . Combining two cases via Lemma D.1.1, we conclude the proof.

Lemma A.5.2 (Restatement of Lemma 3.1) . The amortized bonus defined in Eq. (7) satisfies ∑ T t =1 C P t u t ( s ) ≤ ∑ T t =1 b t ( s ) and ∑ T t =1 ̂ q t ( s ) b t ( s ) = O ( C P log T ) for any s .

Proof. In the following proof, we use the equivalent definition of b t ( s ) given in Eq. (14). We first show ∑ T t =1 C P t u t ( s ) ≤ ∑ T t =1 b t ( s ) . On the one hand, we have

where the second step uses the construction of the bin, i.e., if u t ( s ) falls into a bin (2 -j -1 , 2 -j ] , then, it is lower-bounded by 2 -j -1 ; the fourth step follows the facts that C P t ≤ 2 L and ∑ T t =1 C P t ≤ C P . On the other hand, one can show

where the second step uses the construction of the bin, i.e., if u t ( s ) falls into a bin (2 -j -1 , 2 -j ] , then, it is upper-bounded by 2 -j , and the third step follows the definitions of y t ( s ) and z j t ( s ) .

Combining Eq. (20) and Eq. (21), we complete the proof of ∑ T t =1 C P t u t ( s ) ≤ ∑ T t =1 b t ( s ) .

For the proof of ∑ T t =1 ̂ q t ( s ) b t ( s ) = O ( | S | C P log T ) , one can show

where the first inequality uses the fact q ̂ P t ,π t ( s ) ≤ u t ( s ) , and the last equality uses | S | ≤ T .

A.6 Proof of Theorem 3.2

For REG, we choose η = min {√ | S | 2 | A | log( ι ) LT , 1 8 L } . First consider the case η = 1 8 L :

̸

̸

where the second step uses η ≤ 1 8 L , and the third step applies choice of η in the case of η = 1 8 L . For the case of η = 1 8 L , we have T ≤ 64 L | S | 2 | A | ln( ι ) and show

Finally, choosing δ = 1 T and putting the bound above together with ERROR, BIAS 1 , and BIAS 2 , we complete the proof of Theorem 3.2.

B Omitted Details for Section 4

B.1 Bottom Layer Reduction: STABILISE

Proof of Theorem 4.3. Define indicators

Now we consider the regret of ALG j . Notice that ALG j makes an update only when g t,j h t,j = 1 . By the guarantee of the base algorithm (Definition 4.1), we have

We first bound the last term: Notice that E [ h t,j | g t,j ] = 2 -j -1 g t,j by Algorithm 2. Therefore,

By Freedman's inequality, with probability at least 1 -1 T 2 ,

which gives

with probability at least 1 -1 T 2 using Eq. (23). Therefore, the last term in Eq. (22) is bounded by 1 T 2 LT ≤ L T .

Next, we deal with other terms in Eq. (22). Again, by E [ h t,j | g t,j ] = 2 -j -1 g t,j , Eq. (22) implies

which implies after rearranging:

.

(using that when g t,j = 1 , 1 2 -j -1 ≤ 2 w t , and the definition of θ j )

Now, summing this inequality over all j ∈ { 0 , 1 , . . . , ⌈ log 2 T ⌉} , we get

where N ≤ O (log T ) is the number of ALG j 's that has been executed at least once.

On the other hand,

Combining the two parts and using the assumption β 2 ≥ L finishes the proof.

B.2 Top Layer Reduction: Corral

In this subsection, we use a base algorithm that satisfies Definition 4.2 to construct an algorithm with √ T + C P regret under unknown C P . The idea is to run multiple base algorithms, each with a different hypothesis on C P ; on top of them, run another multi-armed bandit algorithm to adaptively choose among them. The goal is to let the top-level bandit algorithm perform almost as well as the best base algorithm. This is the Corral idea outlined in Agarwal et al. [2017], Foster et al. [2020], Luo et al. [2022], and the algorithm is presented in Algorithm 3.

The top-level bandit algorithm is an FTRL with log-barrier regularizer. We first state the standard regret bound of FTRL under log-barrier regularizer, whose proof can be found in, e.g., Theorem 7 of Wei and Luo [2018].

Lemma B.2.1. The FTRL algorithm over a convex subset Ω of the ( M -1) -dimensional simplex ∆( M ) :

as long as ηw t,i | ℓ t,i | ≤ 1 2 for all t, i .

Proof of Theorem 4.4. The Corral algorithm is essential an FTRL with log-barrier regularizer. To apply Lemma B.2.1, we first verify the condition ηw t,i | ℓ t,i | ≤ 1 2 where ℓ t,i = ˆ c t,i -r t,i . By our choice of η ,

ensures for all u ∈ Ω ,

Algorithm 3 (A Variant of) Corral

Initialize : a log-barrier algorithm with each arm being an instance of any base algorithm satisfying Definition 4.2. The hypothesis on C P is set to 2 i for arm i ( i = 1 , 2 , . . . , M ≜ ⌈ log 2 T ⌉ ). Initialize : ρ 0 ,i = M, ∀ i for

t

= 1

Let

For all i , send w t,i to instance i .

Draw i t ∼ w t .

Execute the π t output by instance i t

Receive the loss c t,i t for policy π t (whose expectation is ℓ t ( π t ) ) and send it to instance i t . Define for all i :

The right-hand side of the last equality is non-zero only when ρ t,i > ρ t -1 ,i , implying that ρ t,i = 1 w t,i . Therefore, we further bound it by

which can be combined to get the desired property ηw t,i | ˆ c t,i -r t,i | ≤ 1 2 .

Hence, by the regret guarantee of log-barrier FTRL (Lemma B.2.1), we have

where i ⋆ is the smallest i such that 2 i upper bounds the true corruption amount C P .

,

2

, . . . , T

do

Bounding stability-term :

where

and

Bounding bonus-term :

Combining the two terms and using η = Θ ( 1 √ β 1 T + β 2 ) , M = Θ(log T ) , we get

On the other hand, by Definition 4.2 and that C P ∈ [2 i ⋆ -1 , 2 i ⋆ ] , we have

Combining Eq. (25) and Eq. (26), we get

which finishes the proof.

Algorithm 4 Algorithm with Optimistic Transition Achieving Gap-Dependent Bounds (Known C P )

Input: confidence parameter δ ∈ (0 , 1) . Initialize: ∀ ( s, a ) , learning rate γ 1 ( s, a ) = 256 L 2 | S | ; epoch index i = 1 and epoch starting time t i = 1 ; ∀ ( s, a, s ′ ) , set counters m 1 ( s, a ) = m 1 ( s, a, s ′ ) = m 0 ( s, a ) = m 0 ( s, a, s ′ ) = 0 ; empirical transition ¯ P 1 and confidence width B 1 based on Eq. (2); optimistic transition ˜ P i by Definition C.1.1. for t = 1 , . . . , T do

Let ϕ t be defined in Eq. (28) and compute

Compute policy π from q such that π ( a | s ) ∝ q ( s, a ) .

t t,k t,k Construct loss estimator ̂ ℓ t as defined in Eq. (5). Compute amortized bonus b t based on Eq. (29). Compute learning rate γ t +1 according to Definition 5.2. Increment counters: for each k < L , m i ( s t,k , a t,k , s t,k +1 ) + ← 1 , m i ( s t,k , a t,k ) + ← 1 .

t ̂ t t ̂ t Execute policy π and obtain trajectory ( s , a ) for k = 0 , . . . , L -1 .

C Omitted Details for Section 5

In this section, we consider a variant of the algorithm proposed in Jin et al. [2021], which instead ensures exploration via optimistic transitions and also switch the regularizer from Tsallis entropy to log-barrier. We present the pseudocode in Algorithm 4, and show that it automatically adapts to easier environments with improved gap-dependent regret bounds.

Remark C.1. Throughout the analysis, we denote by Q P,π ( s, a ; r ) the state-action value of stateaction pair ( s, a ) with respect to transition function P ( P could be an optimistic transition function), policy π and loss function r ; similarly, we denote by V P,π ( s ; r ) the corresponding state-value function.

Specifically, the state value function V P,π ( s ; r ) is computed in a backward manner from layer L to layer 0 as following:

̸

Similarly, the state-action value function Q P,π ( s, a ; r ) is calculated in the same manner:

̸

Clearly, V P,π ( s 0 ; r ) , the state value of s 0 , is equal to

̸

This equality can be further extended to the other state u as:

̸

where q P,π ( s, a | u ) denotes the probability of arriving at ( s, a ) starting at state u under policy π and transition function P .

C.1 Description of the Algorithm

The construction of this algorithm follows similar ideas to the work of Jin et al. [2021], while also including several key differences. One of them is that we rely on the log-barrier regularizer, defined with a positive learning rate γ t ( s, a ) as

The choice of log-barrier is important for adversarial transitions as discussed in Section 3, while the adaptive choice of learning rate is important for adapting to easy environments. The formal definition of the learning rate is given in Definition 5.2, and further details and properties of the learning rate are provided in Section C.7.

As the algorithm runs a new instance of FTRL for each epoch i , we modify the definition of the amortized bonus b t accordingly,

The bonus b t ( s ) in Eq. (29) is defined based on each epoch i , in the sense that ∑ t τ =1 I {⌈ log 2 u τ ( s ) ⌉ = ⌈ log 2 u t ( s ) ⌉} counts, among all previous rounds τ = t i , . . . , t in epoch i , the number of times that the value of u τ ( s ) falls into the same bin as u t ( s ) .

Again, for the ease of analysis, in the analysis we use the equivalent definition of the bonus b t ( s ) defined in Eq. (14), except that the counter z j t ( s ) now will be reset to zero at the beginning of each epoch i .

Next, we formally define the optimistic transition as follows.

Definition C.1.1 (Optimistic Transition) . For epoch i , the optimistic transition ˜ P i : S × A × S → [0 , 1] is defined as:

where ¯ P i is the empirical transition defined in Eq. (2) .

Note that the optimistic transition ˜ P i ( ·| s, a ) is a valid distribution as we have

We summarize the properties of the optimistic transition functions in Appendix C.8.

C.2 Self-Bounding Properties of the Regret

In order to achieve best-of-both-worlds guarantees, our goal is to bound Reg T ( π ⋆ ) in terms of two self-bounding quantities for some x > 0 (plus other minor terms):

̸

̸

̸

̸

These two quantities enjoy a certain self-bounding property which is critical to achieve the gapdependent bound under Condition (9), as they can be related back to the regret against policy π ⋆ itself. To see this, we first show the following implication of Condition (9).

Lemma C.2.1. Under Condition (9) , the following holds.

̸

̸

Proof. From the definition of Reg T ( π ⋆ ) , we first note that:

̸

̸

where the third step applies Corollary D.3.7 and the last step uses the Condition (9). We continue to bound the first term above as:

̸

̸

̸

̸

where the last step uses ∆( s, a ) ∈ (0 , L ] and Corollary D.3.6.

̸

We are now ready to show the following important self-bounding properties (recall ∆ MIN = min s = s L ,a = π ⋆ ( s ) ∆( s, a ) ).

̸

Lemma C.2.2 (Self-Bounding Quantities) . Under Condition (31), we have for any z > 0 :

̸

̸

Besides, it always holds that S 1 ( x ) ≤ x · LT and S 2 ( x ) ≤ √ x · L | S || A | T .

Proof. For any z > 0 , we have

̸

̸

̸

̸

̸

̸

̸

̸

̸

where the second step follows from the AM-GM inequality: 2 √ xy ≤ x + y for any x, y ≥ 0 , and the last step follows from Condition (31).

By similar arguments, we have S 2 ( x ) bounded for any z > 0 as:

̸

̸

̸

Finally, by direct calculation, we can show that

̸

̸

̸

according to the fact that ∑ s = s L ∑ a ∈ A q P t ,π t ( s, a ) ≤ L . On the other hand, S 2 ( x ) is bounded as

̸

̸

with the help of the Cauchy-Schwarz inequality in the first step.

Finally, we show that Algorithm 4 achieves the following adaptive regret bound, which directly leads to the best-of-both-worlds guarantee in Theorem 5.1.

Lemma C.2.3. Algorithm 4 with δ = 1 / T 2 and learning rate defined in Definition 5.2 ensures that, for any mapping π ⋆ : S → A , the regret Reg T ( π ⋆ ) is bounded by O (( C P +1 ) L 2 | S | 4 | A | 2 log 2 ( ι ) ) plus

The proof of this result is detailed in Section C.3. We emphasize that this bound holds for any mapping π ⋆ , and is not limited to that policy in Eq. (9). This is important for proving the robustness result when losses are arbitrary, as shown in the following proof of Theorem 5.1.

Proof of Theorem 5.1. When losses are arbitrary, we simply select π ⋆ = ˚ π where ˚ π is one of the optimal deterministic policies in hindsight and obtain the following bound of Reg T (˚ π ) :

where the first step follows from Lemma C.2.3 and the second step follows from Lemma C.2.2.

Next, suppose that Condition (9) holds. We set π to write the regret against π ⋆ as:

⋆ as defined in the condition and use Lemma C.2.3

̸

̸

̸

̸

where µ > 0 is an absolute constant and ξ = O ( L 2 | S | 4 | A | 2 log 2 ( ι ) ) .

For any z > 0 , according to Lemma C.2.2 (where we set the z there as z / 2 µ ), we have:

̸

̸

̸

Let x = 1 -z z and U = L 3 | S | 2 | A | log 2 ( ι ) ∆ MIN + ∑ s = s L ∑ a = π ⋆ ( s ) L 2 | S || A | log 2 ( ι ) ∆( s,a ) . Rearranging the above inequality leads to

̸

Picking the optimal x to minimize the upper bound of Reg T ( π ⋆ ) yields

where the second step uses √ x + y ≤ √ x + √ y for any x, y ∈ R ≥ 0 and the last step uses 2 √ xy ≤ x + y for any x, y ≥ 0 .

C.3 Regret Decomposition of Reg T ( π ⋆ ) and Proof of Lemma C.2.3

In the following sections, we will first decompose Reg T ( π ⋆ ) for any mapping π ⋆ : S → A into several parts, and then bound each part separately from Section C.4 to Section C.7, in order to prove Lemma C.2.3.

For any mapping π ⋆ : S → A , we start from the following decomposition of Reg T ( π ⋆ ) as

where ̂ P t = ˜ P i ( t ) denotes the optimistic transition for episode t for simplicity (which is consistent with earlier definition such that ̂ q t = q ̂ P t ,π t ). Here, ESTREG is the estimated regret controlled by FTRL, while ERROR 1 and ERROR 2 are estimation errors incurred on the selected policies { π t } T t =1 , and that on the comparator policy π ⋆ respectively.

In order to achieve the gap-dependent bound under Condition (9), we consider these two estimation error terms ERROR 1 and ERROR 2 together as:

where E t [ · ] denotes the conditional expectation given the history prior to episode t . To better analyze the conditional expectation of the loss estimators, we define α t , β t : S × A → R as:

which ensures that E t [ ̂ ℓ t ( s, a ) ] = α t ( s, a ) + β t ( s, a ) for any state-action pair ( s, a ) .

With the help of α t , β t , we have

which helps us further rewrite Eq. (33) as

Based on this decomposition of ERROR 1 + ERROR 2 , we then bound each parts respectively in the following lemmas.

Lemma C.3.1. For any δ ∈ (0 , 1) and any policy sequence { π t } T t =1 , Algorithm 4 ensures that

Lemma C.3.2. For any δ ∈ (0 , 1) and any mapping π ⋆ : S → A , Algorithm 4 ensures that

Lemma C.3.3. For any δ ∈ (0 , 1) and any mapping π ⋆ : S → A , Algorithm 4 ensures that

Lemma C.3.4. With the learning rates { γ t } T t =1 defined in Definition 5.2, Algorithm 4 ensures that for any δ ∈ (0 , 1) and any mapping π ⋆ that,

Proof of Lemma C.2.3. According to the previous discussion, we have the regret against any mapping π ⋆ : S → A decomposed as:

where the first term is mainly caused by the difference between { P t } T t =1 and P , which is unavoidable and can be bounded by O ( C P L ) as shown in Lemma C.3.1; the second term is the extra cost of using the amortized losses b t to handle the biases of loss estimators, which is controlled by ˜ O ( C P ) as shown in Lemma C.3.2; the third term measures the estimation error related to the optimistic transitions { ̂ P t } T t =1 , which can be bounded by some self-bounding quantities in Lemma C.3.2; the final term is the estimated regret calculated with respect to the optimistic transitions { ̂ P t } T t =1 , which is controlled by FTRL as shown in Lemma C.3.4. Putting all these bounds together finishes the proof.

C.4 Proof of Lemma C.3.1

The result is immediate by directly applying Corollary D.3.7.

C.5 Proof of Lemma C.3.2

For this proof, we bound E [ ∑ T t =1 〈 q ̂ P t ,π t , b t -β t 〉] and E [ ∑ T t =1 〈 q ̂ P t ,π ⋆ , β t -b t 〉] separately.

where the second step bounds ℓ t ( s, a ) ≤ 1 ; the third step follows the fact that q ̂ P t ,π t ( s, a ) ≤ u t ( s, a ) for all ( s, a ) . Let E i be a set of episodes that belong to epoch i and let N be the total number of epochs through T episodes. Then, we turn to bound

where the last step repeats the same argument of Lemma 3.1 for every epoch and the number of epochs is at most O ( | S || A | log T ) according to [Jin et al., 2021, Lemma D.3.12].

Bounding E [ ∑ T t =1 〈 q ̂ P t ,π ⋆ , β t -b t 〉] . For this term, we show that for any given epoch i , ∑ t ∈ E i 〈 q ̂ P t ,π ⋆ , β t -b t 〉 ≤ 0 which yields ∑ T t =1 〈 q ̂ P t ,π ⋆ , β t -b t 〉 ≤ 0 . To this end, we first consider a fixed epoch i and upper-bound

where the second step uses Corollary D.3.6 to bound ( q P t ,π t ( s ) -q P,π t ( s ) ) ≤ C P t .

For any epoch i and episode t ∈ E i , we have ̂ P t = ˜ P i ( t ) , which gives that

where the last step applies Lemma 3.1 for every epoch i . Thus, E [ ∑ T t =1 〈 q ̂ P t ,π ⋆ , β t -b t 〉] ≤ 0 holds.

C.6 Proof of Lemma C.3.3

We introduce the following lemma to evaluate the estimated performance via the true occupancy measure, which helps us analyze the the estimation error.

Lemma C.6.1. For any transition function pair ( P, ̂ P ) ( P and ̂ P can be optimistic transition), policy π , and loss function ℓ , it holds that 〈 q P,π , ℓ 〉 = 〈 q ̂ P,π , Z P, ̂ P,π ℓ 〉 , where Z P, ̂ P,π ℓ is defined as

for all state-action pairs ( s, a ) .

Proof. By direct calculation, we have:

where the second to last step follows from recursively repeating the first three steps. The proof is completed by noticing 〈 q P,π , ℓ 〉 = V P,π ( s 0 , ℓ ) .

According to Lemma C.6.1, we rewrite 〈 q ̂ P t ,π t , α t 〉 and 〈 q ̂ P t ,π ⋆ , α t 〉 with q P,π t and q P,π ⋆ for any policy π ⋆ as

Therefore, we can further decompose as

̸

̸

We will bound these terms with some self-bounding quantities in Lemma C.6.2, Lemma C.6.3 and Lemma C.6.4 respectively. In these proofs, we will follow the idea of Lemma D.1.1 to first bound these terms conditioning on the events E EST and E CON defined in Proposition D.5.2 and Eq. (4), while ensuring that these terms are always bounded by O ( | S | 2 | A | T 2 ) in the worst case.

C.6.1 Bounding Term 1

Lemma C.6.2. For any δ ∈ (0 , 1) and any mapping π ⋆ : S → A , Algorithm 4 ensures that E [ TERM 1 ( π ⋆ )] is bounded by

Proof. Clearly, we have α t ( s, a ) ≤ | S | T for every state-action pair ( s, a ) , due to the fact that u t ( s ) ≥ 1 / | S | T (Lemma D.2.8). Therefore, we have TERM 1 ( π ⋆ ) ≤ | S | 2 | A | T 2 holds always. In the remaining, our main goal is to bound TERM 1 ( π ⋆ ) conditioning on E CON ∧ E EST .

For any state-action pair ( s, a ) ∈ S × A and episode t , we have

conditioning on the event E CON .

Therefore, under event E CON ∧ E EST, TERM 1 ( π ⋆ ) is bounded by

̸

̸

̸

̸

where the first step follows from the non-negativity of ℓ t ( s, a ) -α t ( s, a ) ; the third step applies Lemma D.5.4 with G = 1 as ∑ a [ q P,π t ( s, a ) -q P,π ⋆ ( s, a ) ] + ≤ q P,π t ( s ) ≤ u t ( s ) ; the fifth step follows from the fact that ∑ s = s L ∑ a ∈ A [ q P,π t ( s, a ) -q P,π ⋆ ( s, a ) ] + ≤ 2 L ∑ s = s L ∑ a = π ⋆ ( s ) q P,π t ( s, a ) according to Corollary D.3.5; the last step applies Corollary D.3.6. Applying Lemma D.1.1 with event E CON ∧ E EST yields that

̸

̸

̸

̸

C.6.2 Bounding Term 2

Lemma C.6.3. For any δ ∈ (0 , 1) and any mapping π ⋆ : S → A , Algorithm 4 ensures that E [ TERM 2 ( π ⋆ )] = O ( S 1 ( L | S | 2 | A | log 2 ( ι ) ) + δL | S | 2 | A | T + L | S | 2 | A | ( C P +log( ι ) ) log ( T ) ) .

Proof. Suppose that E CON ∧ E EST occurs. We have

Therefore, we can show that

̸

̸

̸

̸

̸

̸

̸

̸

where the last step follows from the definition of optimistic transition and the fact that α t ( s, a ) ≤ 1 . By direct calculation, we have TERM 2 ( π ⋆ ) bounded by

̸

where the last step follows from Lemma D.5.3.

Moreover, we have

̸

̸

̸

̸

where the second step uses the Cauchy-Schwarz inequality; the third step applies Lemma D.5.3; the fifth step follows from Corollary D.3.5; the last step uses Corollary D.3.6 and the fact that √ x + y ≤ √ x + √ y for any x, y ≥ 0 .

Finally, applying Lemma D.1.1 finishes the proof.

C.6.3 Bounding Term 3

Lemma C.6.4. For any δ ∈ (0 , 1) and the policy π ⋆ , Algorithm 4 ensures that

Proof. Suppose that E CON ∧ E EST occurs. We first have:

̸

̸

̸

̸

̸

̸

̸

̸

̸

where the last step follows from Lemma D.2.7 and the fact that V ̂ P t ,π ( s ′ ; α t ) ≤ L for any π .

By applying Lemma D.5.3, we have the second term bounded by O (( C P +log( ι ) ) L 2 | S | 4 | A | log ( ι ) ) . On the other hand, for the first term, we can bound ∣ ∣ ∣ V ̂ P t ,π ⋆ ( s ′ ; α t ) -V ̂ P t ,π t ( s ′ ; α t ) ∣ ∣ ∣ as

̸

̸

where the first step follows from Lemma D.3.2; the second step follows from the fact that Q ̂ P t ,π ⋆ ( u, v ; α t ) ∈ [0 , L ] ; the third step uses the same reasoning as the proof of Corollary D.3.5; and the last step uses Corollary C.8.2.

Finally, we consider the following term

̸

̸

̸

̸

̸

̸

where D = 1 log( ι ) and

Importantly, both Q ˜ P i ( t ) ,π t ( s, a ; ̂ ℓ t ) and V ˜ P i ( t ) ,π t ( s ; ̂ ℓ t ) can be computed, which ensures that the learning rate is properly defined.

C.7.1 Properties of the Learning Rate

In this section, we prove key properties of ν t ( s, a ) and of γ t ( s, a ) . We first present some results in Lemma C.7.2 that are useful to bound ν t ( s, a ) , and then use these results to bound γ t ( s, a ) .

Lemma C.7.2. For any state-action pair ( s, a ) and any episode t , it holds that

which ensures q ̂ P t ,π t ( s, a ) ( Q ̂ P t ,π t ( s, a ; ̂ ℓ t ) -V ̂ P t ,π t ( s ; ̂ ℓ t ) ) ∈ [ -L, L ] . Suppose that the highprobability event E CON holds. Then, it further holds for all state-action pair ( s, a ) that

where ̂ P t here is the optimistic transition ˜ P i ( t ) defined in Definition C.1.1.

̸

̸

̸

where the second step applies Cauchy-Schwarz inequality; the third step follows from the fact that ∑ s ∈ S k ∑ a ∈ A ∑ s ′ ∈ S k ( s )+1 q P,π t ( s, a ) P ( s ′ | s, a ) q P,π t ( u, v | s ′ ) = q P,π t ( u, v ) ; the fourth step follows from Lemma D.5.3 conditioning on the event E EST ; the fifth step uses the fact that √ x + y ≤ √ x + √ y for x, y ≥ 0 ; the last step follows from Corollary D.3.6.

C.7 Proof of Lemma C.3.4

In this section we bound ESTREG using a learning rate that depends on t and ( s, a ) , which is crucial to obtain a self-bounding quantity. Another key observation is that the estimated transition function is constant within each epoch, so we first bound ESTREG within one epoch before summing them.

Recall that E i is a set of episodes that belong to epoch i and N is the total number of epochs through T episodes. By using the fact that ̂ P t = ˜ P i for episode t belonging to epoch i , we make the following decomposition ESTREG ( π ⋆ )

This learning rate is defined in Definition 5.2 and restated below.

Definition C.7.1. For any t , if it is the starting episode of an epoch, we set γ t ( s, a ) = 256 L 2 | S | ; otherwise, we set

̸

̸

Proof. We first verify q ̂ P t ,π t ( s, a ) Q ̂ P t ,π t ( s, a ; ̂ ℓ t ) ≤ L

where the second step follows from the definition of loss estimator, the fourth step follows from q ̂ P t ,π t ( s, a ) q ̂ P t ,π t ( u, v | s, a ) ≤ q ̂ P t ,π t ( u, v ) ≤ u t ( u, v ) , and the last step uses the fact that ∑ u ∈ S h ∑ v ∈ A I t ( u, v ) = 1 . Following the same idea, we can show that q ̂ P t ,π t ( s ) V ̂ P t ,π t ( s ; ̂ ℓ t ) ≤ L as well.

̸

̸

where the second step follows from the fact that ( x + y ) 2 ≤ 2 ( x 2 + y 2 ) for any x, y ∈ R . By direct calculation, we have

̸

̸

where the second step uses Cauchy-Schwarz inequality; the third step uses the fact that ̂ ℓ t ( s, a ) · ̂ ℓ t ( s ′ , a ′ ) = 0 for any ( s, a ) = ( s ′ , a ′ ) ; the fourth step takes the conditional expectation of ̂ ℓ t ( x, y ) 2 ; the sixth step follows from the fact that q ̂ P t ,π t ( x, y | s, a ) q ̂ P t ,π t ( s, a ) ≤ q ̂ P t ,π t ( x, y ) ≤ u t ( x, y ) according to the definition of upper occupancy bound; the last step follows from Corollary D.3.6. Similarly, for the second term in Eq. (36), we have

̸

̸

̸

where the first three steps are following the same idea of previous analysis; the fourth step uses the fact that ∑ b = a q ̂ P t ,π t ( s ) π t ( b | s ) q ̂ P t ,π t ( x, y | s, b ) ≤ q ̂ P t ,π t ( x, y ) ; the last step follows from Corollary D.3.6 as well.

Conditioning on the event E CON , we have the term in Eq. (37) further bounded as

̸

̸

̸

̸

where the second step follows from the fact that q P,π t ( x, y ) ≤ u t ( x, y ) as P ∈ P i ( t ) according to the event E CON ; the last step follows from the fact that ∑ x ∈ S k ∑ y ∈ A q ̂ P t ,π t ( x, y | s, a ) ≤ 1 . Following the same argument, for the term in Eq. (38), we have

̸

̸

Plugging Eq. (39) and Eq. (40) into Eq. (36), we have

̸

̸

̸

where the third step follows from the facts that q ̂ P t ,π t ( x, y | s, a ) q ̂ P t ,π t ( s, a ) ≤ q ̂ P t ,π t ( x, y ) ≤ u t ( x, y ) for any ( s, a ) , ( x, y ) ∈ S × A , and ∑ b = a q ̂ P t ,π t ( s ) π t ( b | s ) q ̂ P t ,π t ( x, y | s, b ) ≤ q ̂ P t ,π t ( x, y ) similarly.

̸

̸

The first part of Lemma C.7.2 ensures that ν t ( s, a ) ∈ [0 , L 2 ] , which can be used to bound the growth of the learning rate.

Proposition C.7.3. The learning rate γ t defined in Definition 5.2 satisfies for any state-action pair (s, a):

where i ( t ) is the epoch to which episode t belongs and t i is the first episode of epoch i .

Proof. We prove this statement by induction on t . The equation trivially holds for t = t i ( t ) which is the first round of epoch i ( t ) . For the induction step, we first note that Dν t ( s, a ) ∈ [0 , L 2 ] . We introduce some notations to simplify the proof: we use c to denote Dν t ( s, a ) , x to denote γ t ( s, a ) , and the induction hypothesis is x ≤ √ S + γ where S = D ∑ t -1 j = t i ( t ) ν j ( s, a ) and γ = γ t i ( t ) ( s, a ) . Proving the induction step is the same as proving:

First, we can verify that f ( x ) = x + c 2 x is an increasing function of x for x ≥ √ c/ 2 . As c ∈ [0 , L 2 ] and x ≥ γ ≥ L , we can use the induction hypothesis x ≤ √ S + γ to upper bound x in the left-hand side of Eq. (41), and we get:

where the second to last step follows from c ≤ L 2 and γ ≥ L , which ensures that 2 √ S +2 √ γ ≥ 2 √ S + γ 2 ≥ 2 √ S + c . The last step follows from the concavity of the square-root function which ensures that √ S ≤ √ S + c -c 2 √ S + c .

Then, we can use the second part of Lemma C.7.2 to continue the bound of Proposition C.7.3 and derive a bound that only depends on the suboptimal actions.

̸

Proposition C.7.4. If ν t is defined as in Definition 5.2, we have for any deterministic policy π : S → A and any state s = s L :

̸

Proof. For each epoch i and state-action pair ( s, a ) , we have:

where the first step follows from Jensen's inequality, and the second step uses Lemma C.7.2. Note that, for a = π ( s ) , we have:

̸

̸

Therefore, we have

̸

̸

where the first step follows from Eq. (42); the second step uses the fact that √ x + y ≤ √ x + √ y for x, y ≥ 0 ; the fourth step follows from Lemma D.1.1 with the event E CON and Corollary C.8.2; the last step applies Corollary D.3.6.

C.7.2 Bounding ESTREG i ( π ⋆ ) for Varying Learning Rate

We now focus on bounding ESTREG i ( π ⋆ ) for an individual epoch i .

Notations for FTRL Analysis. For any fixed epoch i and any integer t ∈ [ t i , t i +1 -1] , we introduce the following notations.

With these notations, we have q t = ̂ q t = q ̂ P t ,π t = q ˜ P i ,π t . Also, according to the loss shifting technique (specifically Corollary D.4.2), q t and ˜ q t can be equivalently written as

̸

where g t : S × A → R is defined as

Finally, We also define u = q ˜ P i ,π ⋆ and

where q max s,a is the occupancy measure associated with a policy that maximizes the probability of visiting ( s, a ) given the optimistic transition ˜ P i . Then, we have v ∈ Ω( ˜ P i ) because u, q max s,a ∈ Ω( ˜ P i ) for all ( s, a ) , and Ω( ˜ P i ) is convex [Jin and Luo, 2020, Lemma 10].

Therefore, ESTREG i ( π ⋆ ) can be rewritten as:

where term (I) is equivalent to

because 〈 q t -v, g t -̂ ℓ t 〉 = 0 according to Lemma D.4.1.

We then present the following lemma to provide a bound for ESTREG i ( π ⋆ ) .

Lemma C.7.5. For g t define in Eq. (45) , v defined in Eq. (46) , and any non-decreasing learning rate such that γ 1 ( s, a ) ≥ 256 L 2 | S | for all ( s, a ) , Algorithm 4 ensures that ESTREG i ( π ⋆ ) is bounded by

̸

Proof. We start from the decomposition in Eq. (47).

Bounding (I). By adding and subtracting F t ( q t ) -G t (˜ q t ) , we decompose (I) into a stability term and a penalty term:

As γ 1 ( s, a ) ≥ 256 L 2 | S | , Lemma C.7.6 ensures 1 2 q t ( s, a ) ≤ ˜ q t ( s, a ) ≤ 2 q t ( s, a ) for all ( s, a ) , which allows us to apply [Jin and Luo, 2020, Lemma 13] to bound the stability term by

The penalty term without expectation is bounded as:

̸

̸

where the second step uses the optimality of ˜ q t -1 and ˜ q t i +1 -1 so that G t -1 (˜ q t -1 ) ≤ G t -1 ( q t ) and G t i +1 -1 (˜ q t i +1 -1 ) ≤ G t i +1 -1 ( v ) ; the third step follows the definitions in Eq. (43); the seventh step lower-bounds v ( s, a ) by 1 ι 2 q max s,a ( s, a ) for each ( s, a ) ; the eighth step uses the definition of q max s,a .

Now, by taking the equivalent perspective in Eq. (44) and Eq. (48) and repeating the exact same argument, the same bound holds with ̂ ℓ t repalced by g t . Thus, we have shown

̸

Bounding (II). By direct calculation, we have

̸

̸

̸

̸

̸

̸

̸

where the first step uses the definition of v in Eq. (46) and the third step applies the Hölder's inequality. Then, we further show:

where the second step repeatedly applies the triangle inequality, and those terms are bounded by 2 L in the next step; the fourth step bounds ℓ t ( s, a ) ≤ 1 for all ( s, a ) and bounds ∑ t i +1 -1 t = t i b t ( s ) ≤ 2 C P | S | T by using the fact u t ( s ) ≥ 1 | S | T for all t, s (see Lemma D.2.8); the fifth step bounds t i +1 -t i by T .

Finally, we combine the bounds in Eq. (52) and Eq. (53) to complete the proof.

Lemma C.7.6 (Multiplicative Stability) . For ˜ q t and q t defined in Eq. (44) , if the learning rate fulfills γ t ( s, a ) ≥ γ 1 ( s, a ) = 256 L 2 | S | for all t and ( s, a ) , then, 1 2 q t ( s, a ) ≤ ˜ q t ( s, a ) ≤ 2 q t ( s, a ) holds for all ( s, a ) .

Proof. We first show that

̸

where the second step uses ( x -y ) 2 ≤ x 2 + y 2 for any x, y ≥ 0 ; the third step bounds ℓ t ( s, a ) 2 ≤ 1 and b t ( s ) 2 ≤ 16 L 2 u t ( s ) 2 according to the definition of Eq. (7); the fourth step holds as γ t ( s, a ) ≥ γ 1 ( s, a ) = 256 L 2 | S | and q t ( s, a ) ≤ u t ( s, a ) for all ( s, a ) ; the fifth steps uses the fact that ∑ a q t ( s, a ) 2 ≤ ( ∑ a q t ( s, a )) 2 ≤ u t ( s ) 2 .

Once Eq. (54) holds, one only needs to repeat the same argument of [Jin and Luo, 2020, Lemma 12] to obtain the claimed multiplicative stability.

Lemma C.7.7. Under the conditions of Lemma C.7.5, event E CON , and the learning γ t ( s, a ) defined in Definition 5.2, the following holds for any deterministic policy π ⋆ : S → A :

̸

̸

̸

Proof. We start by bounding the second part for each t :

where the second step uses the definition of ν t and ( x -y ) 2 ≤ 2( x 2 + y 2 ) for any x, y ∈ R .

̸

where the first step uses the fact that γ t ( s, a ) ≥ 256 L 2 | S | , the second step bounds b t ( s ) 2 ≤ 4 Lb t ( s ) and ∑ a q t ( s, a ) 2 ≤ q t ( s ) , and the last step applies Lemma 3.1.

Then, we bound the remaining term:

Using Definition 5.2, we can bound Eq. (56) as:

̸

̸

where the first step applies the definition of γ t ( s, a ) which gives: ν t ( s,a ) γ t ( s,a ) = 2 D ( γ t +1 ( s, a ) -γ t ( s, a )) ; the second step simplifies the telescopic sum and uses γ t i ( s, a ) ≥ 0 ; the third step uses D = 1 log( ι ) ; the fourth step uses Proposition C.7.3; the sixth step applies Proposition C.7.4.

C.7.3 Bounding ESTREG

Using the equality ESTREG ( π ⋆ ) = E [ ∑ N i =1 ESTREG i ( π ⋆ ) ] where N is the number of epochs, we are now ready to complete the proof of Lemma C.3.4.

Lemma C.7.8. Over the course of T episodes, Algorithm 4 runs at most O ( | S || A | log( T )) epochs.

Proof. By definition, the algorithm resets each time a counter of the number of visits to a specific state-action pair doubles. Each state-action pair is visited at most once for each of the T rounds, so it can trigger a new epoch at most log T times. Summing on all state-action pairs finishes the proof.

Proof of Lemma C.3.4. Using Lemma C.7.5 and Lemma C.7.7, we have

ESTREG ( π ⋆ )

̸

̸

where the first step follows from the definition of ESTREG ( π ⋆ ) ; the third step uses the fact that N = O ( | S || A | log ( T )) and the Cauchy-Schwarz inequality; the fourth step uses √ xy ≤ x + y for any x, y ≥ 0 with x = LC P and y = | A | ; the last step follows from Cauchy-Schwarz inequality again.

̸

̸

̸

̸

̸

̸

C.8 Properties of Optimistic Transition

We summarize the properties guaranteed by the optimistic transition defined in Definition C.1.1.

Lemma C.8.1. For any epoch i , any transition P ′ ∈ P i , any policy π , and any initial state u ∈ S , it holds that

Proof. We prove this result via a forward induction from layer k ( u ) to layer L -1 .

Base Case: for the initial state u , q ˜ P i ,π ( u, a | u ) = q P ′ ,π ( u, a | u ) = π ( a | u ) for any action a ∈ A . For the other state s ∈ S k ( u ) , we have q ˜ P i ,π ( s, a | u ) = q P ′ ,π ( s, a | u ) = 0 .

Induction step: Suppose q ˜ P i ,π ( s, a | u ) ≤ q P ′ ,π ( s, a | u ) holds for all the state-action pair ( s, a ) with k ( s ) < h . Then, for any ( s, a ) ∈ S h × A , we have

where the second step follows from the induction hypothesis and the definition of optimistic transition in Definition C.1.1.

Corollary C.8.2. Conditioning on the event E CON , it holds for any epoch i and any policy π that

Lemma C.8.3. (Optimism of Optimistic Transition) Suppose the high-probability event E CON holds. Then for any policy π , any ( s, a ) ∈ S × A , and any valid loss function ℓ : S × A → R ≥ 0 , it holds that

Proof. According to Corollary C.8.2, we have for all epoch i that

Therefore, we have

where the second step follows from Eq. (58). The statement for the Q -function can be proven in the same way.

Next, we argue that our optimistic transition provides a tighter performance estimation compared to the approach of Jin et al. [2021]. Specifically, Jin et al. [2021] proposes to subtract the following exploration bonuses BONUS i : S × A → R from the loss functions

where B i ( s, a, s ′ ) is the confidence bound defined in Eq. (4). This makes sure Q ¯ P i ,π ( s, a ; ℓ -BONUS i ) is no larger than the true Q -function Q P,π ( s, a ; ℓ ) as well, but is a looser lower bound as shown below.

Lemma C.8.4. (Tighter Performance Estimation) For any policy π , any ( s, a ) ∈ S × A , and any bounded loss function ℓ : S × A → [0 , 1] , it holds that

Proof. We prove this result via a backward induction from layer L to layer 0 .

Base Case: for the terminal state s L , we have Q ¯ P i ,π ( s, a ; ℓ -BONUS i ) = Q ˜ P i ,π ( s, a ; ℓ ) = 0 . Induction step: suppose the induction hypothesis holds for all the state-action pairs ( s, a ) ∈ S × A with k ( s ) > h . For any state-action pair ( s, a ) ∈ S h × A , we first have

Clearly, when ∑ u ∈ S k ( s )+1 B i ( s, a, u ) ≥ 1 , we have Q ¯ P i ,π ( s, a ; ℓ -BONUS i ) ≤ 0 by the definition of BONUS i , which directly implies that Q ¯ P i ,π ( s, a ; ℓ -BONUS i ) ≤ Q ˜ P i ,π ( s, a ; ℓ ) . So we continue the bound under the condition ∑ u ∈ S k ( s )+1 B i ( s, a, u ) < 1 :

where the second step follows from the fact that V ˜ P i ,π ( u ; ℓ ) ≤ L ; the third step follows from the definition of optimistic transition ˜ P i .

Combining these two cases proves that Q ¯ P i ,π ( s, a ; ℓ -BONUS i ) ≤ Q ˜ P i ,π ( s, a ; ℓ ) for any ( s, a ) ∈ S h × A , finishing the induction.

D Supplementary Lemmas

D.1 Expectation

Lemma D.1.1. ([Jin et al., 2021, Lemma D.3.6]) Suppose that a random variable X satisfies the following conditions:

X < R where R is a constant.

X < Y conditioning on event A , where Y ≥ 0 is a random variable.

Then, it holds that E [ X ] ≤ E [ Y ] + Pr [ A c ] · R where A c is the complementary event of A .

D.2 Confidence Bound with Known Corruption

In this subsection, we show that the empirical transition is centered around the transition P . Let [ T ] := { 1 , · · · , T } . Recall that E i := { t ∈ [ T ] : episode t belongs to epoch i } , ι = | S || A | T δ , and we define the following quantities:

Note that based on definition of T i ( s, a ) , we have m i ( s, a ) = | T i ( s, a ) | . Then, we present the following lemma which shows the concentration bound between P ( s ′ | s, a ) and ¯ P i ( s ′ | s, a ) .

Lemma D.2.1. (Detailed restatement of Lemma 2.2) Event E CON occurs with probability at least 1 -δ where,

and B i ( s, a, s ′ ) is defined in Eq. (4) as:

Proving this lemma requires several auxiliary results stated below. For any episode t ∈ [ T ] and any layer k < L , we use s IMG t,k to denote an imaginary random state sampled from P ( ·| s t,k , a t,k ) . For any ( s, a, s ′ ) ∈ W k , let

We now proceed with a couple lemmas.

Lemma D.2.2 (Lemma 2, [Jin et al., 2020]) . Event E 1 occurs with probability at least 1 -3 δ/ 4 where

and ¯ ω i ( s, a, s ′ ) for any ( s, a, s ′ ) ∈ W k and 0 ≤ k ≤ L -1 is defined as

We note that as long as | A | T ≥ 16 / 3 , event E 1 can occur with probability at least 1 -3 δ/ 4 (unlike 1 -4 δ in Lemma 2 of Jin et al. [2020] ).

Lemma D.2.3. Event E 2 occurs with probability at least 1 -δ/ 4 where

and ω i ( s, a, s ′ ) for any ( s, a, s ′ ) ∈ W k and 0 ≤ k ≤ L -1 is defined as

̸

Proof. For any fixed ( s, a, s ′ ) , if m i ( s, a ) = 0 , the claimed bound in E 2 holds trivially, so we consider the case m i ( s, a ) = 0 below. By definition we have:

Then, we construct the martingale difference sequence { X t ( s, a, s ′ ) } ∞ t =1 w.r.t. filtration {F t,k ( s ) } ∞ t =1 (see [Lykouris et al., 2019, Definition 4.9] for the formal definition of these filtrations) where

With the definition of X t ( s, a, s ′ ) , one can show

where the second step uses ( x -y ) 2 ≤ 2( x 2 + y 2 ) for any x, y ∈ R ; the third step uses ( x -y ) 2 ≤ x 2 + y 2 for x, y ∈ R ≥ 0 and the fact that C P t ( s, a, s ′ ) ∈ [0 , 1] , thereby C P t ( s, a, s ′ ) 2 ≤ C P t ( s, a, s ′ ) ; the last step holds based on the definitions of T i ( s, a ) , s IMG t,k ( s )+1 , and s t,k ( s )+1 as well as the fact that C P t ( s, a, s ′ ) is F t,k ( s ) -measurable.

By using the result in Eq. (63), we bound the average second moment σ 2 as

By applying Lemma D.2.4 with b = 2 and the upper bound of σ 2 shown in Eq. (64), as well as using the fact that m i ( s, a ) = | T i ( s, a ) | , for any ( s, a, s ′ ) , we have the following with probability at least 1 -δ/ (4 T | S | 2 | A | ) ,

where the first inequality applies Lemma D.2.4; the second inequality bounds all logarithmic terms by log ι with an appropriate constant factor (using m i ( s, a ) ≤ T and | A | ≥ 2 ); the third step follows the fact that P t ( s ′ | s, a ) ≤ P ( s ′ | s, a ) + C P t ( s, a, s ′ ) ; the fourth step uses the definition C P i ( s, a, s ′ ) = ∑ t ∈ T i ( s,a ) C P t ( s, a, s ′ ) ; the fifth step applies the inequality √ 4 xy ≤ x + y , ∀ x, y ≥ 0 for x = C P i ( s, a, s ′ ) and y = log ι . Applying a union bound over all ( s, a, s ′ ) and epochs i ≤ T , we complete the proof.

Lemma D.2.4 (Anytime Version of Azuma-Bernstein) . Let { X i } ∞ i =1 be b -bounded martingale difference sequence with respect to F i . Let σ 2 = 1 N ∑ N i =1 E [ X 2 i | F i -1 ] . Then, with probability at least 1 -δ , for any N ∈ N + , it holds that:

Proof. This follows the same argument as Lemma G.2 in [Lykouris et al., 2019].

Lemma D.2.5. Event E occurs with probability at least 1 -δ , where

Proof. Conditioning on events E 1 and E 2 , we have

Using a union bound for E 1 and E 2 , we complete the proof.

Armed with above results, we are now ready to prove Lemma D.2.1.

Proof of Lemma D.2.1. Conditioning on event E , for any fixed ( s, a, s ′ ) with m i ( s, a ) = 0 (otherwise the desired bound holds trivially), we have

′

̸

where the second step holds under E ; the third step uses √∑ n i =1 x i ≤ ∑ n i =1 √ x i for all x i ∈ R ≥ 0 ; the fourth step and the fifth step use 2 √ xy ≤ x + y for x, y ≥ 0 . Rearranging the above, we obtain

Thus, conditioning on event E , one can show for all ( s, a, s ′ ) ∈ W k and k < L -1

where the second step uses Eq. (66), the third step applies the definition of ¯ ω i ( s, a, s ′ ) . Finally, using the fact C P i ( s, a, s ′ ) ≤ C P , we complete the proof.

Then, we present an immediate corollary of Lemma D.2.1.

Corollary D.2.6. Consider any epoch i and any transition P ′ ∈ P i . The following holds (recall ̂ m i defined at the beginning of the appendix),

Proof. As P ′ ∈ P i , we start from Eq. (67):

where the last step uses the Cauchy-Schwarz inequality and the fact that ∑ s ′ ∈ k ( s )+1 C P i ( s, a, s ′ ) ≤ C P . Since ∥ P ( ·| s, a ) -¯ P i ( ·| s, a ) ∥ 1 ≤ 2 , we combine this trivial bound and the bound of ∥ P ( ·| s, a ) -¯ P i ( ·| s, a ) ∥ 1 in Eq. (68) to arrive at

finishing the proof.

We conclude this subsection with two other useful lemmas.

Lemma D.2.7. Conditioning on event E CON , it holds for all tuple ( s, a, s ′ ) and epoch i that

Proof. Fix the epoch i and tuple ( s, a, s ′ ) . According to the definitions of E CON in Eq. (59) and ̂ m , we have

Therefore, by direct calculation, we have

where the second step and last step follow from the fact that √ xy ≤ 1 2 ( x + y ) for any x, y ≥ 0 . Finally, rearranging the above inequality finishes the proof.

̸

Lemma D.2.8. (Lower Bound of Upper Occupancy Measure) For any episode t and state s = s L , it always holds that u t ( s ) ≥ 1 / | S | T .

Proof. Fix the episode t and state s . We prove the lemma by constructing a specific transition ̂ P ∈ P i ( t ) , such that q ̂ P,π ( s ) ≥ 1 / | S | T for any policy π , which suffices due to the definition of u t ( s ) .

Specifically, ̂ P is defined as, for any tuple ( s, a, s ′ ) ∈ W k and k = 0 , . . . , L -1 :

By direct calculation, one can verify that ̂ P is a valid transition function. Then, we show that ̂ P ∈ P i ( t ) by verifying the condition for any transition tuple ( s, a, s ′ ) ∈ W k and k = 0 , . . . , L -1 :

where the last step follows from the definition of confidence intervals in Eq. (4).

Finally, we show that q ̂ P,π ( s ) ≥ 1 / | S | T as:

which concludes the proof.

D.3 Difference Lemma

Lemma D.3.1 (Theorem 5.2.1 of [Kakade, 2003]) . (Performance Difference Lemma) For any policies π 1 , π 2 and any loss function ℓ : S × A → R ,

̸

̸

In fact, the same also holds for our optimistic transition where the layer structure is violated. For completeness, we include a proof below.

Lemma D.3.2. For any policies π 1 , π 2 , any loss function ℓ : S × A → R , and any transition ˜ P where ∑ s ′ ∈ S k ( s ) +1 ˜ P ( s ′ | s, a ) ≤ 1 for all state-action pairs ( s, a ) (the remaining probability is assigned to s L ), we have

̸

̸

Proof. By direct calculation, we have for any state s = s L :

̸

where the last step follows from recursively repeating the first three steps.

Lemma D.3.3. (Occupancy Measure Difference, [Jin et al., 2021, Lemma D.3.1]) For any transition functions P 1 , P 2 and any policy π ,

where q P ′ ,π ( s | w ) is the probability of visiting s starting w under policy π and transition P ′ . Lemma D.3.4. ([Dann et al., 2023, Lemma 17]) For any policies π 1 , π 2 and transition function P ,

̸

̸

Corollary D.3.5. For any policies π 1 , mapping π 2 : S → A (that is, a deterministic policy), and transition function P , we have

̸

Proof. According to Lemma D.3.4, we have

̸

Note that, for every state s , it holds that

̸

̸

̸

where the first step follows from the fact that π 2 ( a | s ) = 1 when a = π 2 ( s ) , and π 2 ( b | s ) = 0 for any other action b = π 2 ( s ) .

Therefore, we have

̸

which concludes the proof.

According to Lemma D.3.3, we can estimate the occupancy measure difference caused by the corrupted transition function P t at episode t .

Corollary D.3.6. For any episode t and any policy π , we have

̸

̸

̸

Proof. By direct calculation, we have for any episode t and any s = s L

where the first step follows from Lemma D.3.3, the second step bounds q P t ,π ( s | w ) ≤ 1 , and the last two steps follows from the definition of C P t .

̸

Moreover, taking the summation over all states s = s L , we have

̸

̸

where the third step follows from the fact that ∑ s ∈ S h q P t ,π ( s | w ) = 1 for any h ≥ k ( w ) .

̸

̸

̸

̸

̸

Corollary D.3.7. For any policy sequence { π t } T t =1 and loss functions { ℓ t } T t =1 such that ℓ t ∈ S × A → [0 , 1] for any t ∈ { 1 , · · · , T } , it holds that

Proof. By direct calculation, we have

̸

where the first step applies the Hölder's inequality; the third step follows form the fact that q P ′ ,π t ( s, a ) = q P ′ ,π t ( s ) π t ( a | s ) for any state-action pair ( s, a ) and any transition function P ′ ; the fifth step applies Corollary D.3.6; the last step follows from the definition of C P .

Following the same idea in the proof of [Dann et al., 2023, Lemma 16], we also consider a tighter bound of the difference between occupancy measures in the following lemma.

̸

Lemma D.3.8. Suppose the event E CON holds. For any state s = s L , episode t and transition function P ′ t ∈ P i ( t ) , we have

̸

Proof. According to Lemma D.3.3, we have

̸

̸

where the second step first subtracts and adds q P ′ t ,π t ( s | w ) and then applies Lemma D.3.3 again for | q P ′ t ,π t ( s | w ) -q P,π t ( s | w ) | , and the third step follows from Lemma D.2.7 .

Clearly, we can bound term (a) as:

̸

On the other hand, for term (b), we decompose it into three terms in Eq. (70), Eq. (71), and Eq. (72):

According to the AM-GM inequality, we have the term in Eq. (70) bounded as

where the first step follows from re-arranging the summation; the second step applies the fact that √ xy ≤ x + y for any x, y ≥ 0 ; the third step rearranges the summation order, and the fourth step follows form the facts that ∑ x ∈ S h ∑ y ∈ A q P,π t ( x, y | w ) P ( z | x, y ) = q P,π t ( z | w ) and ∑ ( u,v,w ) ∈ W k q P,π t ( u, v ) P ( w | u, v ) q P,π t ( x, y | w ) = q P,π t ( x, y ) for any k and ( x, y, z ) .

Similarly, we have the term in Eq. (71) bounded as

̸

̸

where the first step follows from the fact that ̂ m i ( t ) ( s, a ) ≥ C + log ( ι ) according to its definition; the third step follows from the facts that ∑ x ∈ S h ∑ y ∈ A q P,π t ( x, y | w ) ≤ 1 and ∑ ( u,v,w ) ∈ W k q P,π t ( u, v ) P ( w | u, v ) q P,π t ( x, y | w ) = q P,π t ( x, y ) .

For the term in Eq. (72), we have

̸

according to the fact that x ∈ S h y ∈ A

Putting all the bounds for the terms in Eq. (70), Eq. (71), and Eq. (72) together yields the bound of TERM (B) that

̸

Combining this bound with that of TERM (A) finishes the proof.

D.4 Loss Shifting Technique with Optimistic Transition

Lemma D.4.1. Fix an optimistic transition function ˜ P (defined in Section 5). For any policy π and any loss function ℓ : S × A → R , we define function g : S × A → R similar to that of Jin et al. [2021] as:

where the state-action and state value function Q ˜ P,π and V ˜ P,π are defined with respect to the optimistic transition ˜ P and π as following:

̸

Then, it holds for any policy π ′ that,

̸

where V ˜ P,π ( s 0 ; ℓ ) is only related to ˜ P , π and ℓ , and is independent with π ′ .

Proof. By the extended performance difference lemma of the optimistic transition in Lemma D.3.1, we have the following equality holds for any policy π ′ :

̸

On the other hand, we also have

̸

Therefore, subtracting V ˜ P,π ′ ( s 0 ; ℓ ) yields that

̸

The proof is finished after using the definition of g ˜ P,π .

Therefore, we have the following result for the FTRL framework which is similar to [Jin et al., 2021, Corollary A.1.2.].

Corollary D.4.2. The FTRL update in Algorithm 4 can be equivalently written as:

where g τ ( s, a ) = Q ˜ P i ,π τ ( s, a ; ̂ ℓ τ ) -V ˜ P i ,π τ ( s ; ̂ ℓ τ ) for any state-action pair ( s, a ) .

D.5 Estimation Error

Lemma D.5.1. ([Jin et al., 2020, Lemma 10]) With probability at least 1 -δ , we have for all k = 0 , . . . , L -1 ,

and

Proof. Simply replacing the stationary transition P with the sequence of transitions { P t } T t =1 in the proof of [Jin et al., 2020, Lemma 10] suffices.

Proposition D.5.2. Let E EST be the event such that we have for all k = 0 , . . . , L -1 simultaneously

and

We have Pr[ E EST ] ≥ 1 -δ.

Proof. The proof directly follows from the definition of ι , which ensures that ι ≥ L / δ .

Based on the event E EST , we introduce the following lemma which is critical in analyzing the estimation error.

Lemma D.5.3. Suppose the event E EST defined in Proposition D.5.2 holds. Then, we have for all k = 0 , . . . , L -1 ,

and,

where the second step follows from the definitions of C P and ̂ m i ( t ) ( s, a ) , and the last step applies Lemma D.5.1.

Similarly, we have

where the first step adds and subtracts q P t ,π t ; the second step follows from Corollary D.3.6; the fourth step uses Proposition D.5.2 due to the fact that ̂ m i ( s, a ) ≥ max { m i ( s, a ) , 1 } .

Lemma D.5.4. (Extension of [Dann et al., 2023, Lemma 16] for adversarial transition) Suppose the high-probability events E EST (defined in Proposition D.5.2) and E CON (defined in Lemma D.2.1) hold together. Let P s t be a transition function in P i ( t ) which depends on s , and let g t ( s ) ∈ [0 , G ] for some G > 0 . Then,

̸

Proof. By Lemma D.3.8, under the event E CON , we have

̸

̸

By Lemma D.5.3, the last two terms can be bounded under the event E EST as

̸

̸

̸

For the first term in Eq. (73), we first rewrite it as

̸

̸

For any θ > 0 and layer h , conditioning on the event E EST , we have

̸

̸

where the second step uses the fact that √ xy ≤ x + y for all x, y ≥ 0 , the fourth step follows from Lemma D.5.3.

Then, for any layer h = 0 , . . . L -1 , picking the optimal θ gives

̸

Finally, taking the summation over all the layers yields

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

which finishes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the challenge of online learning in adversarial MDPs where the loss functions and transition functions are chosen by a malicious adversary. Although previous algorithms achieving $O(\\sqrt{T})$ regret with fixed transition functions could not handle adversarial transitions, in this paper, the authors propose a new algorithms that can handle both adversarial losses and transitions, with regret increasing smoothly based on the degree of maliciousness $C^P$. The first algorithm achieves $O(\\sqrt{T} + C^P)$ regret, where $C^P$ measures how adversarial the transitions are and can be at most $O(T)$. Second, a black-box reduction approach is introduced to remove the requirement of knowing $C^P$. The algorithm is further refined to adapt to easier environments and achieve $O(U + \\sqrt{U C^L} + C^P)$ regret, where $U$ is a gap-dependent coefficient and $C_L$ represents the amount of corruption on losses.\n\nSTRENGTHS:\n- A variant of the UOB-REPS algorithm is suggested to achieve regret in completely adversarial environments when the adversarial transition parameter, $C^P$, is known. This is accomplished by using an enlarged confidence set with the log-barrier regularizer and introducing a novel amortized bonus term.\n- The requirement of knowing the adversarial transition parameter, $C^P$, is eliminated by proposing a black-box reduction approach. This approach provides the same guarantee (up to logarithmic factors) even if $C^P$ is unknown.\n- The algorithms are further refined to simultaneously adapt to the maliciousness of the loss functions and achieve low regret. This enables the algorithms to handle various degrees of adversarial behavior in the loss functions while maintaining their regret performance.\n\nWEAKNESSES:\n- The organization of the paper may be making it difficult for me to comprehend. I think it is recommended to have the main algorithm presented in the main paper for better clarity. Since the paper covers multiple algorithms, it would be helpful to focus on explaining how they differ from existing methods, the challenges they address, and the novelty they bring to the field. Providing a dedicated conclusion section would enhance the understanding of the paper by summarizing the key findings and contributions.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nGiven that this paper is primarily focused on theoretical aspects, its direct impact on society is expected to be neutral.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online reinforcement learning in tabular MDPs when the losses and transitions can be adversarially changing from round to round. They show that one can achieve regret guarantees which are $O(\\sqrt{T} + C^P)$ where $C^P$ measures the degree to which transitions are changing.\n\nSpecifically they \n- devise an algorithm which achieves the aforementioned guarantee when the value of $C^P$ is known.\n- apply further reductions to get an algorithm which does not need to know the value of $C^P$.\n- get gap-dependent regret bounds when the value of $C^P$ is known.\n\nSTRENGTHS:\n- The results seem very impressive. To my knowledge this is the first paper which studies adversarial MDPs with changing transitions. The authors quantify the notion of changing transitions and prove a regret bound.\n- The algorithms and techniques seem very novel, and might be useful for future work.\n- References and connections to previous work are stated clearly.\n\nWEAKNESSES:\n- The writing is a bit unclear, specifically regarding the technical details. This paper is a very technical paper with long proofs, but I do think it would be helpful to state the algorithms (or some simplified versions of the algorithms) in the main text and relegate some of the finer details to the appendix.\n- Upon first reading, the proof ideas in Section 3 onwards did not make much sense, because they were trying to cover a lot of fine details, even though the overall proof sketch was not really discussed much. I'd encourage the authors to put more effort into improving readability.\n\nTypos:\n- Line 116: should this be $S$ instead of $X$?\n\nQUESTIONS:\n1. This paper considers transitions and losses which are specified ahead of time by an adversary. Do you think it is possible to consider adversarial losses/transitions which the adversary can suggest after seeing a history of the learner's decisions? (a harder setting)\n2. What is the role of the splitting into epochs feature of the algorithm? What goes wrong if the algorithm updates the empirical transition in every round, as opposed to in every epoch?\n3. Is there some intuition for why the algorithm uses the upper occupancy bound following line 161, as opposed to just using the occupancy measure associated with $\\bar{P}$?\n4. What is the message of Lemma 3.1? Why is the inequality $\\sum_t C_t^P/u_t(s) \\le \\sum_t b_t(s)$ useful?\n5. This might be a standard technique, but why are the costs in Algorithm 3 offset by a reward $r_\\tau$? What does this capture?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider no-regret learning in adversarial MDPs, when the dynamics may change adversarially across episodes. They design an algorithm which provides a regret guarantee of $\\tilde{O}(\\sqrt{T} + C)$ where $C$ is the total deviation from some fixed transition function, and the benchmark is the best fixed Markov policy in hindsight. The authors show that this result can be obtained even when $C$ is unknown to the algorithm, by constructing a black-box reduction from an algorithm which knows $C$. The authors also consider the stochastically constrained adversarial setting and provide a gap-dependent regret guarantee, but in this setting knowing $C$ is required.\n\nSTRENGTHS:\n* The results established in the paper improve upon previous results in corruption-robust reinforcement learning, and in particular constitute the first $\\tilde{O}(\\sqrt{T} + C)$-type bounds even when the losses are adversarial.\n* The techniques presented in the paper seem novel and interesting, and may be of independent interest in various RL scenarios.\n* The authors provide an overview of the analysis presenting the main challenges and ideas, making it easier to understand their main contributions and high-level techniques.\n* The black-box reduction from an algorithm which knows $C$ to one which doesn't need to know $C$ seems particularly interesting to me, and may be of interest in other non-stationary scenarios when trying to design algorithm that adapt to some corruption budget.\n\nWEAKNESSES:\n* Since the dynamics of the MDP changes adversarially across episodes, it makes less sense to set the benchmark as a Markov policy, as it is no longer the case that the optimal policy on a sequence of MDPs is WLOG Markov. The authors do not address this point in the current version of the paper, and I would like to hear from them why this assumption is needed and whether or not competing against a more general benchmark policy is hard.\n* At first, it may seem that $C$ measures the total variation of the transition dynamics across time, as is the case in other non-stationary RL formulations. However, the authors define $C$ to be the sum of variations from a single fixed transition function. This is a weaker definition, and in particular the results presented in this paper do not include a setup where the dynamics have a drift of $1/ \\sqrt{T}$ per episode, and for every fixed transition $P'$ the total deviation from it would be $\\Omega(T)$. In such a setting, the authors' bounds would be meaningless even though obtaining sublinear regret (even dynamic regret) is indeed possible.\n\nQUESTIONS:\nSee my remarks above under \"Weaknesses\" - I'd appreciate it if the authors could address the limited generality of the benchmark policy, as well as the problem of handling drift in the transitions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the problem of reinforcement learning under adversarial reward and transitions. When evaluating the regret against the best fixed policy in hindsight, the algorithm proposed in this paper achieves the optimal regret O(\sqrt{T} + C^P), which is followed by other favorable extensions including being agnostic to the corruption amount $C^P$ and $C^L$, as well as gap-dependent regret bounds.

STRENGTHS:
The theoretical results presented in this paper is solid. Several new techniques, including a new model selection framework that allows adversarial environments, is of independent interestis.

WEAKNESSES:
The main weakness is the lack of justification for the specific type of regret studied in this paper. To the best of my knowledge, this is the only paper that studies the regret against the best policy in hindsight in a mild-corruption setting, i.e. C^P \leq o(T), so some words justifying this new notion would be helpful. I'm not sure when such a regret notion is desirable to study in this scenario, since if C^P \leq o(T), the best policy in hindsight will converge to the optimal policy in the underlying uncorrupted MDP, so the two notions would converge anyways.

Minor: 

There is a missing reference for corruption-robust RL

1. Zhang, X., Chen, Y., Zhu, X., & Sun, W. (2021, July). Robust policy gradient against strong data corruption. In International Conference on Machine Learning (pp. 12391-12401). PMLR.

QUESTIONS:
See above.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper studies learning algorithms for adversarial MDP with adversarial transition functions. The authors developed an algorithm that enjoys O(\sqrt{T} + C^P) regret where C^P measures how adversarial the transition functions are. The developed algorithm could work without knowing C^P. Finally, the authors show that further refinements of the algorithm can adapt to easier environments.

STRENGTHS:
This paper tackles an important question of how to learn adversarial MDPs with adversarial transition functions. Prior work shows that learning with adversarial transition is information-theoretically impossible without paying exponential dependence on the episode length. This paper provides the first algorithm, based on a modification of UOB-REPS, which allows the regret bound to degrade mildly upon the amount of corruption. Furthermore, it also resolves the question of an unknown amount of adversarial transition and considers the question of adaptivity.

WEAKNESSES:
NA

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06