The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition

√ T ) regret when the losses are adversarial and simultaneously O ( polylog ( T ) ) regret when the losses are ( almost ) stochastic . Recent work by [ Jin and Luo , 2020 ] achieves this goal when the fixed transition is known , and leaves the case of unknown transition as a major open question . In this work , we resolve this open problem by using the same Follow-the-Regularized-Leader ( FTRL ) framework together with a set of new techniques . Specifically , we first propose a loss-shifting trick in the FTRL analysis , which greatly simplifies the approach of [ Jin and Luo , 2020 ] and already improves their results for the known transition case . Then , we extend this idea to the unknown transition case and develop a novel analysis which upper bounds the transition estimation error by ( a fraction of ) the regret itself in the stochastic setting , a key property to ensure O ( polylog ( T ) ) regret . 1 Introduction . We study the problem of learning finite-horizon Markov Decision Processes ( MDPs ) with unknown transition through T episodes . In each episode , the learner starts from a fixed initial state and repeats the following for a fixed number of steps : select an available action , incur some loss , and transit to the next state according to a fixed but unknown transition function . The goal of the learner is to minimize her regret , which is the difference between her total loss and that of the optimal stationary policy in hindsight . When the losses are stochastically generated , [ Simchowitz and Jamieson , 2019 , Yang et al. , 2021 ] show that O ( log T ) regret is achievable ( ignoring dependence on some gap-dependent quantities for simplicity ) . On the other hand , even when the losses are adversarially generated , [ Rosenberg and Mansour , 2019a , Jin et al. , 2020 ] show that Õ ( √ T ) regret is achievable.1 Given that the existing algorithms for these two worlds are substantially different , Jin and Luo [ 2020 ] asked the natural question of whether one can achieve the best of both worlds , that is , enjoying ( poly ) logarithmic regret in the stochastic world while simultaneously ensuring some worst-case robustness in the adversarial world . Taking inspiration from the bandit literature and using the classic Follow-the-regularizedLeader ( FTRL ) framework with a novel regularizer , they successfully achieved this goal , albeit under a strong restriction that the transition has to be known ahead of time . Since it is highly unclear how 1Throughout the paper , we use Õ ( · ) to hide polylogarithmic terms . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to ensure that the transition estimation error is only O ( polylog ( T ) ) , extending their results to the unknown transition case is highly challenging and was left as a key open question . In this work , we resolve this open question and propose the first algorithm with such a best-of-bothworlds guarantee under unknown transition . Specifically , our algorithm enjoys Õ ( √ T ) regret always , and simultaneously O ( log2 T ) regret if the losses are i.i.d . samples of a fixed distribution . More generally , our polylogarithmic regret holds under a general condition similar to that of [ Jin and Luo , 2020 ] , which requires neither independence nor identical distributions . For example , it covers the corrupted i.i.d . setting where our algorithm achieves Õ ( √ C ) regret with C ≤ T being the total amount of corruption . Techniques Our results are achieved via three new techniques . First , we propose a new loss-shifting trick for the FTRL analysis when applied to MDPs . While similar ideas have been used for the special case of multi-armed bandits ( e.g. , [ Wei and Luo , 2018 , Zimmert and Seldin , 2019 , Lee et al. , 2020b , Zimmert and Seldin , 2021 ] ) , its extension to MDPs has eluded researchers , which is also the reason why [ Jin and Luo , 2020 ] resorts to a different approach with a highly complex analysis involving analyzing the inverse of the non-diagonal Hessian of a complicated regularizer . Instead , inspired by the well-known performance difference lemma , we design a key shifting function in the FTRL analysis , which helps reduce the variance of the stability term and eventually leads to an adaptive bound with a certain self-bounding property known to be useful for the stochastic world . To better illustrate this idea , we use the known transition case as a warm-up example in Section 3 , and show that the simple Tsallis entropy regularizer ( with a diagonal Hessian ) is already enough to achieve the best-of-both-worlds guarantee . This not only greatly simplifies the approach of Jin and Luo [ 2020 ] ( paving the way for extension to unknown transition ) , but also leads to bounds with better dependence on some parameters , which on its own is a notable result already . Our second technique is a new framework to deal with unknown transition under adversarial losses , which is important for incorporating the loss-shifting trick mentioned above . Specifically , when the transition is unknown , prior works [ Rosenberg and Mansour , 2019a , b , Jin et al. , 2020 , Lee et al. , 2020a ] perform FTRL over the set of all plausible occupancy measures according to a confident set of the true transition , which can be seen as a form of optimism encouraging exploration . Since our loss-shifting trick requires a fixed transition , we propose to move the optimism from the decision set of FTRL to the losses fed to FTRL . More specifically , we perform FTRL over the empirical transition in some doubling epoch schedule , and add ( negative ) bonuses to the loss functions so that the algorithm is optimistic and never underestimates the quality of a policy , an idea often used in the stochastic setting ( e.g. , [ Azar et al. , 2017 ] ) . See Section 4 for the details of our algorithm . Finally , we develop a new analysis to show that the transition estimation error of our algorithm is only polylogarithmic in T , overcoming the most critical obstacle in achieving best-of-both-worlds . An important aspect of our analysis is to make use of the amount of underestimation of the optimal policy , a term that is often ignored since it is nonpositive for optimistic algorithms . We do so by proposing a novel decomposition of the regret inspired by the work of Simchowitz and Jamieson [ 2019 ] , and show that in the stochastic world , every term in this decomposition can be bounded by a fraction of the regret itself plus some polylogarithmic terms , which is enough to conclude the final polylogarithmic regret bound . See Section 5 for a formal summary of this idea . Related work For earlier results in each of the two worlds , we refer the readers to the systematic surveys in [ Simchowitz and Jamieson , 2019 , Yang et al. , 2021 , Jin et al. , 2020 ] . The work closest to ours is [ Jin and Luo , 2020 ] which assumes known transition , and as mentioned , we strictly improve their bounds and more importantly extend their results to the unknown transition case . Two recent works [ Lykouris et al. , 2021 , Chen et al. , 2021 ] also consider the corrupted stochastic setting , where both the losses and the transition function can be corrupted by a total amount of C. This is more general than our results since we assume a fixed transition and only allow the losses to be corrupted . On the other hand , their bounds are worse than ours when specified to our setting — [ Lykouris et al. , 2021 ] ensures a gap-dependent polylogarithmic regret bound of O ( C log3 T + C2 ) , while [ Chen et al. , 2021 ] achievesO ( log3 T +C ) but with a potentially larger gap-dependent quantity . Therefore , neither result provides a meaningful guarantee in the adversarial world when C = T , while our algorithm always ensures a robustness guarantee with Õ ( √ T ) regret . Their algorithms are also very different from ours and are not based on FTRL . The question of achieving best-of-both-worlds guarantees for the special case of multi-armed bandits was first proposed in [ Bubeck and Slivkins , 2012 ] . Since then , many improvements using different approaches have been established over the years [ Seldin and Slivkins , 2014 , Auer and Chiang , 2016 , Seldin and Lugosi , 2017 , Wei and Luo , 2018 , Lykouris et al. , 2018 , Gupta et al. , 2019 , Zimmert et al. , 2019 , Zimmert and Seldin , 2021 , Lee et al. , 2021 ] . One notable and perhaps surprising approach is to use the FTRL framework , originally designed only for the adversarial settings but later found to be able to automatically adapt to the stochastic settings as long as certain regularizers are applied [ Wei and Luo , 2018 , Zimmert et al. , 2019 , Zimmert and Seldin , 2021 ] . Our approach falls into this category , and our regularizer design is also based on these prior works . As mentioned , however , obtaining our results requires the new loss-shifting technique as well as the novel analysis on controlling the estimation error , both of which are critical to address the extra challenges presented in MDPs . 2 Preliminaries . We consider the problem of learning an episodic MDP through T episodes , where the MDP is formally defined by a tuple ( S , A , L , P , { ` t } Tt=1 ) with S being a finite state set , A being a finite action set , L being the horizon , ` t : S × A → [ 0 , 1 ] being the loss function of episode t , and P : S ×A× S → [ 0 , 1 ] being the transition function so that P ( s′|s , a ) is the probability of moving to state s′ after executing action a at state s. Without loss of generality [ Jin et al. , 2020 ] , the MDP is assumed to have a layer structure , that is , the state set S is partitioned into L+ 1 subsets S0 , S1 , . . . , SL such that the state transition is only possible from one layer to the next layer ( in other words , P ( s′|s , a ) must be zero unless s ∈ Sk and s′ ∈ Sk+1 for some k ∈ { 0 , . . . , L− 1 } ) . Moreover , S0 contains s0 only ( the initial state ) , and SL contains sL only ( the terminal state ) . We use k ( s ) to represent the layer to which state s belongs . Ahead of time , the environment decides an MDP with P and { ` t } Tt=1 unknown to the learner . The interaction proceeds through T episodes . In episode t , the learner selects a stochastic policy πt : S×A→ [ 0 , 1 ] where πt ( a|s ) denotes the probability of taking action a at state s.2 Starting from the initial state st0 = s0 , the learner then repeatedly selects an action a t k drawn from πt ( · |stk ) , suffers loss ` t ( stk , a t k ) , and transits to the next state s t k+1 ∈ Sk+1 for k = 0 , . . . , L− 1 , until reaching the terminal state sL . At the end of the episode , the learner receives some feedback on the loss function ` t. In the full-information setting , the learner observes the entire loss function ` t , while in the more challenging bandit feedback setting , the learner only observes the losses of those visited state-action pairs , that is , ` t ( st0 , a t 0 ) , . . . , ` t ( s t L−1 , a t L−1 ) . With slight abuse of notation , we denote the expected loss of a policy π for episode t by ` t ( π ) = E [ ∑L−1 k=0 ` t ( sk , ak ) ∣∣∣P , π ] , where the trajectory { ( sk , ak ) } k=0 , ... , L−1 is the generated by executing policy π under transition P . The regret of the learner against some policy π is then defined as RegT ( π ) = E [ ∑T t=1 ` t ( πt ) − ` t ( π ) ] , and we denote by π̊ one of the optimal policies in hindsight such that RegT ( ̊π ) = maxπ RegT ( π ) . Adversarial world versus stochastic world We consider two different setups depending on how the loss functions ` 1 , . . . , ` T are generated . In the adversarial world , the environment decides the loss functions arbitrarily with knowledge of the learner ’ s algorithm ( but not her randomness ) . In this case , the goal is to minimize the regret against the best policy RegT ( ̊π ) , with the best existing upper bound being Õ ( L|S| √ |A|T ) [ Rosenberg and Mansour , 2019a , Jin et al. , 2020 ] and the best lower bound being Ω ( L √ |S||A|T ) [ Jin et al. , 2018 ] ( for both full-information and bandit feedback ) . In the stochastic world , following [ Jin and Luo , 2020 ] ( which generalizes the bandit case of [ Zimmert and Seldin , 2019 , 2021 ] ) , we assume that the loss functions satisfy the following condition : there exists a deterministic policy π ? : S → A , a gap function ∆ : S × A → R+ and a constant C > 0 such that RegT ( π ? ) ≥ E T∑ t=1 ∑ s6=sL ∑ a6=π ? ( s ) qt ( s , a ) ∆ ( s , a ) − C , ( 1 ) 2Note that πt ( ·|sL ) is not meaningful since no action will be taken at sL . For conciseness , however , we usually define functions over S ×A instead of ( S \ { sL } ) ×A . where qt ( s , a ) is the probability of the learner visiting ( s , a ) in episode t. This general condition covers the heavily-studied i.i.d . setting where ` 1 , . . . , ` T are i.i.d . samples of a fixed distribution , in which case C = 0 , π ? is simply the optimal policy , and ∆ is the gap function with respect to the optimal Q-function . More generally , the condition also covers the corrupted i.i.d . setting with C being the total amount of corruption . We refer the readers to [ Jin and Luo , 2020 ] for detailed explanation . In this stochastic world , our goal is to minimize regret against π ? , that is , RegT ( π ? ) .3 With unknown transition , this general setup has not been studied before , but for specific examples such as the i.i.d . setting , regret bounds of order O ( log T∆MIN ) where ∆MIN = mins , a6=π ? ( s ) ∆ ( s , a ) have been derived [ Simchowitz and Jamieson , 2019 , Yang et al. , 2021 ] . Occupancy measure and FTRL To solve this problem with online learning techniques , a commonly used concept is the occupancy measure . Specifically , an occupancy measure qP̄ , π : S ×A→ [ 0 , 1 ] associated with a policy π and a transition function P̄ is such that qP̄ , π ( s , a ) equals the probability of visiting state-action pair ( s , a ) under the given policy π and transition P̄ . Our earlier notation qt in Eq . ( 1 ) is thus simply a shorthand for qP , πt . Moreover , by definition , ` t ( π ) can be rewritten as〈 qP , π , ` t 〉 by naturally treating qP , π and ` t as vectors in R|S|×|A| , and thus the regret RegT ( π ) can be written as E [ ∑T t=1 〈 qt − qP , π , ` t 〉 ] , connecting the problem to online linear optimization . Given a transition function P̄ , we denote by Ω ( P̄ ) = { qP̄ , π : π is a stochastic policy } the set of all valid occupancy measures associated with the transition P̄ . It is known that Ω ( P̄ ) is a simple polytope with O ( |S||A| ) constraints [ Zimin and Neu , 2013 ] . When P is unknown , our algorithm uses an estimated transition P̄ as a proxy and searches for a “ good ” occupancy measure within Ω ( P̄ ) . More specifically , this is done by the classic Follow-the-Regularized-Leader ( FTRL ) framework which solves the following at the beginning of episode t : q̂t = argmin q∈Ω ( P̄ ) 〈 q , ∑ τ < t ̂̀ τ 〉 + φt ( q ) , ( 2 ) where ̂̀τ is some estimator for ` τ and φt is some regularizer . The learner ’ s policy πt is then defined through πt ( a|s ) ∝ q̂t ( s , a ) . Note that we have q̂t = qP̄ , πt but not necessarily q̂t = qt unless P̄ = P .

Summary: This paper presents an algorithm for episodic Markov decision processes with unknown losses and transition kernel. The paper proves that this algorithm achieves order sqrt(T) regret bounds when losses are chosen by an oblivious adversary and order log^2*(T) when losses are stochastic. This is the first algorithm to achieve such best of both works regret guarantees when transitions are unknown.  The key technical innovations are (1) the use of loss-shifting technique in the MDP setting, (2) running FTRL in phases where the empirical transition model is fixed per phase, (3) new biased loss estimators that combine importance weighting using upper occupancy measures with   additional bonus terms to promote exploration and (4) carefully tuned regularizers for FTRL.