Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses

√ T ) regret where T is the number of episodes , improving the Õ ( T 2/3 ) regret bound by [ 27 ] . When the number of states is infinite , under the assumption that the state-action values are linear in some low-dimensional features , we obtain Õ ( T 2/3 ) regret with the help of a simulator , matching the result of [ 24 ] while importantly removing the need of an exploratory policy that their algorithm requires . To our knowledge , this is the first algorithm with sublinear regret for linear function approximation with adversarial losses , bandit feedback , and no exploratory assumptions . Finally , we also discuss how to further improve the regret or remove the need of a simulator using dilated bonuses , when an exploratory policy is available.1 1 Introduction . Policy optimization methods are among the most widely-used methods in reinforcement learning . Its empirical success has been demonstrated in various domains such as computer games [ 26 ] and robotics [ 21 ] . However , due to its local-search nature , global optimality guarantees of policy optimization often rely on unrealistic assumptions to ensure global exploration ( see e.g. , [ 1 , 3 , 24 , 30 ] ) , making it theoretically less appealing compared to other methods . Motivated by this issue , a line of recent works [ 7 , 27 , 2 , 35 ] equip policy optimization with global exploration by adding exploration bonuses to the update , and prove favorable guarantees even without making extra exploratory assumptions . Moreover , they all demonstrate some robustness aspect of policy optimization ( such as being able to handle adversarial losses or a certain degree of model misspecification ) . Despite these important progresses , however , many limitations still exist , including worse regret rates comparing to the best value-based or model-based approaches [ 27 , 2 , 35 ] , or requiring full-information feedback on the entire loss function ( as opposed to the more realistic bandit feedback ) [ 7 ] . ∗Equal contribution . 1In an improved version of this paper , we show that under the linear MDP assumption , an exploratory policy is not even needed . See https : //arxiv.org/abs/2107.08346 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address these issues , in this work , we propose a new type of exploration bonuses called dilated bonuses , which satisfies a certain dilated Bellman equation and provably leads to improved exploration compared to existing works ( Section 3 ) . We apply this general idea to advance the state-of-the-art of policy optimization for learning finite-horizon episodic MDPs with adversarial losses and bandit feedback . More specifically , our main results are : • First , in the tabular setting , addressing the main open question left in [ 27 ] , we improve their Õ ( T 2/3 ) regret to the optimal Õ ( √ T ) regret . This shows that policy optimization , which performs local optimization , is as capable as other occupancy-measure-based global optimization algorithms [ 15 , 20 ] in terms of global exploration . Moreover , our algorithm is computationally more efficient than those global methods since they require solving some convex optimization in each episode . ( Section 4 ) • Second , to further deal with large-scale problems , we consider a linear function approximation setting where the state-action values are linear in some known low-dimensional features and also a simulator is available , the same setting considered by [ 24 ] . We obtain the same Õ ( T 2/3 ) regret while importantly removing the need of an exploratory policy that their algorithm requires . Unlike the tabular setting ( where we improve existing regret rates of policy optimization ) , note that researchers have not been able to show any sublinear regret for policy optimization without exploratory assumptions for this problem , which shows the critical role of our proposed dilated bonuses . In fact , there are simply no existing algorithms with sublinear regret at all for this setting , be it policy-optimization-type or not . This shows the advantage of policy optimization over other approaches , when combined with our dilated bonuses . ( Section 5 ) • Finally , while the main focus of our work is to show how dilated bonuses are able to provide global exploration , we also discuss their roles in improving the regret rate to Õ ( √ T ) in the linear setting above or removing the need of a simulator for the special case of linear MDPs ( with Õ ( T 6/7 ) regret ) , when an exploratory policy is available . ( Section 6 ) Related work . In the tabular setting , except for [ 27 ] , most algorithms apply the occupancymeasure-based framework to handle adversarial losses ( e.g. , [ 25 , 15 , 9 , 8 ] ) , which as mentioned is computationally expensive . For stochastic losses , there are many more different approaches such as model-based ones [ 13 , 10 , 5 , 12 , 34 ] and value-based ones [ 14 , 11 ] . Theoretical studies for linear function approximation have gained increasing interest recently [ 32 , 33 , 16 ] . Most of them study stochastic/stationary losses , with the exception of [ 24 , 7 ] . Our algorithm for the linear MDP setting bears some similarity to those of [ 2 , 35 ] which consider stationary losses . However , our algorithm and analysis are arguably simpler than theirs . Specifically , they divide the state space into a known part and an unknown part , with different exploration principle and bonus design for different parts . In contrast , we enjoy a unified bonus design for all states . Besides , in each episode , their algorithms first execute an exploratory policy ( from a policy cover ) , and then switch to the policy suggested by the policy optimization algorithm , which inevitably leads to linear regret when facing adversarial losses . 2 Problem Setting . We consider an MDP specified by a state space X ( possibly infinite ) , a finite action space A , and a transition function P with P ( ·|x , a ) specifying the distribution of the next state after taking action a in state x . In particular , we focus on the finite-horizon episodic setting in which X admits a layer structure and can be partitioned into X0 , X1 , . . . , XH for some fixed parameter H , where X0 contains only the initial state x0 , XH contains only the terminal state xH , and for any x ∈ Xh , h = 0 , . . . , H − 1 , P ( ·|x , a ) is supported on Xh+1 for all a ∈ A ( that is , transition is only possible from Xh to Xh+1 ) . An episode refers to a trajectory that starts from x0 and ends at xH following some series of actions and the transition dynamic . The MDP may be assigned with a loss function ` : X ×A→ [ 0 , 1 ] so that ` ( x , a ) specifies the loss suffered when selecting action a in state x . A policy π for the MDP is a mapping X → ∆ ( A ) , where ∆ ( A ) denotes the set of distributions over A and π ( a|x ) is the probability of choosing action a in state x . Given a loss function ` and a policy π , the expected total loss of π is given by V π ( x0 ; ` ) = E [ ∑H−1 h=0 ` ( xh , ah ) ∣∣ ah ∼ πt ( ·|xh ) , xh+1 ∼ P ( ·|xh , ah ) ] . It can also be defined via the Bellman equation involving the state value function V π ( x ; ` ) and the state-action value function Qπ ( x , a ; ` ) ( a.k.a . Q-function ) defined as below : V ( xH ; ` ) = 0 , Qπ ( x , a ; ` ) = ` ( x , a ) + Ex′∼P ( ·|x , a ) [ V π ( x′ ; ` ) ] , and V π ( x ; ` ) = Ea∼π ( ·|x ) [ Qπ ( x , a ; ` ) ] . We study online learning in such a finite-horizon MDP with unknown transition , bandit feedback , and adversarial losses . The learning proceeds through T episodes . Ahead of time , an adversary arbitrarily decides T loss functions ` 1 , . . . , ` T , without revealing them to the learner . Then in each episode t , the learner decides a policy πt based on all information received prior to this episode , executes πt starting from the initial state x0 , generates and observes a trajectory { ( xt , h , at , h , ` t ( xt , h , at , h ) ) } H−1h=0 . Importantly , the learner does not observe any other information about ` t ( a.k.a . bandit feedback ) .2 The goal of the learner is to minimize the regret , defined as Reg = T∑ t=1 V πtt ( x0 ) −min π T∑ t=1 V πt ( x0 ) , where we use V πt ( x ) as a shorthand for V π ( x ; ` t ) ( and similarly Qπt ( x , a ) as a shorthand for Qπ ( x , a ; ` t ) ) . Without further structures , the best existing regret bound is Õ ( H|X| √ |A|T ) [ 15 ] , with an extra √ X factor compared to the best existing lower bound [ 14 ] . Occupancy measures . For a policy π and a state x , we define qπ ( x ) to be the probability ( or probability measure when |X| is infinite ) of visiting state x within an episode when following π . When it is necessary to highlight the dependence on the transition , we write it as qP , π ( x ) . Further define qπ ( x , a ) = qπ ( x ) π ( a|x ) and qt ( x , a ) = qπt ( x , a ) . Finally , we use q ? as a shorthand for qπ ? where π ? ∈ argminπ ∑T t=1 V π t ( x0 ) is one of the optimal policies . Note that by definition , we have V π ( x0 ; ` ) = ∑ x , a q π ( x , a ) ` ( x , a ) . In fact , we will overload the notation and let V π ( x0 ; b ) = ∑ x , a q π ( x , a ) b ( x , a ) for any function b : X ×A→ R ( even though it might not correspond to a real loss function ) . Other notations . We denote by Et [ · ] and Vart [ · ] the expectation and variance conditioned on everything prior to episode t. For a matrix Σ and a vector z ( of appropriate dimension ) , ‖z‖Σ denotes the quadratic norm √ z > Σz . The notation Õ ( · ) hides all logarithmic factors . 3 Dilated Exploration Bonuses . In this section , we start with a general discussion on designing exploration bonuses ( not specific to policy optimization ) , and then introduce our new dilated bonuses for policy optimization . For simplicity , the exposition in this section assumes a finite state space , but the idea generalizes to an infinite state space . When analyzing the regret of an algorithm , very often we run into the following form : Reg = T∑ t=1 V πtt ( x0 ) − T∑ t=1 V π ? t ( x0 ) ≤ o ( T ) + T∑ t=1 ∑ x , a q ? ( x , a ) bt ( x , a ) = o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) , ( 1 ) for some function bt ( x , a ) usually related to some estimation error or variance that can be prohibitively large . For example , in policy optimization , the algorithm performs local search in each state essentially using a multi-armed bandit algorithm and treating Qπt ( x , a ) as the loss of action a in state x . Since Qπt ( x , a ) is unknown , however , the algorithm has to use some estimator of Qπt ( x , a ) instead , whose bias and variance both contribute to the bt function . Usually , bt ( x , a ) is large for a rarely-visited state-action pair ( x , a ) and is inversely related to qt ( x , a ) , which is exactly why most analysis relies 2Full-information feedback , on the other hand , refers to the easier setting where the entire loss function ` t is revealed to the learner at the end of episode t. on the assumption that some distribution mismatch coefficient related to q ? ( x , a ) /qt ( x , a ) is bounded ( see e.g. , [ 3 , 31 ] ) . On the other hand , an important observation is that while V π ? ( x0 ; bt ) can be prohibitively large , its counterpart with respect to the learner ’ s policy V πt ( x0 ; bt ) is usually nicely bounded . For example , if bt ( x , a ) is inversely related to qt ( x , a ) as mentioned , then V πt ( x0 ; bt ) = ∑ x , a qt ( x , a ) bt ( x , a ) is small no matter how small qt ( x , a ) could be for some ( x , a ) . This observation , together with the linearity property V π ( x0 ; ` t − bt ) = V π ( x0 ; ` t ) − V π ( x0 ; bt ) , suggests that we treat ` t − bt as the loss function of the problem , or in other words , add a ( negative ) bonus to each state-action pair , which intuitively encourages exploration due to underestimation . Indeed , assuming for a moment that Eq . ( 1 ) still roughly holds even if we treat ` t − bt as the loss function : T∑ t=1 V πt ( x0 ; ` t − bt ) − T∑ t=1 V π ? ( x0 ; ` t − bt ) . o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) . ( 2 ) Then by linearity and rearranging , we have Reg = T∑ t=1 V πtt ( x0 ) − T∑ t=1 V π ? t ( x0 ) . o ( T ) + T∑ t=1 V πt ( x0 ; bt ) . ( 3 ) Due to the switch from π ? to πt in the last term compared to Eq . ( 1 ) , this is usually enough to prove a desirable regret bound without making extra assumptions . The caveat of this discussion is the assumption of Eq . ( 2 ) . Indeed , after adding the bonuses , which itself contributes some more bias and variance , one should expect that bt on the right-hand side of Eq . ( 2 ) becomes something larger , breaking the desired cancellation effect to achieve Eq . ( 3 ) . Indeed , the definition of bt essentially becomes circular in this sense . Dilated Bonuses for Policy Optimization To address this issue , we take a closer look at the policy optimization algorithm specifically . As mentioned , policy optimization decomposes the problem into individual multi-armed bandit problems in each state and then performs local optimization . This is based on the well-known performance difference lemma [ 17 ] : Reg = ∑ x q ? ( x ) T∑ t=1 ∑ a ( πt ( a|x ) − π ? ( a|x ) ) Qπtt ( x , a ) , showing that in each state x , the learner is facing a bandit problem with Qπtt ( x , a ) being the loss for action a . Correspondingly , incorporating the bonuses bt for policy optimization means subtracting the bonus Qπt ( x , a ; bt ) from Qπtt ( x , a ) for each action a in each state x . Recall that Q πt ( x , a ; bt ) satisfies the Bellman equation Qπt ( x , a ; bt ) = bt ( x , a ) + Ex′∼P ( ·|x , a ) Ea′∼πt ( ·|x′ ) [ Bt ( x′ , a′ ) ] . To resolve the issue mentioned earlier , we propose to replace this bonus function Qπt ( x , a ; bt ) with its dilated version Bt ( s , a ) satisfying the following dilated Bellman equation : Bt ( x , a ) = bt ( x , a ) + ( 1 + 1 H ) Ex′∼P ( ·|x , a ) Ea′∼πt ( ·|x′ ) [ Bt ( x ′ , a′ ) ] ( 4 ) ( with Bt ( xH , a ) = 0 for all a ) . The only difference compared to the standard Bellman equation is the extra ( 1 + 1H ) factor , which slightly increases the weight for deeper layers and thus intuitively induces more exploration for those layers . Due to the extra bonus compared to Qπt ( x , a ; bt ) , the regret bound also increases accordingly . In all our applications , this extra amount of regret turns out to be of the form 1H ∑T t=1 ∑ x , a q ? ( x ) πt ( a|x ) Bt ( x , a ) , leading to ∑ x q ? ( x ) T∑ t=1 ∑ a ( πt ( a|x ) − π ? ( a|x ) ) ( Qπtt ( x , a ) −Bt ( x , a ) ) ≤ o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) + 1 H T∑ t=1 ∑ x , a q ? ( x ) πt ( a|x ) Bt ( x , a ) . ( 5 ) With some direct calculation , one can show that this is enough to show a regret bound that is only a constant factor larger than the desired bound in Eq . ( 3 ) ! This is summarized in the following lemma . Lemma 3.1 . If Eq . ( 5 ) holds with Bt defined in Eq . ( 4 ) , then Reg ≤ o ( T ) + 3 ∑T t=1 V πt ( x0 ; bt ) . The high-level idea of the proof is to show that the bonuses added to a layer h is enough to cancel the large bias/variance term ( including those coming from the bonus itself ) from layer h+ 1 . Therefore , cancellation happens in a layer-by-layer manner except for layer 0 , where the total amount of bonus can be shown to be at most ( 1 + 1H ) H ∑T t=1 V πt ( x0 ; bt ) ≤ 3 ∑T t=1 V πt ( x0 ; bt ) . Recalling again that V πt ( x0 ; bt ) is usually nicely bounded , we thus arrive at a favorable regret guarantee without making extra assumptions . Of course , since the transition is unknown , we can not compute Bt exactly . However , Lemma 3.1 is robust enough to handle either a good approximate version of Bt ( see Lemma B.1 ) or a version where Eq . ( 4 ) and Eq . ( 5 ) only hold in expectation ( see Lemma B.2 ) , which is enough for us to handle unknown transition . In the next three sections , we apply this general idea to different settings , showing what bt and Bt are concretely in each case .

Summary: The paper studies the regret of policy optimization for adversarial MDPs with bandit feedback. It develops a general solution that adds dilated bonuses to the policy update for exploration and applies the algorithm to the tabular, linear-Q, and linear MDP settings.   It shows that such a solution improves and generalizes the state-of-the-art.