Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses

√ T ) regret where T is the number of episodes , improving the Õ ( T 2/3 ) regret bound by [ 27 ] . When the number of states is infinite , under the assumption that the state-action values are linear in some low-dimensional features , we obtain Õ ( T 2/3 ) regret with the help of a simulator , matching the result of [ 24 ] while importantly removing the need of an exploratory policy that their algorithm requires . To our knowledge , this is the first algorithm with sublinear regret for linear function approximation with adversarial losses , bandit feedback , and no exploratory assumptions . Finally , we also discuss how to further improve the regret or remove the need of a simulator using dilated bonuses , when an exploratory policy is available.1 1 Introduction . Policy optimization methods are among the most widely-used methods in reinforcement learning . Its empirical success has been demonstrated in various domains such as computer games [ 26 ] and robotics [ 21 ] . However , due to its local-search nature , global optimality guarantees of policy optimization often rely on unrealistic assumptions to ensure global exploration ( see e.g. , [ 1 , 3 , 24 , 30 ] ) , making it theoretically less appealing compared to other methods . Motivated by this issue , a line of recent works [ 7 , 27 , 2 , 35 ] equip policy optimization with global exploration by adding exploration bonuses to the update , and prove favorable guarantees even without making extra exploratory assumptions . Moreover , they all demonstrate some robustness aspect of policy optimization ( such as being able to handle adversarial losses or a certain degree of model misspecification ) . Despite these important progresses , however , many limitations still exist , including worse regret rates comparing to the best value-based or model-based approaches [ 27 , 2 , 35 ] , or requiring full-information feedback on the entire loss function ( as opposed to the more realistic bandit feedback ) [ 7 ] . ∗Equal contribution . 1In an improved version of this paper , we show that under the linear MDP assumption , an exploratory policy is not even needed . See https : //arxiv.org/abs/2107.08346 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address these issues , in this work , we propose a new type of exploration bonuses called dilated bonuses , which satisfies a certain dilated Bellman equation and provably leads to improved exploration compared to existing works ( Section 3 ) . We apply this general idea to advance the state-of-the-art of policy optimization for learning finite-horizon episodic MDPs with adversarial losses and bandit feedback . More specifically , our main results are : • First , in the tabular setting , addressing the main open question left in [ 27 ] , we improve their Õ ( T 2/3 ) regret to the optimal Õ ( √ T ) regret . This shows that policy optimization , which performs local optimization , is as capable as other occupancy-measure-based global optimization algorithms [ 15 , 20 ] in terms of global exploration . Moreover , our algorithm is computationally more efficient than those global methods since they require solving some convex optimization in each episode . ( Section 4 ) • Second , to further deal with large-scale problems , we consider a linear function approximation setting where the state-action values are linear in some known low-dimensional features and also a simulator is available , the same setting considered by [ 24 ] . We obtain the same Õ ( T 2/3 ) regret while importantly removing the need of an exploratory policy that their algorithm requires . Unlike the tabular setting ( where we improve existing regret rates of policy optimization ) , note that researchers have not been able to show any sublinear regret for policy optimization without exploratory assumptions for this problem , which shows the critical role of our proposed dilated bonuses . In fact , there are simply no existing algorithms with sublinear regret at all for this setting , be it policy-optimization-type or not . This shows the advantage of policy optimization over other approaches , when combined with our dilated bonuses . ( Section 5 ) • Finally , while the main focus of our work is to show how dilated bonuses are able to provide global exploration , we also discuss their roles in improving the regret rate to Õ ( √ T ) in the linear setting above or removing the need of a simulator for the special case of linear MDPs ( with Õ ( T 6/7 ) regret ) , when an exploratory policy is available . ( Section 6 ) Related work . In the tabular setting , except for [ 27 ] , most algorithms apply the occupancymeasure-based framework to handle adversarial losses ( e.g. , [ 25 , 15 , 9 , 8 ] ) , which as mentioned is computationally expensive . For stochastic losses , there are many more different approaches such as model-based ones [ 13 , 10 , 5 , 12 , 34 ] and value-based ones [ 14 , 11 ] . Theoretical studies for linear function approximation have gained increasing interest recently [ 32 , 33 , 16 ] . Most of them study stochastic/stationary losses , with the exception of [ 24 , 7 ] . Our algorithm for the linear MDP setting bears some similarity to those of [ 2 , 35 ] which consider stationary losses . However , our algorithm and analysis are arguably simpler than theirs . Specifically , they divide the state space into a known part and an unknown part , with different exploration principle and bonus design for different parts . In contrast , we enjoy a unified bonus design for all states . Besides , in each episode , their algorithms first execute an exploratory policy ( from a policy cover ) , and then switch to the policy suggested by the policy optimization algorithm , which inevitably leads to linear regret when facing adversarial losses . 2 Problem Setting . We consider an MDP specified by a state space X ( possibly infinite ) , a finite action space A , and a transition function P with P ( ·|x , a ) specifying the distribution of the next state after taking action a in state x . In particular , we focus on the finite-horizon episodic setting in which X admits a layer structure and can be partitioned into X0 , X1 , . . . , XH for some fixed parameter H , where X0 contains only the initial state x0 , XH contains only the terminal state xH , and for any x ∈ Xh , h = 0 , . . . , H − 1 , P ( ·|x , a ) is supported on Xh+1 for all a ∈ A ( that is , transition is only possible from Xh to Xh+1 ) . An episode refers to a trajectory that starts from x0 and ends at xH following some series of actions and the transition dynamic . The MDP may be assigned with a loss function ` : X ×A→ [ 0 , 1 ] so that ` ( x , a ) specifies the loss suffered when selecting action a in state x . A policy π for the MDP is a mapping X → ∆ ( A ) , where ∆ ( A ) denotes the set of distributions over A and π ( a|x ) is the probability of choosing action a in state x . Given a loss function ` and a policy π , the expected total loss of π is given by V π ( x0 ; ` ) = E [ ∑H−1 h=0 ` ( xh , ah ) ∣∣ ah ∼ πt ( ·|xh ) , xh+1 ∼ P ( ·|xh , ah ) ] . It can also be defined via the Bellman equation involving the state value function V π ( x ; ` ) and the state-action value function Qπ ( x , a ; ` ) ( a.k.a . Q-function ) defined as below : V ( xH ; ` ) = 0 , Qπ ( x , a ; ` ) = ` ( x , a ) + Ex′∼P ( ·|x , a ) [ V π ( x′ ; ` ) ] , and V π ( x ; ` ) = Ea∼π ( ·|x ) [ Qπ ( x , a ; ` ) ] . We study online learning in such a finite-horizon MDP with unknown transition , bandit feedback , and adversarial losses . The learning proceeds through T episodes . Ahead of time , an adversary arbitrarily decides T loss functions ` 1 , . . . , ` T , without revealing them to the learner . Then in each episode t , the learner decides a policy πt based on all information received prior to this episode , executes πt starting from the initial state x0 , generates and observes a trajectory { ( xt , h , at , h , ` t ( xt , h , at , h ) ) } H−1h=0 . Importantly , the learner does not observe any other information about ` t ( a.k.a . bandit feedback ) .2 The goal of the learner is to minimize the regret , defined as Reg = T∑ t=1 V πtt ( x0 ) −min π T∑ t=1 V πt ( x0 ) , where we use V πt ( x ) as a shorthand for V π ( x ; ` t ) ( and similarly Qπt ( x , a ) as a shorthand for Qπ ( x , a ; ` t ) ) . Without further structures , the best existing regret bound is Õ ( H|X| √ |A|T ) [ 15 ] , with an extra √ X factor compared to the best existing lower bound [ 14 ] . Occupancy measures . For a policy π and a state x , we define qπ ( x ) to be the probability ( or probability measure when |X| is infinite ) of visiting state x within an episode when following π . When it is necessary to highlight the dependence on the transition , we write it as qP , π ( x ) . Further define qπ ( x , a ) = qπ ( x ) π ( a|x ) and qt ( x , a ) = qπt ( x , a ) . Finally , we use q ? as a shorthand for qπ ? where π ? ∈ argminπ ∑T t=1 V π t ( x0 ) is one of the optimal policies . Note that by definition , we have V π ( x0 ; ` ) = ∑ x , a q π ( x , a ) ` ( x , a ) . In fact , we will overload the notation and let V π ( x0 ; b ) = ∑ x , a q π ( x , a ) b ( x , a ) for any function b : X ×A→ R ( even though it might not correspond to a real loss function ) . Other notations . We denote by Et [ · ] and Vart [ · ] the expectation and variance conditioned on everything prior to episode t. For a matrix Σ and a vector z ( of appropriate dimension ) , ‖z‖Σ denotes the quadratic norm √ z > Σz . The notation Õ ( · ) hides all logarithmic factors . 3 Dilated Exploration Bonuses . In this section , we start with a general discussion on designing exploration bonuses ( not specific to policy optimization ) , and then introduce our new dilated bonuses for policy optimization . For simplicity , the exposition in this section assumes a finite state space , but the idea generalizes to an infinite state space . When analyzing the regret of an algorithm , very often we run into the following form : Reg = T∑ t=1 V πtt ( x0 ) − T∑ t=1 V π ? t ( x0 ) ≤ o ( T ) + T∑ t=1 ∑ x , a q ? ( x , a ) bt ( x , a ) = o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) , ( 1 ) for some function bt ( x , a ) usually related to some estimation error or variance that can be prohibitively large . For example , in policy optimization , the algorithm performs local search in each state essentially using a multi-armed bandit algorithm and treating Qπt ( x , a ) as the loss of action a in state x . Since Qπt ( x , a ) is unknown , however , the algorithm has to use some estimator of Qπt ( x , a ) instead , whose bias and variance both contribute to the bt function . Usually , bt ( x , a ) is large for a rarely-visited state-action pair ( x , a ) and is inversely related to qt ( x , a ) , which is exactly why most analysis relies 2Full-information feedback , on the other hand , refers to the easier setting where the entire loss function ` t is revealed to the learner at the end of episode t. on the assumption that some distribution mismatch coefficient related to q ? ( x , a ) /qt ( x , a ) is bounded ( see e.g. , [ 3 , 31 ] ) . On the other hand , an important observation is that while V π ? ( x0 ; bt ) can be prohibitively large , its counterpart with respect to the learner ’ s policy V πt ( x0 ; bt ) is usually nicely bounded . For example , if bt ( x , a ) is inversely related to qt ( x , a ) as mentioned , then V πt ( x0 ; bt ) = ∑ x , a qt ( x , a ) bt ( x , a ) is small no matter how small qt ( x , a ) could be for some ( x , a ) . This observation , together with the linearity property V π ( x0 ; ` t − bt ) = V π ( x0 ; ` t ) − V π ( x0 ; bt ) , suggests that we treat ` t − bt as the loss function of the problem , or in other words , add a ( negative ) bonus to each state-action pair , which intuitively encourages exploration due to underestimation . Indeed , assuming for a moment that Eq . ( 1 ) still roughly holds even if we treat ` t − bt as the loss function : T∑ t=1 V πt ( x0 ; ` t − bt ) − T∑ t=1 V π ? ( x0 ; ` t − bt ) . o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) . ( 2 ) Then by linearity and rearranging , we have Reg = T∑ t=1 V πtt ( x0 ) − T∑ t=1 V π ? t ( x0 ) . o ( T ) + T∑ t=1 V πt ( x0 ; bt ) . ( 3 ) Due to the switch from π ? to πt in the last term compared to Eq . ( 1 ) , this is usually enough to prove a desirable regret bound without making extra assumptions . The caveat of this discussion is the assumption of Eq . ( 2 ) . Indeed , after adding the bonuses , which itself contributes some more bias and variance , one should expect that bt on the right-hand side of Eq . ( 2 ) becomes something larger , breaking the desired cancellation effect to achieve Eq . ( 3 ) . Indeed , the definition of bt essentially becomes circular in this sense . Dilated Bonuses for Policy Optimization To address this issue , we take a closer look at the policy optimization algorithm specifically . As mentioned , policy optimization decomposes the problem into individual multi-armed bandit problems in each state and then performs local optimization . This is based on the well-known performance difference lemma [ 17 ] : Reg = ∑ x q ? ( x ) T∑ t=1 ∑ a ( πt ( a|x ) − π ? ( a|x ) ) Qπtt ( x , a ) , showing that in each state x , the learner is facing a bandit problem with Qπtt ( x , a ) being the loss for action a . Correspondingly , incorporating the bonuses bt for policy optimization means subtracting the bonus Qπt ( x , a ; bt ) from Qπtt ( x , a ) for each action a in each state x . Recall that Q πt ( x , a ; bt ) satisfies the Bellman equation Qπt ( x , a ; bt ) = bt ( x , a ) + Ex′∼P ( ·|x , a ) Ea′∼πt ( ·|x′ ) [ Bt ( x′ , a′ ) ] . To resolve the issue mentioned earlier , we propose to replace this bonus function Qπt ( x , a ; bt ) with its dilated version Bt ( s , a ) satisfying the following dilated Bellman equation : Bt ( x , a ) = bt ( x , a ) + ( 1 + 1 H ) Ex′∼P ( ·|x , a ) Ea′∼πt ( ·|x′ ) [ Bt ( x ′ , a′ ) ] ( 4 ) ( with Bt ( xH , a ) = 0 for all a ) . The only difference compared to the standard Bellman equation is the extra ( 1 + 1H ) factor , which slightly increases the weight for deeper layers and thus intuitively induces more exploration for those layers . Due to the extra bonus compared to Qπt ( x , a ; bt ) , the regret bound also increases accordingly . In all our applications , this extra amount of regret turns out to be of the form 1H ∑T t=1 ∑ x , a q ? ( x ) πt ( a|x ) Bt ( x , a ) , leading to ∑ x q ? ( x ) T∑ t=1 ∑ a ( πt ( a|x ) − π ? ( a|x ) ) ( Qπtt ( x , a ) −Bt ( x , a ) ) ≤ o ( T ) + T∑ t=1 V π ? ( x0 ; bt ) + 1 H T∑ t=1 ∑ x , a q ? ( x ) πt ( a|x ) Bt ( x , a ) . ( 5 ) With some direct calculation , one can show that this is enough to show a regret bound that is only a constant factor larger than the desired bound in Eq . ( 3 ) ! This is summarized in the following lemma . Lemma 3.1 . If Eq . ( 5 ) holds with Bt defined in Eq . ( 4 ) , then Reg ≤ o ( T ) + 3 ∑T t=1 V πt ( x0 ; bt ) . The high-level idea of the proof is to show that the bonuses added to a layer h is enough to cancel the large bias/variance term ( including those coming from the bonus itself ) from layer h+ 1 . Therefore , cancellation happens in a layer-by-layer manner except for layer 0 , where the total amount of bonus can be shown to be at most ( 1 + 1H ) H ∑T t=1 V πt ( x0 ; bt ) ≤ 3 ∑T t=1 V πt ( x0 ; bt ) . Recalling again that V πt ( x0 ; bt ) is usually nicely bounded , we thus arrive at a favorable regret guarantee without making extra assumptions . Of course , since the transition is unknown , we can not compute Bt exactly . However , Lemma 3.1 is robust enough to handle either a good approximate version of Bt ( see Lemma B.1 ) or a version where Eq . ( 4 ) and Eq . ( 5 ) only hold in expectation ( see Lemma B.2 ) , which is enough for us to handle unknown transition . In the next three sections , we apply this general idea to different settings , showing what bt and Bt are concretely in each case .

Summary: This paper makes several contributions:  	 1. In the episodic tabular setting, with adversarial rewards, unknown MDP dynamics and bandit feedback, it provides the first algorithm that achieves O(\sqrt T) regret bound using the regret decomposition approach, which is more computationally cheap than the approach presented in Jin et al. Typically result obtained by using this approach scales with the inverse of the visitation probability. However, here the results don't have this drawback.   	 2. For the linear case, the paper considers the setting with an infinite set of states and the algorithm has access to the simulator of the dynamics of MDP. Then it provides the algorithm that obtains T^2/3 regret without assuming that the smallest eigenvalue of the covariance matrix \Sigma_h is at least \lambda_min and T^1/2 taken this assumption. The latter improves the previous result in the considered setting, which was T^2/3 regret, assuming that the smallest eigenvalue of \Sigma_h is bounded away from zero. The only drawback is that the algorithm is more computationally expensive since it requires O(T^H) number of calls to the simulator. 	 3. The third result presented in the paper is in the same setting as above but without access to the simulator of MDP, which is a very challenging setting. The obtained regret bound is T^6/7, and it makes use of the assumption on the smallest eigenvalue of \Sigma_h.  4. All three results from above make use of dilated bonuses, which is the main contribution of this paper.  The use of the dilated bonuses that decrease the contribution to the regret, induced by the term appearing from the mismatch of q^*(x) and q(x), shows up in the analysis. To the best of my knowledge, this idea has not been considered before, and I think this idea is beneficial for future studies.