Twice regularized MDPs and the equivalence between robustness and regularization

1 Introduction . MDPs provide a practical framework for solving sequential decision problems under uncertainty [ 30 ] . However , the chosen strategy can be very sensitive to sampling errors or inaccurate model estimates . This can lead to complete failure in common situations where the model parameters vary adversarially or are simply unknown [ 22 ] . Robust MDPs aim to mitigate such sensitivity by assuming that the transition and/or reward function ( P , r ) varies arbitrarily inside a given uncertainty set U [ 17 , 27 ] . In this setting , an optimal solution maximizes a performance measure under the worst-case parameters . It can be thought of as a dynamic zero-sum game with an agent choosing the best action while Nature imposes it the most adversarial model . As such , solving robust MDPs involves max-min problems , which can be computationally challenging and limits scalability . In recent years , several methods have been developed to alleviate the computational concerns raised by robust reinforcement learning ( RL ) . Apart from [ 23 , 24 ] that consider specific types of coupled uncertainty sets , all rely on a rectangularity assumption without which the problem can be NP-hard [ 2 , 40 ] . This assumption is key to deriving tractable solvers of robust MDPs such as robust value iteration [ 2 , 11 ] or more general robust modified policy iteration ( MPI ) [ 18 ] . Yet , reducing time complexity in robust Bellman updates remains challenging and is still researched today [ 15 , 11 ] . At the same time , the empirical success of regularization in policy search methods has motivated a wide range of algorithms with diverse motivations such as improved exploration [ 12 , 20 ] or stability [ 34 , 13 ] . Geist et al . [ 10 ] proposed a unified view from which many existing algorithms can be derived . Their regularized MDP formalism opens the path to error propagation analysis ∗Contact author : estherderman @ campus.technion.ac.il 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in approximate MPI [ 33 ] and leads to the same bounds as for standard MDPs . Nevertheless , as we further show in Sec . 3 , policy regularization accounts for reward uncertainty only : it does not encompass uncertainty in the model dynamics . Despite a vast literature on how regularized policy search works and convergence rates analysis [ 36 , 5 ] , little attention has been given to understanding why it can generate strategies that are robust to external perturbations [ 13 ] . To our knowledge , the only works that relate robustness to regularization in RL are [ 6 , 16 , 9 ] . Derman & Mannor [ 6 ] employ a distributionally robust optimization approach to regularize an empirical value function . Unfortunately , computing this empirical value necessitates several policy evaluation procedures , which is quickly unpractical . Husain et al . [ 16 ] provide a dual relationship with robust MDPs under uncertain reward . Their duality result applies to general regularization methods and gives a robust interpretation of soft-actor-critic [ 13 ] . Although these two works justify the use of regularization for ensuring robustness , they do not enclose any algorithmic novelty . Similarly , Eysenbach & Levine [ 9 ] specifically focus on maximum entropy methods and relate them to either reward or transition robustness . We shall further detail on these most related studies in Sec . 7 . The robustness-regularization duality is well established in statistical learning theory [ 41 , 35 , 19 ] , as opposed to RL theory . In fact , standard setups such as classification or regression may be considered as single-stage decision-making problems , i.e. , one-step MDPs , a particular case of RL setting . Extending this robustness-regularization duality to RL would yield cheaper learning methods with robustness guarantees . As such , we introduce a regularization function ΩU that depends on the uncertainty set U and is defined over both policy and value spaces , thus inducing a twice regularized Bellman operator ( see Sec . 5 ) . We show that this regularizer yields an equivalence of the form vπ , U = vπ , ΩU , where vπ , U is the robust value function for policy π and vπ , ΩU the regularized one . This equivalence is derived through the objective function each value optimizes . More concretely , we formulate the robust value function vπ , U as an optimal solution of the robust optimization problem : max v∈RS 〈v , µ0〉 s. t. v ≤ inf ( P , r ) ∈U Tπ ( P , r ) v , ( RO ) where Tπ ( P , r ) is the evaluation Bellman operator [ 30 ] . Then , we show that vπ , U is also an optimal solution of the convex ( non-robust ) optimization problem : max v∈RS 〈v , µ0〉 s. t. v ≤ Tπ ( P0 , r0 ) v − ΩU ( π , v ) , ( CO ) where ( P0 , r0 ) is the nominal model . This establishes equivalence between the two optimization problems . Moreover , the inequality constraint of ( CO ) enables to derive a twice regularized ( R2 ) Bellman operator defined according to ΩU , a policy and value regularizer . For ball-constrained uncertainty sets , ΩU has an explicit form and under mild conditions , the corresponding R2 Bellman operators are contracting . The equivalence between the two problems ( RO ) and ( CO ) together with the contraction properties of R2 Bellman operators enable to circumvent robust optimization problems at each Bellman update . As such , it alleviates robust planning and learning algorithms by reducing them to regularized ones , which are known to be as complex as classical methods . To summarize , we make the following contributions : ( i ) We show that regularized MDPs are a specific instance of robust MDPs with uncertain reward . Besides formalizing a general connection between the two settings , our result enables to explicit the uncertainty sets induced by standard regularizers . ( ii ) We extend this duality to MDPs with uncertain transition and provide the first regularizer that recovers robust MDPs with s-rectangular balls and arbitrary norm . ( iii ) We introduce twice regularized MDPs ( R2 MDPs ) that apply both policy and value regularization to retrieve robust MDPs . We establish contraction of the corresponding Bellman operators . This leads us to proposing an R2 MPI algorithm with similar time complexity as vanilla MPI , thus opening new perspectives towards practical and scalable robust RL . Notations . We designate the extended reals by R : = { −∞ , ∞ } . Given a finite set Z , the class of real-valued functions ( resp . probability distributions ) over Z is denoted by RZ ( resp . ∆Z ) , while the constant function equal to 1 over Z is denoted by 1Z . Similarly , for any set X , ∆XZ denotes the class of functions defined over X and valued in ∆Z . The inner product of two functions a , b ∈ RZ is defined as 〈a , b〉 : = ∑ z∈Z a ( z ) b ( z ) , which induces the ` 2-norm ‖a‖ : = √ 〈a , a〉 . The ` 2-norm coincides with its dual norm , i.e. , ‖a‖ = max‖b‖≤1〈a , b〉 = : ‖a‖∗ . Let a function f : RZ → R. The Legendre-Fenchel transform ( or convex conjugate ) of f is f∗ ( y ) : = maxa∈RZ { 〈a , y〉 − f ( a ) } . Given a set Z ⊆ RZ , the characteristic function δZ : RZ → R is δZ ( a ) = 0 if a ∈ Z ; +∞ otherwise . The Legendre-Fenchel transform of δZ is the support function σZ ( y ) = maxa∈Z〈a , y〉 [ 3 , Ex . 1.6.1 ] . 2 Preliminaries . This section describes the background material that we use throughout our work . Firstly , we recall useful properties in convex analysis . Secondly , we address classical discounted MDPs and their linear program ( LP ) formulation . Thirdly , we briefly detail on regularized MDPs and the associated operators and lastly , we focus on the robust MDP setting . Convex Analysis . Let Ω : ∆Z → R be a strongly convex function . Throughout this study , the function Ω plays the role of a policy and/or value regularization function . Its Legendre-Fenchel transform Ω∗ satisfies several smoothness properties , hence its alternative name `` smoothed max operator '' [ 25 ] . Our work makes use of the following result [ 14 , 25 ] . Proposition 2.1 . Given Ω : ∆Z → R strongly convex , the following properties hold : ( i ) ∇Ω∗ is Lipschitz and satisfies∇Ω∗ ( y ) = arg maxa∈∆Z 〈a , y〉 − Ω ( a ) , ∀y ∈ R Z . ( ii ) For any c ∈ R , y ∈ RZ , Ω∗ ( y+c1Z ) = Ω∗ ( y ) + c. ( iii ) The Legendre-Fenchel transform Ω∗ is non-decreasing . Discounted MDPs and LP formulation . Consider an infinite horizon MDP ( S , A , µ0 , γ , P , r ) with S and A finite state and action spaces respectively , 0 < µ0 ∈ ∆S an initial state distribution and γ ∈ ( 0 , 1 ) a discount factor . Denoting X : = S ×A , P ∈ ∆XS is a transition kernel mapping each state-action pair to a probability distribution over S and r ∈ RX is a reward function . A policy π ∈ ∆SA maps any state s ∈ S to an action distribution πs ∈ ∆A , and we evaluate its performance through the following measure : ρ ( π ) : = E [ ∞∑ t=0 γtr ( st , at ) ∣∣∣∣ µ0 , π , P ] = 〈vπ ( P , r ) , µ0〉 . ( 1 ) Here , the expectation is conditioned on the process distribution determined by µ0 , π and P , and for all s ∈ S , vπ ( P , r ) ( s ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , π , P ] is the value function at state s. Maximizing ( 1 ) defines the standard RL objective , which can be solved thanks to the Bellman operators : Tπ ( P , r ) v : = r π + γPπv ∀v ∈ RS , π ∈ ∆SA , T ( P , r ) v : = max π∈∆SA Tπ ( P , r ) v ∀v ∈ R S , G ( P , r ) ( v ) : = { π ∈ ∆SA : Tπ ( P , r ) v = T ( P , r ) v } ∀v ∈ R S , where rπ : = [ 〈πs , r ( s , · ) 〉 ] s∈S and Pπ = [ Pπ ( s′|s ) ] s′ , s∈S with Pπ ( s′|s ) : = 〈πs , P ( s′|s , · ) 〉 . Both Tπ ( P , r ) and T ( P , r ) are γ-contractions with respect to ( w.r.t . ) the supremum norm , so each admits a unique fixed point vπ ( P , r ) and v ∗ ( P , r ) , respectively . The set of greedy policies w.r.t . value v defines G ( P , r ) ( v ) , and any policy π ∈ G ( P , r ) ( v∗ ( P , r ) ) is optimal [ 30 ] . For all v ∈ R S , the associated function q ∈ RX is given by q ( s , a ) = r ( s , a ) + γ〈P ( ·|s , a ) , v〉 ∀ ( s , a ) ∈ X . In particular , the fixed point vπ ( P , r ) satisfies v π ( P , r ) = 〈πs , q π ( P , r ) ( s , · ) 〉 where q π ( P , r ) is its associated q-function . The problem in ( 1 ) can also be formulated as an LP [ 30 ] . Given a policy π ∈ ∆SA , we characterize its performance ρ ( π ) by the following v-LP [ 30 , 26 ] : min v∈RS 〈v , µ0〉 subject to ( s.t . ) v ≥ rπ + γPπv . ( Pπ ) This primal objective provides a policy view on the problem . Alternatively , one may take a state visitation perspective by studying the dual objective instead : max µ∈RS 〈rπ , µ〉 s. t. µ ≥ 0 and ( IdRS − γPπ∗ ) µ = µ0 , ( Dπ ) where Pπ∗ is the adjoint policy transition operator 2 : [ Pπ∗ µ ] ( s ) : = ∑ s̄∈S P π ( s|s̄ ) µ ( s̄ ) ∀µ ∈ RS , and IdS is the identity function in RS . Let I ( s′|s , a ) : = δs′=s ∀ ( s , a , s′ ) ∈ X ×S the trivial transition matrix and define its adjoint transition operator as I∗µ ( s ) : = ∑ ( s̄ , ā ) ∈X I ( s|s̄ , ā ) µ ( s̄ , ā ) ∀s ∈ 2It is the adjoint operator of Pπ in the sense that 〈Pπv , v′〉 = 〈v , Pπ∗ v′〉 ∀v , v′ ∈ RS . S. The correspondence between occupancy measures and policies lies in the one-to-one mapping µ 7→ µ ( · , · ) I∗µ ( · ) = : πµ and its inverse π 7→ µπ given by µπ ( s , a ) : = ∞∑ t=0 γtP ( st = s , at = a ∣∣∣∣µ0 , π , P ) ∀ ( s , a ) ∈ X . As such , one can interchangeably work with the primal LP ( Pπ ) or the dual ( Dπ ) . Regularized MDPs . A regularized MDP is a tuple ( S , A , µ0 , γ , P , r , Ω ) with ( S , A , µ0 , γ , P , r ) an infinite horizon MDP as above , and Ω : = ( Ωs ) s∈S a finite set of functions such that for all s ∈ S , Ωs : ∆A → R is strongly convex . Each function Ωs plays the role of a policy regularizer Ωs ( πs ) . With a slight abuse of notation , we shall denote by Ω ( π ) : = ( Ωs ( πs ) ) s∈S the family of state-dependent regularizers.3 The regularized Bellman evaluation operator is given by [ Tπ , Ω ( P , r ) v ] ( s ) : = T π ( P , r ) v ( s ) − Ωs ( πs ) ∀v ∈ R S , s ∈ S , and the regularized Bellman optimality operator by T ∗ , Ω ( P , r ) v : = maxπ∈∆SA T π , Ω ( P , r ) v ∀v ∈ R S [ 10 ] . The unique fixed point of Tπ , Ω ( P , r ) ( respectively T ∗ , Ω ( P , r ) ) is denoted by v π , Ω ( P , r ) ( resp . v ∗ , Ω ( P , r ) ) and defines the regularized value function ( resp . regularized optimal value function ) . Although the regularized MDP formalism stems from the aforementioned Bellman operators in [ 10 ] , it turns out that regularized MDPs are MDPs with modified reward . Indeed , for any policy π ∈ ∆SA , the regularized value function is vπ , Ω ( P , r ) = ( IS − γP π ) −1 ( rπ − Ω ( π ) ) , which corresponds to a non-regularized value with expected reward r̃π : = rπ − Ω ( π ) . Note that the modified reward r̃π ( s ) is no longer linear in πs because of the strong convexity of Ωs . Also , this modification does not apply to the reward function r but only to its expectation rπ , as we can not regularize the original reward without making it policy-independent . Robust MDPs . In general , the MDP model is not explicitly known but rather estimated from sampled trajectories . As this may result in over-sensitive outcome [ 22 ] , robust MDPs reduce such performance variation . Formally , a robust MDP ( S , A , µ0 , γ , U ) is an MDP with uncertain model belonging to U : = P ×R , i.e. , uncertain transition P ∈ P ⊆ ∆XS and reward r ∈ R ⊆ R X [ 17 , 40 ] . The uncertainty set U typically controls the confidence level of a model estimate , which in turn determines the agent ’ s level of robustness . It is given to the agent , who seeks to maximize performance under the worst-case model ( P , r ) ∈ U . Although untractable in general , this problem can be solved in polynomial time for rectangular uncertainty sets , i.e. , when U = ×s∈SUs = ×s∈S ( Ps × Rs ) [ 40 , 23 ] . For any policy π ∈ ∆SA and state s ∈ S , the robust value function at s is vπ , U ( s ) : = min ( P , r ) ∈U v π ( P , r ) ( s ) and the robust optimal value function v ∗ , U ( s ) : = maxπ∈∆AS v π , U ( s ) . Each of them is the unique fixed point of the respective robust Bellman operators : [ Tπ , Uv ] ( s ) : = min ( P , r ) ∈U Tπ ( P , r ) v ( s ) ∀v ∈ R S , s ∈ S , π ∈ ∆AS , [ T ∗ , Uv ] ( s ) : = max π∈∆SA [ Tπ , Uv ] ( s ) ∀v ∈ RS , s ∈ S , which are γ-contractions . For all v ∈ RS , the associated robust q-function is given by q ( s , a ) = min ( P , r ) ∈U { r ( s , a ) + γ〈P ( ·|s , a ) , v〉 } ∀ ( s , a ) ∈ X , so that vπ , U = 〈πs , qπ , U ( s , · ) 〉 where qπ , U is the robust q-function associated to vπ , U .

Summary: The paper highlights two limitations of current approaches for Robust RL:   1. Robust optimization methods are computationally demanding. 2. They do not account for uncertainty in the model dynamics.  The paper proposes to work around these limitations by learning robust MDPs using regularization. Their contributions are the following:  1. Showing that regularized MDPs are an instance of Robust MDPs with uncertain rewards; thus, policy iteration on reward-robust MDPs has the same time complexity as on regularized MDPs  2. Extend point 1 to MDPs with the uncertain transition. 3. Generalize regularized MDPs to twin regularized MDPs to retrieve robust MDPs.