Twice regularized MDPs and the equivalence between robustness and regularization

1 Introduction . MDPs provide a practical framework for solving sequential decision problems under uncertainty [ 30 ] . However , the chosen strategy can be very sensitive to sampling errors or inaccurate model estimates . This can lead to complete failure in common situations where the model parameters vary adversarially or are simply unknown [ 22 ] . Robust MDPs aim to mitigate such sensitivity by assuming that the transition and/or reward function ( P , r ) varies arbitrarily inside a given uncertainty set U [ 17 , 27 ] . In this setting , an optimal solution maximizes a performance measure under the worst-case parameters . It can be thought of as a dynamic zero-sum game with an agent choosing the best action while Nature imposes it the most adversarial model . As such , solving robust MDPs involves max-min problems , which can be computationally challenging and limits scalability . In recent years , several methods have been developed to alleviate the computational concerns raised by robust reinforcement learning ( RL ) . Apart from [ 23 , 24 ] that consider specific types of coupled uncertainty sets , all rely on a rectangularity assumption without which the problem can be NP-hard [ 2 , 40 ] . This assumption is key to deriving tractable solvers of robust MDPs such as robust value iteration [ 2 , 11 ] or more general robust modified policy iteration ( MPI ) [ 18 ] . Yet , reducing time complexity in robust Bellman updates remains challenging and is still researched today [ 15 , 11 ] . At the same time , the empirical success of regularization in policy search methods has motivated a wide range of algorithms with diverse motivations such as improved exploration [ 12 , 20 ] or stability [ 34 , 13 ] . Geist et al . [ 10 ] proposed a unified view from which many existing algorithms can be derived . Their regularized MDP formalism opens the path to error propagation analysis ∗Contact author : estherderman @ campus.technion.ac.il 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in approximate MPI [ 33 ] and leads to the same bounds as for standard MDPs . Nevertheless , as we further show in Sec . 3 , policy regularization accounts for reward uncertainty only : it does not encompass uncertainty in the model dynamics . Despite a vast literature on how regularized policy search works and convergence rates analysis [ 36 , 5 ] , little attention has been given to understanding why it can generate strategies that are robust to external perturbations [ 13 ] . To our knowledge , the only works that relate robustness to regularization in RL are [ 6 , 16 , 9 ] . Derman & Mannor [ 6 ] employ a distributionally robust optimization approach to regularize an empirical value function . Unfortunately , computing this empirical value necessitates several policy evaluation procedures , which is quickly unpractical . Husain et al . [ 16 ] provide a dual relationship with robust MDPs under uncertain reward . Their duality result applies to general regularization methods and gives a robust interpretation of soft-actor-critic [ 13 ] . Although these two works justify the use of regularization for ensuring robustness , they do not enclose any algorithmic novelty . Similarly , Eysenbach & Levine [ 9 ] specifically focus on maximum entropy methods and relate them to either reward or transition robustness . We shall further detail on these most related studies in Sec . 7 . The robustness-regularization duality is well established in statistical learning theory [ 41 , 35 , 19 ] , as opposed to RL theory . In fact , standard setups such as classification or regression may be considered as single-stage decision-making problems , i.e. , one-step MDPs , a particular case of RL setting . Extending this robustness-regularization duality to RL would yield cheaper learning methods with robustness guarantees . As such , we introduce a regularization function ΩU that depends on the uncertainty set U and is defined over both policy and value spaces , thus inducing a twice regularized Bellman operator ( see Sec . 5 ) . We show that this regularizer yields an equivalence of the form vπ , U = vπ , ΩU , where vπ , U is the robust value function for policy π and vπ , ΩU the regularized one . This equivalence is derived through the objective function each value optimizes . More concretely , we formulate the robust value function vπ , U as an optimal solution of the robust optimization problem : max v∈RS 〈v , µ0〉 s. t. v ≤ inf ( P , r ) ∈U Tπ ( P , r ) v , ( RO ) where Tπ ( P , r ) is the evaluation Bellman operator [ 30 ] . Then , we show that vπ , U is also an optimal solution of the convex ( non-robust ) optimization problem : max v∈RS 〈v , µ0〉 s. t. v ≤ Tπ ( P0 , r0 ) v − ΩU ( π , v ) , ( CO ) where ( P0 , r0 ) is the nominal model . This establishes equivalence between the two optimization problems . Moreover , the inequality constraint of ( CO ) enables to derive a twice regularized ( R2 ) Bellman operator defined according to ΩU , a policy and value regularizer . For ball-constrained uncertainty sets , ΩU has an explicit form and under mild conditions , the corresponding R2 Bellman operators are contracting . The equivalence between the two problems ( RO ) and ( CO ) together with the contraction properties of R2 Bellman operators enable to circumvent robust optimization problems at each Bellman update . As such , it alleviates robust planning and learning algorithms by reducing them to regularized ones , which are known to be as complex as classical methods . To summarize , we make the following contributions : ( i ) We show that regularized MDPs are a specific instance of robust MDPs with uncertain reward . Besides formalizing a general connection between the two settings , our result enables to explicit the uncertainty sets induced by standard regularizers . ( ii ) We extend this duality to MDPs with uncertain transition and provide the first regularizer that recovers robust MDPs with s-rectangular balls and arbitrary norm . ( iii ) We introduce twice regularized MDPs ( R2 MDPs ) that apply both policy and value regularization to retrieve robust MDPs . We establish contraction of the corresponding Bellman operators . This leads us to proposing an R2 MPI algorithm with similar time complexity as vanilla MPI , thus opening new perspectives towards practical and scalable robust RL . Notations . We designate the extended reals by R : = { −∞ , ∞ } . Given a finite set Z , the class of real-valued functions ( resp . probability distributions ) over Z is denoted by RZ ( resp . ∆Z ) , while the constant function equal to 1 over Z is denoted by 1Z . Similarly , for any set X , ∆XZ denotes the class of functions defined over X and valued in ∆Z . The inner product of two functions a , b ∈ RZ is defined as 〈a , b〉 : = ∑ z∈Z a ( z ) b ( z ) , which induces the ` 2-norm ‖a‖ : = √ 〈a , a〉 . The ` 2-norm coincides with its dual norm , i.e. , ‖a‖ = max‖b‖≤1〈a , b〉 = : ‖a‖∗ . Let a function f : RZ → R. The Legendre-Fenchel transform ( or convex conjugate ) of f is f∗ ( y ) : = maxa∈RZ { 〈a , y〉 − f ( a ) } . Given a set Z ⊆ RZ , the characteristic function δZ : RZ → R is δZ ( a ) = 0 if a ∈ Z ; +∞ otherwise . The Legendre-Fenchel transform of δZ is the support function σZ ( y ) = maxa∈Z〈a , y〉 [ 3 , Ex . 1.6.1 ] . 2 Preliminaries . This section describes the background material that we use throughout our work . Firstly , we recall useful properties in convex analysis . Secondly , we address classical discounted MDPs and their linear program ( LP ) formulation . Thirdly , we briefly detail on regularized MDPs and the associated operators and lastly , we focus on the robust MDP setting . Convex Analysis . Let Ω : ∆Z → R be a strongly convex function . Throughout this study , the function Ω plays the role of a policy and/or value regularization function . Its Legendre-Fenchel transform Ω∗ satisfies several smoothness properties , hence its alternative name `` smoothed max operator '' [ 25 ] . Our work makes use of the following result [ 14 , 25 ] . Proposition 2.1 . Given Ω : ∆Z → R strongly convex , the following properties hold : ( i ) ∇Ω∗ is Lipschitz and satisfies∇Ω∗ ( y ) = arg maxa∈∆Z 〈a , y〉 − Ω ( a ) , ∀y ∈ R Z . ( ii ) For any c ∈ R , y ∈ RZ , Ω∗ ( y+c1Z ) = Ω∗ ( y ) + c. ( iii ) The Legendre-Fenchel transform Ω∗ is non-decreasing . Discounted MDPs and LP formulation . Consider an infinite horizon MDP ( S , A , µ0 , γ , P , r ) with S and A finite state and action spaces respectively , 0 < µ0 ∈ ∆S an initial state distribution and γ ∈ ( 0 , 1 ) a discount factor . Denoting X : = S ×A , P ∈ ∆XS is a transition kernel mapping each state-action pair to a probability distribution over S and r ∈ RX is a reward function . A policy π ∈ ∆SA maps any state s ∈ S to an action distribution πs ∈ ∆A , and we evaluate its performance through the following measure : ρ ( π ) : = E [ ∞∑ t=0 γtr ( st , at ) ∣∣∣∣ µ0 , π , P ] = 〈vπ ( P , r ) , µ0〉 . ( 1 ) Here , the expectation is conditioned on the process distribution determined by µ0 , π and P , and for all s ∈ S , vπ ( P , r ) ( s ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , π , P ] is the value function at state s. Maximizing ( 1 ) defines the standard RL objective , which can be solved thanks to the Bellman operators : Tπ ( P , r ) v : = r π + γPπv ∀v ∈ RS , π ∈ ∆SA , T ( P , r ) v : = max π∈∆SA Tπ ( P , r ) v ∀v ∈ R S , G ( P , r ) ( v ) : = { π ∈ ∆SA : Tπ ( P , r ) v = T ( P , r ) v } ∀v ∈ R S , where rπ : = [ 〈πs , r ( s , · ) 〉 ] s∈S and Pπ = [ Pπ ( s′|s ) ] s′ , s∈S with Pπ ( s′|s ) : = 〈πs , P ( s′|s , · ) 〉 . Both Tπ ( P , r ) and T ( P , r ) are γ-contractions with respect to ( w.r.t . ) the supremum norm , so each admits a unique fixed point vπ ( P , r ) and v ∗ ( P , r ) , respectively . The set of greedy policies w.r.t . value v defines G ( P , r ) ( v ) , and any policy π ∈ G ( P , r ) ( v∗ ( P , r ) ) is optimal [ 30 ] . For all v ∈ R S , the associated function q ∈ RX is given by q ( s , a ) = r ( s , a ) + γ〈P ( ·|s , a ) , v〉 ∀ ( s , a ) ∈ X . In particular , the fixed point vπ ( P , r ) satisfies v π ( P , r ) = 〈πs , q π ( P , r ) ( s , · ) 〉 where q π ( P , r ) is its associated q-function . The problem in ( 1 ) can also be formulated as an LP [ 30 ] . Given a policy π ∈ ∆SA , we characterize its performance ρ ( π ) by the following v-LP [ 30 , 26 ] : min v∈RS 〈v , µ0〉 subject to ( s.t . ) v ≥ rπ + γPπv . ( Pπ ) This primal objective provides a policy view on the problem . Alternatively , one may take a state visitation perspective by studying the dual objective instead : max µ∈RS 〈rπ , µ〉 s. t. µ ≥ 0 and ( IdRS − γPπ∗ ) µ = µ0 , ( Dπ ) where Pπ∗ is the adjoint policy transition operator 2 : [ Pπ∗ µ ] ( s ) : = ∑ s̄∈S P π ( s|s̄ ) µ ( s̄ ) ∀µ ∈ RS , and IdS is the identity function in RS . Let I ( s′|s , a ) : = δs′=s ∀ ( s , a , s′ ) ∈ X ×S the trivial transition matrix and define its adjoint transition operator as I∗µ ( s ) : = ∑ ( s̄ , ā ) ∈X I ( s|s̄ , ā ) µ ( s̄ , ā ) ∀s ∈ 2It is the adjoint operator of Pπ in the sense that 〈Pπv , v′〉 = 〈v , Pπ∗ v′〉 ∀v , v′ ∈ RS . S. The correspondence between occupancy measures and policies lies in the one-to-one mapping µ 7→ µ ( · , · ) I∗µ ( · ) = : πµ and its inverse π 7→ µπ given by µπ ( s , a ) : = ∞∑ t=0 γtP ( st = s , at = a ∣∣∣∣µ0 , π , P ) ∀ ( s , a ) ∈ X . As such , one can interchangeably work with the primal LP ( Pπ ) or the dual ( Dπ ) . Regularized MDPs . A regularized MDP is a tuple ( S , A , µ0 , γ , P , r , Ω ) with ( S , A , µ0 , γ , P , r ) an infinite horizon MDP as above , and Ω : = ( Ωs ) s∈S a finite set of functions such that for all s ∈ S , Ωs : ∆A → R is strongly convex . Each function Ωs plays the role of a policy regularizer Ωs ( πs ) . With a slight abuse of notation , we shall denote by Ω ( π ) : = ( Ωs ( πs ) ) s∈S the family of state-dependent regularizers.3 The regularized Bellman evaluation operator is given by [ Tπ , Ω ( P , r ) v ] ( s ) : = T π ( P , r ) v ( s ) − Ωs ( πs ) ∀v ∈ R S , s ∈ S , and the regularized Bellman optimality operator by T ∗ , Ω ( P , r ) v : = maxπ∈∆SA T π , Ω ( P , r ) v ∀v ∈ R S [ 10 ] . The unique fixed point of Tπ , Ω ( P , r ) ( respectively T ∗ , Ω ( P , r ) ) is denoted by v π , Ω ( P , r ) ( resp . v ∗ , Ω ( P , r ) ) and defines the regularized value function ( resp . regularized optimal value function ) . Although the regularized MDP formalism stems from the aforementioned Bellman operators in [ 10 ] , it turns out that regularized MDPs are MDPs with modified reward . Indeed , for any policy π ∈ ∆SA , the regularized value function is vπ , Ω ( P , r ) = ( IS − γP π ) −1 ( rπ − Ω ( π ) ) , which corresponds to a non-regularized value with expected reward r̃π : = rπ − Ω ( π ) . Note that the modified reward r̃π ( s ) is no longer linear in πs because of the strong convexity of Ωs . Also , this modification does not apply to the reward function r but only to its expectation rπ , as we can not regularize the original reward without making it policy-independent . Robust MDPs . In general , the MDP model is not explicitly known but rather estimated from sampled trajectories . As this may result in over-sensitive outcome [ 22 ] , robust MDPs reduce such performance variation . Formally , a robust MDP ( S , A , µ0 , γ , U ) is an MDP with uncertain model belonging to U : = P ×R , i.e. , uncertain transition P ∈ P ⊆ ∆XS and reward r ∈ R ⊆ R X [ 17 , 40 ] . The uncertainty set U typically controls the confidence level of a model estimate , which in turn determines the agent ’ s level of robustness . It is given to the agent , who seeks to maximize performance under the worst-case model ( P , r ) ∈ U . Although untractable in general , this problem can be solved in polynomial time for rectangular uncertainty sets , i.e. , when U = ×s∈SUs = ×s∈S ( Ps × Rs ) [ 40 , 23 ] . For any policy π ∈ ∆SA and state s ∈ S , the robust value function at s is vπ , U ( s ) : = min ( P , r ) ∈U v π ( P , r ) ( s ) and the robust optimal value function v ∗ , U ( s ) : = maxπ∈∆AS v π , U ( s ) . Each of them is the unique fixed point of the respective robust Bellman operators : [ Tπ , Uv ] ( s ) : = min ( P , r ) ∈U Tπ ( P , r ) v ( s ) ∀v ∈ R S , s ∈ S , π ∈ ∆AS , [ T ∗ , Uv ] ( s ) : = max π∈∆SA [ Tπ , Uv ] ( s ) ∀v ∈ RS , s ∈ S , which are γ-contractions . For all v ∈ RS , the associated robust q-function is given by q ( s , a ) = min ( P , r ) ∈U { r ( s , a ) + γ〈P ( ·|s , a ) , v〉 } ∀ ( s , a ) ∈ X , so that vπ , U = 〈πs , qπ , U ( s , · ) 〉 where qπ , U is the robust q-function associated to vπ , U .

Summary: The paper studies the connection between robust and regularized MDPs. The authors first show that solving robust MDPs with uncertain rewards is equivalent to solving regularized MDPs with a suitable policy-dependent regularization. Then, they extend this result to MDPs with uncertain transitions, obtaining the equivalence with both policy and value regularization. Finally, they propose R^2 MDPs, i.e., MDPs with both value and policy regularization for which, under certain assumptions, they guarantee good properties (eg contraction) for the corresponding Bellman operators and thus convergence of a modified policy iteration scheme. This allows solving robust MDP problems with the same computational cost as regularized ones.