Robust Reinforcement Learning via Adversarial Training with  Langevin Dynamics

1 INTRODUCTION . Reinforcement learning ( RL ) promise automated solutions to many real-world tasks with beyondhuman performance . Indeed , recent advances in policy gradient methods ( Sutton et al. , 2000 ; Silver et al. , 2014 ; Schulman et al. , 2015 ; 2017 ) and deep reinforcement learning have demonstrated impressive performance in games ( Mnih et al. , 2015 ; Silver et al. , 2017 ) , continuous control ( Lillicrap et al. , 2015 ) , and robotics ( Levine et al. , 2016 ) towards this grand challenge . Despite the success of deep RL , the progress is still upset by the fragility in real-life deployments . In particular , majority of these methods fail to perform well when there is some difference between training and testing scenarios , thereby posting serious safety and security concerns . To this end , learning policies that are robust to environmental shifts , mismatched configurations , and even mismatched control actions is becoming increasingly more important . A powerful framework to learning robust policies is to interpret the changing of the environment as an adversarial perturbation . This notion naturally lends itself to a two-player minimax problem involving a pair of agents , a protagonist and an adversary , where the protagonist learns to fulfill the original task goals while being robust to the disruptions generated by its adversary . Two prominent examples along this research vein , differing in how they model the adversary , are the Robust Adversarial Reinforcement Learning ( RARL ) ( Pinto et al. , 2017 ) and Noisy Robust Markov Decision Process ( NR-MDP ) ( Tessler et al. , 2019 ) . Despite achieving impressive performance in practice , these existing frameworks heavily rely on heuristic algorithms , and hence , suffer from lack of theoretical guarantees , even in idealistic cases with infinite data as well as computational power . One critical challenge in robust RL setting is that while maximizing rewards is a well-studied subject in classical/deep RL , the needed two-player minimax version is significantly more complicated to solve both in theory and practice . For instance , Tessler et al . ( 2019 ) prove that it is in fact strictly suboptimal to directly apply ( deterministic ) policy gradient steps to their NR-MDP max-min objectives . Owing to the lack of a better algorithm , the policy gradient is nonetheless still employed in their experiments ; similar comments also apply to ( Pinto et al. , 2017 ) . Our paper precisely bridges this gap between theory and practice in previous works , by proposing the first theoretically convergent algorithm for robust RL . Our key idea is to switch from optimizing the max-min reward to sampling from the optimal randomized policies , which corresponds to finding a mixed Nash Equilibrium ( NE ) ( Nash et al. , 1950 ) in the max-min objective . It is a classical fact in game theory that , while deterministic minimax objective is often ill-posed , the mixed NE is well-behaved under very mild assumptions . Furthermore , algorithmic approaches to finding mixed NE with finite strategies have been studied extensively ( Freund & Schapire , 1999 ; Nemirovski , 2004 ) and is recently extended to the case of infinite strategies ( Hsieh et al. , 2019 ) . In particular , ( Hsieh et al. , 2019 ) show that , by using the Stochastic Gradient Langevin Dynamics ( SGLD ) ( Welling & Teh , 2011 ) to take samples from randomized strategies , one can find a mixed NE of Generative Adversarial Networks ( Goodfellow et al. , 2014 ) . Our work introduces the same mixed NE perspective to the max-min objectives in robust RL , and substantiate the ensuing theoretical framework with extensive experimental evidence . We apply the new sampling framework to the well-known Deep Deterministic Policy Gradient ( DDPG ) method in the scope of NR-MDP . We demonstrate that the new algorithm achieves clearly superior performance in its generalization capabilities . Intriguingly , we also observe that the idea of mixed strategy in single-player RL ( i.e. , the non-robust or one-player formulation ) can lead to substantially more robust policies over the standard DDPG algorithm . More precisely , we represent the agent ’ s policy as a distribution over deterministic policies , and aim to sample from the distribution µ ( π ) ∝ exp ( − 1σJ ( π ) ) where π denotes a deterministic policy , J the associated expected reward , and σ a temperature parameter going to 0 during training . Our numerical evidence demonstrates that DDPG combined with this sampling approach for the actor update leads to learned policies that generalize better to unseen MDPs , when compared to the state-of-the-art DDPG variant of Tessler et al . ( 2019 ) while using similar computational resources . 2 BACKGROUND . 2.1 STOCHASTIC GRADIENT LANGEVIN DYNAMICS ( SGLD ) . For any probability distribution p ( z ) ∝ exp ( −g ( z ) ) , the Stochastic Gradient Langevin Dynamics ( SGLD ) Welling & Teh ( 2011 ) iterates as zk+1 ← zk − γ [ ∇̂zg ( z ) ] z=zk + √ 2γ ξk , ( 1 ) where γ is the step-size , ∇̂zg ( z ) is an unbiased estimator of ∇zg ( z ) , > 0 is a temperature parameter , and ξk ∼ N ( 0 , I ) is a standard normal vector , independently drawn across different iterations . In some cases , the convergence rate of SGLD can be improved by scaling the noise using a positive-definite symmetric matrix C. We thus define a preconditioned variant of the above update equation 1 as follows : zk+1 ← zk − γC−1 [ ∇̂zg ( z ) ] z=zk + √ 2γ C− 1 2 ξk . ( 2 ) In the experiments , we use a RMSProp-preconditioned version of the SGLD ( Li et al. , 2016 ) . 2.2 INFINITE-DIMENSIONAL BI-LINEAR GAMES . In this section , we review some of the key results from ( Hsieh et al. , 2019 ) . We denote the set of all probability measures on Z by P ( Z ) , and the set of all functions on Z by F ( Z ) . Given a ( sufficiently regular ) function h : Θ×Ω→ R , consider the following objective ( a two-player game with infinitely many strategies ) : max p∈P ( Θ ) min q∈P ( Ω ) f ( p , q ) : = E θ∼p [ E ω∼q [ h ( θ , ω ) ] ] . ( 3 ) A pair ( p∗ , q∗ ) achieving the max-min value in equation 3 is called a mixed Nash Equilibrium ( NE ) . Define the operator G : P ( Ω ) → F ( Θ ) , and its adjoint operator G† : P ( Θ ) → F ( Ω ) as follows : Gq ( θ ) : = E ω∼q [ h ( θ , ω ) ] ∈ F ( Θ ) G†p ( ω ) : = E θ∼p [ h ( θ , ω ) ] ∈ F ( Ω ) . Denoting 〈p , g〉 : = E z∼p [ g ( z ) ] for any probability measure p and function g on Z , we can write f as f ( p , q ) = 〈p , Gq〉 = 〈 G†p , q 〉 . Furthermore , the derivative ( the analogue of gradient in infinite dimension ) of f ( p , q ) with respect to p is simply Gq , and the derivative of f ( p , q ) with respect to q is G†p ; i.e. , ∇pf ( p , q ) = Gq , and ∇qf ( p , q ) = G†p . Algorithm 1 Infinite-Dimensional Entropic Mirror Descent-v2 Input : Initial distributions p1 , q1 , and learning rate η for t = 1 , 2 , . . . , T − 1 do pt+1 ( θ ) ∝ exp ( +η ∑ s≤tGqs ( θ ) ) qt+1 ( ω ) ∝ exp ( −η ∑ s≤tG †ps ( ω ) ) end for Output : p̄T = 1T ∑T t=1 pt and q̄T = 1 T ∑T t=1 qt Algorithm 2 Approximate Infinite-Dimensional Entropic Mirror Descent Input : ω1 , θ1 ← random initialization , SGLD step-size { γt } Tt=1 , thermal noise of SGLD { t } T t=1 , warmup steps for SGLD { Kt } Tt=1 , exponential damping factor β , standard normal noise ξk , ξ′k . for t = 1 , 2 , . . . , T − 1 do ω̄t , ω ( 1 ) t ← ωt ; θ̄t , θ ( 1 ) t ← θt for k = 1 , 2 , . . . , Kt do θ ( k+1 ) t ← θ ( k ) t + γt [ ̂∇θh ( θ , ωt ) ] θ=θ ( k ) t + √ 2γt tξk ω ( k+1 ) t ← ω ( k ) t − γt [ ̂∇ωh ( θt , ω ) ] ω=ω ( k ) t + √ 2γt tξ ′ k ω̄t ← ( 1− β ) ω̄t + βω ( k+1 ) t ; θ̄t ← ( 1− β ) θ̄t + βθ ( k+1 ) t end for ωt+1 ← ( 1− β ) ωt + βω̄t ; θt+1 ← ( 1− β ) θt + βθ̄t end for Output : ωT , θT . Conceptually , problem ( 3 ) can be solved via the so-called infinite-dimensional entropic mirror descent ; see Algorithm 1 . Hsieh et al . ( 2019 ) have proved the convergence ( to the mixed NE ) rate of Algorithm 1 . But this algorithm is infinite-dimensional and requires infinite computational power to implement . For practical interest , by leveraging the SGLD sampling techniques and using some practical relaxations , Hsieh et al . ( 2019 ) have proposed a simplified variant of Algorithm 1 . The pseudocode for their resulting algorithm can be found in Algorithm 2 . 3 TWO-PLAYER MARKOV GAMES . Markov Decision Process : We consider a Markov Decision Process ( MDP ) represented by M1 : = ( S , A , T1 , γ , P0 , R1 ) , where the state and action spaces are denoted by S and A respectively . We focus on continuous control tasks , where the actions are real-valued , i.e. , A = Rd . T1 : S × S × A → [ 0 , 1 ] captures the state transition dynamics , i.e. , T1 ( s′ | s , a ) denotes the probability of landing in state s′ by taking action a from state s. Here γ is the discounting factor , P0 : S → [ 0 , 1 ] is the initial distribution over states S , and R1 : S ×A → R is the reward . Two-Player Zero-Sum Markov Games : Consider a two-player zero-sum Markov game Littman ( 1994 ) ; Perolat et al . ( 2015 ) , where at each step of the game , both players simultaneously choose an action . The reward each player gets after one step depends on the state and the joint action of both players . Furthermore , the transition kernel of the game is controlled jointly by both the players . In this work , we only consider simultaneous games , not the turn-based games . This game can be described by an MDPM2 = ( S , A , A′ , T2 , γ , R2 , P0 ) , where A and A′ are the continuous set of actions the players can take , T2 : S × A × A′ × S → R is the state transition probability , and R2 : S × A×A′ → R is the reward for both players . Consider an agent executing a policy µ : S → A , and an adversary executing a policy ν : S → A′ in the environmentM . At each timestep t , both players observe the state st and take actions at = µ ( st ) and a′t = ν ( st ) . In the zero-sum game , the agent gets a reward rt = R2 ( st , at , a′t ) while the adversary gets a negative reward −rt . This two-player zero-sum Markov game formulation has been used to model the following robust RL settings : • Robust Adversarial Reinforcement Learning ( RARL ) ( Pinto et al. , 2017 ) , where the power of the adversary is limited by the action space A′ of the adversary . • Noisy Robust Markov Decision Process ( NR-MDP ) ( Tessler et al. , 2019 ) , where A′ = A , T2 ( st+1 | st , at , a′t ) = T1 ( st+1 | st , āt ) , and R2 ( st , at , a′t ) = R1 ( st , āt ) , with āt = ( 1− δ ) at + δa′t , for δ ∈ ( 0 , 1 ) . The power of the adversary is limited by δ . In our adversarial game , we consider the following performance objective : J ( µ , ν ) = E [ ∞∑ t=1 γt−1rt ∣∣∣∣ µ , ν , M ] , where ∑∞ t=1 γ t−1rt be the random cumulative return . In particular , we consider the parameterized policies { µθ : θ ∈ Θ } , and { νω : ω ∈ Ω } . By an abuse of notation , we denote J ( θ , ω ) = J ( µθ , νω ) . We consider the following objective : max θ∈Θ min ω∈Ω J ( θ , ω ) . ( 4 ) Note that J is non-convex/concave in both θ and ω . Instead of solving equation 4 directly , we focus on the mixed strategy formulation of equation 4 . In other words , we consider the set of all probability distributions over Θ and Ω , and we search for the optimal distribution that solves the following program : max p∈P ( Θ ) min q∈P ( Ω ) f ( p , q ) : = E θ∼p [ E ω∼q [ J ( θ , ω ) ] ] . ( 5 ) Then , we can use the techniques from Section 2.2 to solve the above problem .

Summary: The paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).