Robust Reinforcement Learning via Adversarial Training with  Langevin Dynamics
Authors: Huang Yu-Ting
Parameswaran Kamalaruban
Paul Rolland
Ya-Ping Hsieh
Volkan Cevher
Venue:  (2020)
Keywords: deep reinforcement learning
robust reinforcement learning
min-max problem

Abstract: We re-think the Two-Player Reinforcement Learning (RL) as an instance of a distribution sampling problem in infinite dimensions. Using the powerful Stochastic Gradient Langevin Dynamics, we propose a new two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our new algorithm consistently outperforms existing baselines, in terms of generalization across differing training and testing conditions, on several MuJoCo environments.

Under review as a conference paper at ICLR 2020
ROBUST REINFORCEMENT LEARNING VIA ADVERSARIAL TRAINING WITH LANGEVIN DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
We re-think the Two-Player Reinforcement Learning (RL) as an instance of dis- tribution sampling problem in infinite dimensions. Using the powerful Stochastic Gradient Langevin Dynamics, we propose a new two-player RL algorithm, which is sampling variant of the two-player policy gradient method. Our new algorithm consistently outperforms existing baselines, in terms of generalization across dif- fering training and testing conditions, on several MuJoCo environments.
INTRODUCTION
Reinforcement learning (RL) promise automated solutions to many real-world tasks with beyond- human performance. Indeed, recent advances in policy gradient methods (Sutton et al. 2000; Silver et al. 2014, Schulman et al., 2015) 2017) and deep reinforcement learning have demonstrated im- pressive performance in games (Mnih et al. 2015] Silver al. 2017). continuous control (Lillicrap et al. 2015), and robotics (Levine et al. (016)) towards this grand challenge.
Despite the success of deep RI., the progress is still upset by the fragility in real-life deployments. In particular, majority of these methods fail to perform well when there is some difference between training and testing scenarios, thereby posting serious safety and security concerns. To this end, learning policies that are robust environmental shifts, mismatched configurations, and even mis- matched control actions is becoming increasingly more important.
A powerful framework to learning robust policies is to interpret the changing of the environment as an adversarial perturbation. This notion naturally lends itself to a two-player minimax problem involving a pair of agents, protagonist and adversary, where the protagonist learns to fulfill the original task goals while being robust to the disruptions generated by its adversary.
Two prominent examples along this research vein, differing in how they model the adversary. are the Robust Adversarial Reinforcement Learning (RARL) (Pinto et al. 2017) and Noisy Robust Markov Decision Process (NR-MDP) (Tessler al.) 2019). Despite achieving impressive performance in practice, these existing frameworks heavily rely on heuristic algorithms, and hence, suffer from lack of theoretical guarantees, even in idealistic cases with infinite data as well computational power.
One critical challenge in robust RL setting is that while maximizing rewards is well-studied sub- ject in classical/deep RL, the needed two-player minimax version significantly more complicated to solve both in theory and practice. For instance, Tessler et al. (2019) prove that it is in fact strictly suboptimal to directly apply (deterministic) policy gradient steps to their NR-MDP max-min objec- tives. Owing to the lack of better algorithm, the policy gradient is nonetheless still employed in their experiments; similar comments also apply to (Pinto et al.. 2017).
Our paper precisely bridges this gap between theory and practice in previous works, by proposing the first theoretically convergent algorithm for robust RI. Our key idea is to switch from optimizing the max-min reward to sampling from the optimal randomized policies, which corresponds to finding a mixed Nash Equilibrium (NE) (Nash et al. 1950) in the max-min objective.
It is classical fact in game theory that, while deterministic minimax objective is often ill-posed, the mixed NE is well-behaved under very mild assumptions. Furthermore, algorithmic approaches to finding mixed NE with finite strategies have been studied extensively (Freund & Schapire 1999. Nemirovski] 2004) and is recently extended to the case of infinite strategies (Hsich et al.] 2019).
In particular, (Hsieh et al. 2019) show that, by using the Stochastic Gradient Langevin Dynamics

---

Under review as a conference paper at ICLR 2020
(SGLD) (Welling & Teh. 2011) to take samples from randomized strategies, can find mixed NE of Generative Adversarial Networks (Goodfellow et al., 2014).
Our work introduces the same mixed NE perspective to the max-min objectives in robust RL, and substantiate the ensuing theoretical framework with extensive experimental evidence. We apply the new sampling framework to the well-known Deep Deterministic Policy Gradient (DDPG) method in the scope of NR-MDP. We demonstrate that the new algorithm achieves clearly superior performance in its generalization capabilities.
Intriguingly, we also observe that the idea of mixed strategy in single-player RL (i.e., the non-robust one-player formulation) algorithm. More precisely, we represent the agent's 's policy as a distribution over deterministic poli- cies, and aim to sample from the distribution (( x (xp(-) where denotes deterministic policy, J the associated expected reward, and a temperature parameter going to O during training.
lead to substantially more robust policies over the standard DDPG
Our numerical evidence demonstrates that DDPG combined with this sampling approach for the actor update leads to learned policies that generalize better to unseen MDPs, when compared to the state-of-the-art DDPG variant of Tessler et 2019) while using similar computational resources.
BACKGROUND
2.1
STOCHASTIC GRADIENT LANGEVIN DYNAMICS (SGLD)
For any probability distribution p (z) exp (-g (z)), the Stochastic Gradient Langevin Dynamics (SGLD) Welling & Teh (201T) iterates as
2h+1 + 24
V V (2) ==2k
+
V
(I)
where is the step-size, V (2) an unbiased estimator V=g e > 0 is a temperature parameter, and & ~ N (0, 1) is standard normal vector, independently drawn across different iterations. In some cases, the convergence rate of SGLD can be improved by scaling the noise using positive-definite symmetric matrix C. We thus define preconditioned variant of the above update equation] as follows:
Zk+1 +- 25 -0-1 V&g
+
22yeC E
(2)
2=Rk
In the experiments, we PRMSstop-preconditonee version of the SGLD LLi al. 2016).
2.2 INFINIEE-DIMENSIEE DIMENSIONAL BI-LINEAR GAMES
In this section, we review some of the key results from (Hsieh et al., 2019). We denote the set of all probability measures on Z by P (Z). and the set all functions on 2 by F (2). Given a
(sufficiently regular) function h θ x Ω + R, consider the following objective (a two-player game with infinitely many strategies):
max min ( sEP(O)
(p, = E tou
E wog
(3)
A pair (p".9") .9") achieving the max-min value in equation 3 is called mixed Nash Equilibrium (NE). Define the operator G (S2) (e), and its adjoint operator Gf (9) (S2) as follows:
Gq (0) = E (0,W) (())
Gtp (w) E (0,ww) (1).
Denoting (p.9) = E [g [=)] for any probability measure and function g Z, write z~p
as (p. g) = Gq) = (G) p, q). Furthermore, the derivative (the analogue of gradient in infinite dimension) of (p. with respect to p simply Gq, and the derivative of f (p, q) with respect to is Gtp; i.e., v,I (p.q) = Gy. and v, (p.9) Gtp.
2

---

Under review as a conference paper at ICLR 2020
Algorithm Infinite-Dimensionie Entropic Mirror Descent-v2
Input: Initial distributions Pr, 91, and learning rate 1) for ( = 1, ,
do Pe+1 (0) exp
(n Eese Gqs (8)
ge+1 (w) x exp
(11) E.sE Gips (w))
end for
Output: pr + Ei= Pt and qr = + Ei=1 9
Algorithm Approximate
Entropic Mirror Descent
Input: w1, 81 < random initialization, SGLD step-size f thermal noise SGLD { 1
warmup steps for SGLD {Ke)t=1 T exponential damping factor 3, standard normal noise Exy Sk. for t = 1, ...,....
1 do
Wg, wi f Wt 0, 0) - 0
for k -1,,.... K, do
al++1) t - + 7
Vah w1)
8=0(h) + /2TteEu
w/ (k+1) s- w - 71 |u (0c, w)
(k)
+ 127
Wi t (1- 3) we + (1+ w=w(t
(1- (1-3))8 + 30
end for w1+1 < (1 3) We + Bür B1t1 t (1 - -3) 0: + 30,
end for Output: WT, 0r.
Conceptually, problem 33) can be solved via the so-called infinite-dimensional entropic mirror de- scent; see Algorithm]] ! Hsich al. (2019) have proved the convergence (to the mixed NE) rate of Algorithm [ But this algorithm is setiiite-dimension implement. For practical interest, by leveraging the SGLD sampling techniques and using some practical relaxations, Hsieh et al. (2019) have proposed simplified variant of Algorithm | The pseudocode for their resulting algorithm can be found in Algorithm
and requires infinite computational power
TWO-PLAYER MARKOV GAMES
Markov Decision Process: We consider a Markov Decision Process (MDP) represented by Mi := (S. 7, Po. R1), where the state and action spaces are denoted by S and A respec- tively. We focus on continuous control tasks, where the actions are real-valued, ie.,,
Rd. Ti : S S x A + [0, 1] captures the state transition dynamics, i.e., Ti (s' ss a) denotes the probability of landing in state by taking action from state s. Here 7 is the discounting factor, P j0, 1] is the initial distribution over states S, and R1 S x A + R is the reward.
Two-Player Zero-Sum Markov Games: Consider two-player zero-sum Markov game Littman [1994]; ; Perolat et al. (2015), where at each step of the game, both players simultaneously choose an action. The reward each player gets after one step depends on the state and the joint action of both players. Furthermore, the transition kernel of the game is controlled jointly by both the players. In this work, we only consider simultaneous games, not the turn-based games.
This game can be described by an MDP M₂ = A', R2, Po), where A and A' are the continuous set actions the players can take, T2 S x x A' x S + I is the state transition probability, and R2 S x A x A'' + R is the reward for both players. Consider an agent executing a policy μ S + A. and an adversary executing a policy v : + A' in the environment M. At each timestep t, both players observe the state s; and take actions a f (st) and d, v (st). In the zero-sum game, the agent gets a reward r, = R2 (St, ae d,) while the adversary gets a negative reward -rt-
3

---

Under review as a conference paper at ICLR 2020
This two-player zero-sum Markov game formulation has been used to model the following robust RL settings:
Robust Adversarial Reinforcement Learning (RARL) (Pinto al. of the adversary is limited by the action space A' of the adversary.
2017
where the power
Noisy Robust Markov Decision Process (NR-MDP) (Tessler et al.
2019 where A'' = A, R1 (st. @t), with at
To ($1+1 St. de, a,) = Ti ($t+1 St, ",, and Rg (s,, az 0,)
(1 - 8) + 5a,, for 5 E (0, 1). The power of the adversary is limited by
In our adversarial game, we consider the following performance objective:
J (u,v)
E
M4 M
where Σt=1 "t be the random cumulative return. In particular, consider the parameterized policies {1g / E }), and (v- : w E 22. By an abuse of notation, we denote (9,w) = J (Ho, vw). We consider the following objective:
max min J w0,
(4)
BEB WEB
Note that J is non-convex/coocave in both 0 and w. Instead of solving equation 4 directly, we focus on the mixed strategy formulation of equation In other words, we consider the set of all probability distributions over θ and Ω and we search for the optimal distribution that solves the following program:
[EVO,.
max min PEP(O) (EP())
(p.
E 0mp w
(5)
Then, we can use the techniques from Section 2.2 to solve the above problem.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of using infinite-dimensionn sampling techniques solve the robust RL problem.
As case study, we consider NR-MDP setting with 5 = 0.1 (as recom- mended Section 6.3 of (Tessler al. 2019)). We design two-player variant of DDPG (Lillierap) et al. 2015) algorithm by adapting the Algorithm 2 As opposed to standard DDPG, in two-player DDPG two actor networks output two deterministic policies, the protagonist and adversary policics, denoted by Me and kn- The critic is trained to estimate the Q-function of the joint-policy. The gradi- ents of the protagonist and adversary parameters are given Proposition 5 of (Tsssler et al. 2019
Two-Player DDPG:
The resulting algorithm is given in Algorithm 3
We compare the performance of our algorithm against the baseline algorithm proposed in TTssler let al. 2019) (see Algorithm 4. (Tessler et al. 2019) have suggested training ratio of for actors and critic updates. Note that the action noise is injected while collecting transitions for the replay buffer. In In/Fujimoto al. (2018), authors noted that the action noise drawn from the Ornstein- Uhlenbeck UUhlenbeck & Ornstein (1930) process offered no performance benefits. Thus we also consider uncorrelated Gaussian noise.
Setup: We evaluate the performance of Algorithm 3 and Algorithm on standard continuous control benchmarks available on OpenAl Gym Brockman et al. (2016) utilizing the MuJoCo envi- eight tasks: Walker, Hopper, Half-
ronment Todorov et al. (2012). Specifically, we benchmark Cheetah, Ant, Swimmer, Reacher, Humanoid, and InvertedPendullmm. Details of these environments be found in Brockman al. (2016) and on the GitHub website.
The Algorithm
algorithms, implementation is based on the codebase from Tessler et al. 2019). For all the two-layer feedforward neural network structure of (64, 64, tanh) for both actors (agent and adversary) and critic. The optimizer we use to update the critic is Adam Kingma & Ba 2015) with learning rate of 10-3, The target networks are soft-updated with 0.999.
4

---

Under review as a conference paper at ICLR 2020
Valker2d-v2
HaitChoetah-v2
900
SGLD
Baseline
SGLD
800
600
600
700
400
600
200
500
400
300
-400
0.5
075
1.0
15
200
0.5
0.75
100
15
20
Relative mass
Relative mass
Hopper-V2
Aet-42
500
Baseline
600
SELD
800
400
600
200
500
100
400
0.5
075
1.0
15
20
05
0.5
10
15
200
Relutive mass
Relative mass
Swimmer-v2
Reacher-v2
Baseline SOLD
-7.0
Baselie SOLD
-7.5
-0.0
-0.
-3.0
-9.5
-18.0
075
20
0.5
0.75
100
20
Rulative
Humanoid-42
InvertedPendulum-22
Baseline SOLD
Baseline SCLD
500
900
800
Reword
300
300
600
100
500
875
1.5
2.0
Figure 1: Average performance (over 5 seeds) of Algorithm (-) and Algorithm E (-) under the NR-MDP setting with 0.1. The evaluation is performed without adversarial perturbations, on range of mass values not encountered during training.
For the baseline, the actors trained with RMSProp optimizer. For algorithm, the actors updated according to Algorithm with warmup steps K, min {15, [(1 + 10-5)'],, and thermal noise a, do x (1 10 yy The hyperparameters that are not related to exploration (see
Table are identical to both algorithms that compared.
5

---

Under review as a conference paper at ICLR 2020
9aseine/ailer2dv22
SGL0YWaker22-gg2
@aseiiniHaattE/ettaaww
SGLD/ailCCwwsthh.vg
30
02
Relitive
Mace Frobebiiity
Probubility
Probeollly
Noton Protaslity
Baseling/Hopper--2
SCIDD/Hoppe-222
BaselinejAnt-22
SGLDyAnt-v2
30
103
103
61
-10
20
07
Notte Probabiltty
Note Probabiltty
Woite Frebablitt
Noite Frebabiity
Bacelins/Swimmer-22
SGLD//wiimmer-v2
BaselingRexchar-22
SGLD/Raacher-v2
-333
-500
323
161
Na5e Probabilty
Nose Probobiity
Rinned Probedillyy
Nchu Probelalliy
Bacolinn/Humanoid V2
SOLD/Humanotd V2
05
1001
SArtbee
30 03
02 Probubiltty
62 0.2 5.8
Probabiity
Roue Prosubilly
Nobi Probubility
Figure 2: Average performance (over 5 seeds) of Algorithm ( and Algorithm (-), under the NR-MDP setting with 0.1. The evaluation performed on a range of noise probability and mass values not encountered during training.
And we tuned only the exploration-related hyper-parameters (for both algorithms) by grid search:
(a) Algorithm 3 with (ao, or) {10-2 2,10 110-3 1,100 10-4, 10 10-5} x {0, 0.01, 0.1, 0.2, 0.3, 0.4) (b) Al- gorithm 4 with E {0, 0.01, 0.1, 0.2, 0.3, 0.4). For each algorithm-environeed the best performing exploration hyperparameter configuration (see Tables and 3
pair, we identified
Each algorithm is trained on 0.5M samples (i.e., 0.5M time steps in the environment). We run our experiments, for each environment, with 5 different seeds. The exploration noise is turned off for evaluation.
Evaluation: We evaluate the robustness of both algorithms under different testing conditions, and in the presence of adversarial disturbances in the testing environment. We train both algorithms with the standard mass variables in OpenAl Gym. At test time, we evaluate the learned policies by changing the mass values (without adversarial perturbations) and estimating the cumulative rewards. As shown in Figure [ our Algorithm outperforms the baseline Algorithm 4 in terms of robust- ness. We also evaluate the robustness of the learned policies under both test condition changes, and adversarial disturbances (see Figure 2).
We evaluate the robustness one-player variants of Algorithm 3 and Al- gorithm i.e. consider the NR-MDP setting with 5 0. In this case, K for
One-Player DDPG:

---

Under review as a conference paper at ICLR 2020
Walker2d-v2
HalfCheetah-v2
Baseline SGLD
700
000
Reward
600
500
200
0.5
075
10
15
2.0
0.5
075
10
15
20
Relathve mass
Relative mass
Hopper-v2
Ant-v2
550
Baseline SOLD
750
Baseline SOLD
050
Rewardd
000
500
0.5
675
100
15
0.5
0.75
1.0
15
20
Relative mass
Relative mass
Swimmer-v2
Reacher-v2
37.5
Baseline
SOLD
-75
Baseine SOLD
-00
-8.5
25.0
22.5
20.0
0.5
075
15
0.75
20
Humanoid-42
Invrttedeendulum-22
450
Baseline SOLD
800
Boword
700
000
200
150
400
Basctine
SELD
1.5
300
1's
(-, under the NR-MDP setting with 8 = 0. The evaluation is performed without adversarial perturbations, on range of mass values not encountered during training.
Figure 3: Average performance (over 5 seeds) of Algorithm (-) and Algorithm 4
Algorithm 3 (this choice of K, makes the computational complexity of both algorithms equal). The results presented in Figures 2 andd 4
Here, we remark that Algorithm 3 with = 0, and K, = simply the standard DDPG with actor being updated by preconditioned version of SGLD. Thus achieve robustness under dif-

---

Under review as a conference paper at ICLR 2020
9aseiieWailer2dv22
SGL0YWaker25-g2
@aseinniHalttEeettaaww
SGLD/aalC/hwxthhvv
30
Relittve
300
Mace Frobebiiity
Probubility
Probecllly
Noton Protaslity
Baseling/Hopper--2
SCILD/Hoppar222
faselins/AAtt-/>
S01 DyAnt-u?
600
62
29
07
01
Notte Probabiltty
Note Probabiltyy
Roise Probasllty
Noloo Probociitty
Bacelins/Swimmer-22
SGLD//wiimmrr-22
BasclingRexchar-22
SGLD/Reacher-v2
Na5e Probab Ity
Nose Prosobiity
Rineed Protedillyy
Ncha Probelalliy
Bacoline/Humanodd V2
SOLD/Humanold V2
5G1 5c1.:/@rwerteeeeenddllee
05
1001
Nocalre Maae
215
2.5 Probabiiity
30
03
02
62 0.2 5.8
Probubiltty
Roue Prosubilly
Nobu Probubility
( and Algorithm (-, under the NR-MDP setting with 0. The evaluation is performed on a range of noise probability and mass values not encountered during training.
Figure 4: Average performance (over 5 seeds) of Algorithm 3
ferent testing conditions with just a simple change in the DDPG algorithm and without additional computational cost.
5 CONCLUSION
In this work, studied the robust reinforcement learning problem. By adapting the approximate ciinite-dimensionll entropic mirror descent from (Hsich al.] 2019), we design a robust variant of DDPG algorithm, under the NR-MDP setting. In experiments, evaluated the robustness of our algorithm on several continuous control tasks, and found that our algorithm clearly outperformed the baseline algorithm from (Tsssler et al. 2019)
REFERENCES
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi- multi-agent autocurricula. arXiv preprint 1909, 07528, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, arXiv preprint arXiv:1606.01540 2016.
8

---

Under review as a conference paper at ICLR 2020
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. 2017.
https:
//git http///peeaaaaaaaeeeeenee.eeeeee
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103, 1999.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1rXiv 09977, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ya-Ping Hsieh, Chen Liu, and Volkan Cevher. Finding mixed nash equilibria of generative adver- sarial networks. In International Conference Machine Learning, pp- 2810-2819, 2019.
Diederik Kingma and Jimmy Ba. A method for stochastic optimization. International Confer- ence on Learning Representations (ICLR), volume 5, 2015.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-137) 2016.
Chunyuan Li, Changyou Chen, David E Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In AAAI, volume pp. 4, 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
Michael L. Littman. Markov games as framework for multi-agent reinforcement learning. In Machine Learning Proceedings. Elsevier, 1994.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
John F Nash et al. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48-444 49, 1950.
Arkadi Nemirovski. Prox-mether with rate of convergence (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15()):229-251, 2004.
Julien Perolat, Bruno Scherrer, Bilal Piot, and Olivier Pietquin. Approximate dynamic programming for two-player zero-sum Markov games. In International Conference on Machine Learning, 2015.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce-
ment learning. In International Conference on Machine Learning. 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In JCML, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 350(7676)334,, 2017.

---

Under review as a conference paper at ICLR 2020
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems. pp. 1057-1063, 2000.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. arXiv preprint urXiv:1901.09166 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 36(5):833, 1930.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681-688, 2011.
10

---

Under review as a conference paper at ICLR 2020
A ALGORITHMS AND HYPERPARAMETER DETAILS
Most of the values for the hyperparameters (in Table I are chosen from Dhariwal et al. (017). ). In
addition, for Algorithm 3
set
1. thermal noise oo x (1 5 x 10 10-5), where do {10-2 {10-3,10-3 10-1100--0000 10-50000 }.
2. warmup steps K, =
min { 15, +(1+ 10-
The best performing values for every environment are presented in Tables
2 and
Table 1: Table of Hyperparameters
Hyperparameter
Value
critic optimizer
Adam
critic learning rate
10-3 3
target update rate
0.999
mini-batch size N
128
discount factor y
0.99
damping factor 3
0.9
replay buffer size action noise parameter
106
{0, 0.01, 0.1,0,,,
RMSProp parameter a
0.999
RMSProp parameter
10-8
RMSProp parameter 10-1
Table 2: Hyperparameters chosen via grid search (for NR-MDP setting with ő = 0.1)
Algorithm
(oo, o))
Algorithm
Walker-v2
(10-2,0.01)
0
HalfCheetah-v2
(10-2.0)
0.01
Hopper-v2
(10-3,0.2)
0.2
Ant-v2
(10-4.0.2)
0.4
Swimmer-v2
(10-5,0.4)
0.4
Reacher-v2
(10-3,0.2)
0.4
Humanoid-v2
(10-4.0.01)
0
InvertedPendulum--22
(10-3,0.01)
0.1
Table 3: Hyperparameters chosen via grid search (for NR-MDP setting with ò
Algorithm
(ao. σ)
Algorithm
Walker-v2
(10-2.0.1)
0.01
HalfCheetah-v2
(10-2,0.01)
0.2
Hopper-v2
(10-5,0.3)
0.4
Ant-v2
(10-2.0.4)
0.4
Swimmer-v2
(10-2.0.2)
0.3
Reacher-v2
(10-3,0.2
0.3
Humanoid-v2
(10-2.0.1)
0
InvertedPenduluumv22
(10-3.0)
0.01
11

---

Under review as a conference paper at ICLR 2020
B ToY EXAMPLE
Setting, We consider the following simplified/modiiied hide and seek (running and chasing) game from Baker et al. 2019):
Two agents (a hider and seeker) are placed in 2d-square environment without any obsta- cles. The hider wants to be away from the seeker as much as possible.
The state (at time t) is represented by s = (ehido, gooo whide 2gook) E S, where éhide is the location of the hider, and ut the velocity the seeker. We assume that both agents observe the full state.
The action space is same for both agents, i.e., abide, aaeek EA= (left, right, up, down).
The reward function is given by Rs = - -Rhide (8+) = Reeek (se) d(Ebidle sqack) i.e.,
inverse of the current distance between the hider and seeker.
The additional parameters fhide, and feeek denote the force of the hider and the seeker respectively. We fix hidde and let fseek E 32,
The transition dynamics T (8t+1 8t, at hide aorkk, phide paock) depends on the parameters fhide, and faeck.
We have pre-trained (three) competitive seeker policies म punk for all faeek (2,3,4).
For training, we fix setting with fhide = 4, and faeek parametric policies Te and seek using the following two approaches (generate episode St Abide, Arrck, RRg, S1+1.... using current parameters, and define G2 ΣT 7" R1tr):
Training.
We train the
Gradient descent ascent (GDA) approach:
Oct1 +- 0, - Gr [Vo m hide (Ahide [Si)] 8=8,
W++1 We + G. V In m (Ageek St) w=w.
Stochastic Gradient Langevin Dynamics (SGLD) approach:
Oc+1 < 0: Gr [Vo In
T hide (Ahide [Si]] am0,
+ 2n (0, T)
Wr+1 < w/ + G, [ In T. (Afoek Ss)] w=w: [.)) /2t (0,
We denote the final hider policy parameters resulting from the above methods @GDA, and ISGLD-
TOSCLD under three different settings: T' (st+1 St, abide hide ageek, f fseek). apeek paock (- st), flide 4, and faeek (2, 4]. The results (averaged over 10 initial seeds, and 100 episodes per seed) are shown in hide
hide and π hide Evaluation. We compare the performance of the policies ToCDA
5 In this simplified setting, on average, π ReccLo performs better than #BeDA* but the difference in performance is not that significant as in the MuJoCo environments.
12

---

Under review as a conference paper at ICLR 2020
-1.45
SGLD:
GDA
-1,50
SGLD:
GDA
S
-1.55
GDA:
-1.60
-1.5
100
150
200
250
300
step
hide Figure The instantaneous reward obtained by executing "I OODA
and hide
in different settings.
BSOLD
C
ADDITIONAL RESULTS
The results for the best performing seeds presented in Figures 6 and 7
13

---

Under review as a conference paper at ICLR 2020
Walkar2d-v2
HalfChettab-22
Daseiie
2003
1400
2000
600
05
075
15
2.0
0.5
675
15
20
Relative
$0.0
SCLD
3500
800
150
782
500
0.5
Swimmer-22
Reacher-y2
Basediee
35
-10
30
25
0.5
076
15
20
05
0.7
1.5
20
Relatve man
Relative mas
Inwerteei-nndulum-2
5010
100
600
300
800
200
100
05
0.
29
Figure 6: Average performance (of the best performing seed) of Algorithm 3 (-) and Algorithm (-), under the NR-MDP setting with d = 0.1. The evaluation performed without adversarial perturbations, range of mass values not encountered during training.
14

---

Under review as a conference paper at ICLR 2020
Algorithm Two-Player DDPG with SGLD Actors (SGLD)
Hyperparameters: see [
Initialize (randomly) policy parameters wi, 01, and Q-function parameter Φ Initialize the target network parameters wtarg < W1, Ovarg + 01. and Ptarg + Φ. Initialize replay buffer D. Initialize m < 0 m' + 0.
1 1.
repeat
Observe state s, and select actions me, (8) ( = (Pw where E, ~ N ~N(00,
Execute the action ā = (1 - a) + da' in the environment. Observe reward r, next state s', and done signal d to indicate whether s' is terminal. Store (s, (s,ā,r, s', d) in replay buffer DD If s' is terminal, reset the environment state. if it's time to update then for however many updates do
Wr:w, (1) < Wi : 0,, (( + A₂ for k 2.... K, do
Sample random minibatch of N transitions B = {(8, a. r, r,,', ,,,',,) from D.
Compute targets y (r, s', d) = r+y (1 d) Q (s', 8) (s') + ōv (s')).
Update critic by one step of (preconditioned) gradient descent using V (Ø), where
L (Φ) = N
E
(r, - Qo (s,
(s,är,'',dEB
Compute the (agent and adversary) policy gradient estimates:
1 8
Ve (0,wr)
Vena (s) VaQo (s, a) (0=1)-)
N
8
V Var "wJ(0rs (0,,w)
Mw (s) V Qo (s, ②) (10
N
SED
9 VoJ
6=((1) m am a)g C diag (vm+e.
05 (1++11) + glh) + n/ + 2no .c. where ~ N I0.
g' + ViJ (0r,w) (-m m' t am' + (1 a) o o o D + diag (vm +e)
(k+1) t w/ nD o + v 22jo, --àg', where g N 10,
wi
We + (1 3) wi + Bwl (k+1)
0g t (1 - B) 0g + 30 (k+1
Update the target networks:
Dearg + Totarg + (1 7
Ocang + Thrarg + (1 - 7) ) k+1)
Wearg < Twharg + (1 - F)o! )( (k+1)
end for Wh+1 < (1 W) + A3, ; Be+1 + (1-93) 02 + 138 t ++ end for end if until convergence Output: wr, 8r.
15

---

Under review as a conference paper at ICLR 2020
Algorithm 4 Two-Player DDPG with RMSProp Actors (Baseline)
Hlpeeraaaameters: see [
Initialize (randomly) policy parameters w1, 01, and Q-function parameter Φ Initialize the target network parameters wtarg t W1, Oearg t- 01, and Starg + Ø. Initialize replay buffer D.
Initialize < 0; m' +- 0.
1.
repeat
Observe state s, and select actions a = ms, (s)
+ where E, N (0,
Execute the action ā (1 - 8)a + da' in the environment.
Observe reward т, next state s', and done signal d to indicate whether terminal. Store (s, s', d) in replay buffer D. If s' is terminal, reset the environment state.
if it's time update then for however many updates do
Sample random minibatch of N transitions B = ((s, a, r,, ,', from D. Compute targets y (r, s', d)
7 (1 d) Qerace (s', , H) (s') ÖVlerrr (s'))-
Update critic by one step of (preconditioned) gradient descent using V (Ø), where
L (Ø)
1
Σ
-o Qo -oo
N
Compute the (agent and adversary) policy gradient estimates:
1- 8
(B,w))
Volla (s) V.Qa (s, (1
N
SED
8
= V V.J (Or,w)
Vuna (s) V Qo (8, a) (0
N
SED
9 - VoJ (0,
m + am + (1 - a) 2 9 C - diag V/++
0=0;
0111 0, + n
g + V (fe, w)
+ (1 0 D diag (Vm' +c)
ml
Wh+1 f- wi - DD
Update the target networks:
Otarg + Torang + (1 T)Φ
Avarg +- Trtarg + (1 T)0
Wtarg +- TWharg + (1 - 7 7)W(+1
t 1 1. end for end if until convergence Output: wr, Oz.
16

---

Under review as a conference paper at ICLR 2020
Walkar2d-22
HalCChetahb-2
1000
1500
500
0.5
675
15
2.0
05
075
15
20
most
Relative mast
Hkppe-u2
Bosalin
SOLD
830
5010
600
350
800
885
300
250
0.5
2.0
0.
Swimmer-42
Reacher-22
-30
0.
0.5
15
2.0
05
0.6
15
29
Relatve mas
Ralative man
Humanoid-v22
www.rtedFeenuulum-22
300
400
300
080
Basciine
85
2.0
05
29
(-) and Algorithm (-), under the NR-MDP setting with ó = 0. The evaluation is performed without adversarial perturbations, range of mass values not encountered during training.
Figure 7: Average performance (of the best performing seed) of Algorithm 3
17


Reviews:
[{"review_id": "SyxZyNppYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new algorithm for solving Noisy Robust MDPs (NR-MDPs) based on computing an approximate gradient using  stochastic gradient langevin dynamics (SGLD). THE NR-MDPs can be thought of as an agent learning in an adversarial environment.\nThe paper is well written and the references pointed to give sufficient background to understand the problem undertaken.\nMost of the theory comes from the \"Finding mixed nash equilibria of generative adversarial networks.\" (ICML, 2019), and the papers main contribution lies in applying the theory to reinforcement learning.\nIt is not altogether unexpected though that Langevin Dynamics give better results than the more standard gradient-based approach considered for saddle-point problems, that's what they are supposed to do. But, this seems to be the first time SGLD has been applied to such a problem.\nThe authors compare on the MuJoCo benchmark with common but not identical instances to \"Action robust reinforcement learning and applications in continuous control\", which also provides the baseline algorithm used for comparison. Mentioned in this paper is the difficulty of solving the \"inverted pendulum\" instance by the algorithm proposed therein. It is infact mentioned as a failure case. Maybe, the authors can show results on the same. \nAdditionally, maybe it would make sense to have an ablation study of the hyper-parameters from Table 1. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxECxAUiB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).\nThe authors propose to combine SGLD with DDPG, to maintain a distribution over deterministic policies (essentially a mixed strategy) - hence overcoming these issues.\nThe results look promising, outperforming [1] in all tested domains.\n\nReview:\nOverall this seems like the first approach of using Bayesian learning in order to reach mixed Nash equilibrium in RL. This is super-important, since many problems can be formulated as zero-sum games for which the solution is not necessarily a pure strategy.\nAs [2] have already shown the ability of this concept to find mixed equilibria in GANs, the main novelty is the introduction to RL.\n\nAs this work does not introduce new theory or a dramatic new concept, I feel that the acceptance of this work lies mainly on the empirical side. In my opinion, the experiments need to be more convincing - for instance, include additional domains (such as the Inverted Pendulum which was a failure case in [1]) and additional forms of robustness (e.g., the probabilistic-robust variant from [1] which is based on deterministic strategies and was shown to work better, and RARL [3] which test robustness to external disturbances).\nAn addition option is to build a low-dimensional toy problem in which the exact solution is known. This will enable you to show that while the naive solution in [1] either does not converge or converges to sub-optimal solutions, the SGLD approach is capable of finding superior solutions (hopefully the global optimum).\n\nThe idea is in the right direction. However, in my opinion, it is not there yet and is thus not ready for ICLR.\n\n--- Post Rebuttal ---\n\nI stand by my original assessment. I feel that such work needs to be more convincing. I for one would feel more confident had the authors provided simpler experiments in which they show that their approach indeed converges to the mixed Nash equilibria while the NR-DDPG approach from [1] does not.\nI did overall like this direction and I believe Robustness in RL is very important.\n\n\n[1] Action Robust Reinforcement Learning and Applications in Continuous Control - Tessler et al. 2019\n[2] Finding mixed nash equilibria of generative adversarial networks - Hsieh et al. 2019\n[3] Robust Adversarial Reinforcement Learning - Pinto et al. 2017", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xt4BU9Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I have read the author responses. As mentioned before I don't have the background to carefully assess the experiments and will have to rely on my fellow reviewers here. However I stand by my opinion that the experimental results would need to be very strong to warrant an acceptance, since the conceptual contribution is relatively limited.\nI was also disappointed by the authors unwillingness to back up their claims of \"theoretically convergent\" with a proof, or at least a theorem. Therefore I still tend towards rejecting the paper.\n-----------------------------------------------\nSummary\nThe present work proposes to use the recently developed Stochastic Gradient Langevin Dynamics (SGLD) to compute mixed approximate equilibria in two-player reinforcement learning, following the methodology proposed by hsieh et al for generative adversarial networks. The authors report practical improvements compared to pure strategies computed as proposed by Tessler et al.\n\nDecision\nThe idea of using randomized strategies for two-player reinforcement learning is interesting and natural. However, as the authors note, mixed strategies are classical in game theory. Furthermore, the adaption of the methodology of hsieh et al is straightforward, limiting the strength of the theoretical contribution. \nUnfortunately, the theoretical contribution is not stated consistently, since in the introduction, the paper states \"Our paper precisely bridges this gap between theory and practice in previous works, by proposing the ﬁrst theoretically convergent algorithm for robust RL\", but this claim is missing in the abstract or conclusion, and there is no theorem that justifies this rather strong claim. \nIn my opinion, this paper should only be accepted if it provides very convincing numerical experiments, which I am not qualified to assess. I happy to increase my score if the experimental results are deemed strong by the reviewers with more expertise in practical reinforcement learning.\n\nQuestions to authors\nYou write \"Our paper precisely bridges this gap between theory and practice in previous works, by proposing the ﬁrst theoretically convergent algorithm for robust RL\". What is the exact mathematical statement here? Does this refer to the algorithm that is used in the numerical experiments?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1eUcyvaFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper approaches two-player zero-sum Markov game by using mixed Nash equilibrium by sampling from randomized policies. The authors propose that the optimization is done using Stochastic Gradient Langevin Dynamics iterations. \nIn my opinion, the use of SGLD to this problem is potentially useful - as partially proved by this work. However, this application is obvious and does not have enough novelty merits to be accepted to this ICLR. The experiment make senses but is not rigorous enough to assure the practitioners on the improvement over baseline. \nI recommend the authors to further develop this idea in both theoretical improvement and experiment settings to further explore the direction.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the problem of robust reinforcement learning. They propose an adversarial perspective on robustness. Improving the robustness can now be seen as two agent playing a competitive game, which means that in many cases the first agent needs to play a mixed strategy. The authors propose an algorithm for optimizing such mixed strategies. 

Although the reviewers are convinced of the relevance of the work (as a first approach of Bayesian learning to reach mixed Nash equilibria, which is useful not only for robustness but for any problem that can be formulated as zero-sum game requiring a mixed strategy), they are not completely convinced by the work in current state. Three of the reviewers commented on the experiments not being rigorous and convincing enough in current form, and thus not (yet!) being able to recommend acceptance to ICLR. 
Decision: Reject